edge_dump.sh #1

  • //
  • guest/
  • perforce_software/
  • sdp/
  • dev/
  • Server/
  • Unix/
  • p4/
  • common/
  • bin/
  • edge_dump.sh
  • View
  • Commits
  • Open Download .zip Download (2 KB)
#!/bin/bash
#==============================================================================
# Copyright and license info is available in the LICENSE file included with
# the Server Deployment Package (SDP), and also available online:
# https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/LICENSE
#------------------------------------------------------------------------------

# This script is designed to create a filter checkpoint for an Edge server
#
# It requires the SDP instance, the edge server ID and the checkpoint name as parameters to the script.

# Run example:
#  ./edge_dump.sh 1 sfo_edge /depotdata/p4_1.edgeckp.9188.gz

export SDP_INSTANCE=${SDP_INSTANCE:-Undefined}
export SDP_INSTANCE=${1:-$SDP_INSTANCE}
if [[ $SDP_INSTANCE == Undefined ]]; then
   echo "Instance parameter not supplied."
   echo "You must supply the Perforce instance as a parameter to this script."
   exit 1
fi

. /p4/common/bin/p4_vars $SDP_INSTANCE
. /p4/common/bin/backup_functions.sh
LOGFILE=$LOGS/edge_dump.log

######### Start of Script ##########

check_vars
set_vars

if [[ "$3" == "" ]]; then
   echo You must pass in the edge server ID and the full path and filename of the checkpoint you want to create.
   exit 2
fi

EDGEID=$2
EDGECKP=$3

# With -k we filter out the various Edge-specific tables which will be replaced with 
# current live versions.
EXCLUDED_TABLES=db.have,db.working,db.resolve,db.locks,db.revsh,db.workingx,db.resolvex

log "Create checkpoint from master offline_db skipping tables not used on the edge and using the edge server form for any filtered data."
{ time /p4/${SDP_INSTANCE}/bin/p4d_${SDP_INSTANCE} -r /p4/${SDP_INSTANCE}/offline_db/ -k $EXCLUDED_TABLES -P EDGEID -jd -z $EDGECKP; } >> $LOGFILE 2>&1 || { die "Edge reseed checkpoint dump failed!"; }

log "Edge reseed checkpoint complete."

# Inform the user about follow up tasks
log "You now need to copy $EDGECKP to the edge server and use /p4/common/bin/recover_edge.sh to reseed the edge server."

cat $LOGFILE

# Change User Description Committed
#11 27264 ashaikh Improve error message for edge_dump.sh when seed file already exists

When running the edge_dump.sh script, you would get a generic error message if a seed file already existed making you refer to logs to figure out what happened.

Current error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0449.log
Edge seed checkpoint dump failed!

Improved error message:
$ /p4/common/bin/edge_dump.sh 1 edge.1
Processing. This may take a while depending on checkpoint duration.
Log file is: /p4/1/logs/edge_dump.20210112-0456.log
/p4/1/checkpoints/p4_1.edge.1.seed.6.gz already exists.

Refer to job SDP-585.

Thanks for the contribution!
#10 25949 C. Thomas Tyler Fixed typo in variable name.
#9 25938 C. Thomas Tyler Fixed bug in safety check.
#8 25920 C. Thomas Tyler Added new 'edge_vars' file to dynamically set list of edge-specific
db tables based on current P4D version.

Updated edge_dump.sh, recover_edge.sh, and load_checkpoint.sh
to use the new edge_vars file.

Made edge_dump.sh and recover_edge.sh shellcheck v0.6.0 compliant,
along with load_checkpoint.sh.
#7 23297 C. Thomas Tyler Added safety checks to avoid running commands that will certainly fail
in upgrade.sh.

Generally, /p4/common/bin will be the same on all hosts in a Helix topolgy.
However, on any given machine, the /p4/<N>/bin/<EXE>_<N>_init scripts
should exist only for executables that run on that machine.

This change to upgrade.sh should work on machines even where only a
proxy or broker runs.  Also, it will not generate errors in cases
where there is, say, a p4p_N_bin symlink in /p4/common/bin but no
/p4/N/bin/p4p_N_init script, which will a common situation since
/p4/common/bin will contain all executables used anywhere, while
/p4/N/bin is host-specific.

Also made cosmetic fixes and style convergence change.

In dump_edge.sh and recover_edge_dump.sh, just fixed cosmetic typos.
#6 23266 C. Thomas Tyler Fixes and Enhancements:
* Enabled daily_checkpoint.sh operate on edge servers, to
keep /p4/N/offline_db current on those hosts for site-local
recovery w/o requiring a site-local replica (though having
a site-local replica can still be useful).
* Disabled live_checkpoint.sh for edge servers.
* More fully support topologies using edge severs, in both
geographically distributed and horizaontal scaling "wokspace
server" solutions.
* Fix broken EDGESERVER value definition.
* Modified name of SDP counter that gets set when a checkpoint is taken
to incorporate ServerID, so now the counter name will look like
lastSDPCheckpoint.master.1, or lastSDPCheckpoint.p4d_edge_sfo, rather
than just lastSDPCheckpoint.

There will be multiple such counters in a topology that uses edge
servers, and/or which takes checkpoints on replicas.

* Added comments for all functions.

For the master server, journalPrefix remains:
/p4/N/checkpoints/p4_N

The /p4/N/checkpoints is reserved for writing by the
master/commit server only.

For non-standby (possibly filtered) replicas and edge serves,
journalPrefix is:
/p4/N/checkpoints.<ShortServerID>/p4_N.<ShortServerID>

Here, ShortServerID is just the ServerID with the 'p4d_' prefix
trimmed, since it is redundant in this context.  See mkrep.sh,
which enshines a ServerID (server spec) naming standard, with
values like 'p4d_fr_bos' (forwarding replica in Boston) and
p4d_edge_blr (Edge server in Bangalore).  So the journalPrefix
for the p4d_edge_bos replica would be:
/p4/N/checkpoints.edge_bos/p4_N.edge_bos

For "standby" (aka journalcopy) replicas, journalPrefix is set
to /p4/N/journals.rep. which is written to the $LOGS volume, due
to the nature of standby replicas using journalPrefix to write
active server logs to pre-rotated journals.

Some take-away to be updated in docs:
* The /p4/N/checkpoints folder must be reserved for checkpoints that
originate on the master. It should be safe to rsync this folder
(with --delete if desired) to any replica or edge server.  This is
consistent with the current SDP.
* I want to change 'journals.rep' to 'checkpoints.<ShortServerID>'
for non-standby replicas, to ensure that checkpoints and journals
taken on those hosts are written to a volume where they are backed
up.
* In sites with multiple edge serves, some sharing achive files
('workspace servers'), multiple edge servers will share the same
SAN. So we one checkpoints dir per ServerID, and we want that
dir to be on the /hxdepots volume.

Note that the journalPrefix for replicas was a fixed /p4/N/journals.rep.
This was on the /hxlogs volume - a presumably fast-for-writes volume,
but typically NOT backed up and not very large. This change puts it
under /p4/N/checkpoints.* for edge servers and non-standby replicas,
but ensures other replica types and edge servers can generate
checkpoints to a location that is backed up and has plenty of storage
capacity.  For standby replicas only (which cannot be filtered),
the journalPrefix remains /p4/N/journals.rep on the /hxlogs volume.
#5 22752 Russell C. Jackson (Rusty) Corrected comments about -K flag.
#4 22720 Robert Cowham Fix script to create a filtered checkpoint for edge server.
#3 22617 Russell C. Jackson (Rusty) Added +x
#2 22615 Russell C. Jackson (Rusty) Added suggested comments.
By passing review since this is a comment only change.
#1 22614 Russell C. Jackson (Rusty) New script to create a filtered checkpoint to use to rebuild an edge server.
This is not required unless you are filtering data via the server spec. If you are not
doing that, then you can reseed the edge using the latest checkpoint of the master.