The Server Deployment Package (SDP) is the implementation of Perforce’s recommendations for operating and managing a production Perforce Helix Core Version Control System. It is intended to provide the Helix Core administration team with tools to help:
High Availability (HA)
Disaster Recovery (DR)
This guide is intended to provide instructions for failover in an SDP environment using built in Helix Core features.
For more details see:
Please Give Us Feedback
Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to firstname.lastname@example.org.
We need to consider planned vs unplanned failover. Planned may be due to upgrading the core Operating System or some other dependency in your infrastructure, or a similar activity.
Unplanned covers risks you are seeking to mitigate with failover:
loss of a machine, or some machine related hardware failure
loss of a VM cluster
failure of storage
loss of a data center or machine room
HA failover should not require a P4PORT change for end users. Depending on your topology, you can avoid changing P4PORT by having users set P4PORT to an alias, e.g.
perforce.p4demo.com (or just
perforce_syd.p4demo.com (or just
perforce_syd). During failover, that would be targeted by doing something like:
2. Planned Failover
In this instance you can run
p4 failover with the active participation of its upstream server.
We are going to provide examples with the following assumptions:
master_1is current commit, running on machine
p4d_ha_bosis HA server
perforceis set to
You need to ensure:
you are running p4d 2018.2 or later for your commit and all replica instances, preferably 2020.1
source /p4/common/bin/p4_vars 1 p4 info | grep version
your failover target server instance is of type
On HA machine:
p4 info : ServerID: p4d_ha_bos Server services: standby Replica of: perforce:1999 :
it has Options mandatory set in its server spec
p4 server -o p4d_ha_bos | grep Options Options: mandatory
you have a valid
On HA machine:
Monitoring is enabled - so the following works:
p4 monitor show -al
DNS changes are possible so that downstream replicas can seamlessly connect to HA server
pullstatus is valid
p4 pull -lj
2.2. Failing over
The actions are:
p4 failoverin reporting mode on HA machine:
Successful output looks like:
Checking if failover might be possible ... Checking for archive file content not transferred ... Verifying content of recently update archive files ... After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.
p4 failover --yes
Output should be something like:
Starting failover process ... Refusing new commands on server from which failover is occurring ... Giving commands already running time to complete ... Stalling commands on server from which failover is occurring ... Waiting for 'journalcopy' to complete its work ... Waiting for 'pull -L' to complete its work ... Waiting for 'pull -u' to complete its work ... Checking for archive file content not transferred ... Verifying content of recently updated archive files ... Stopping server from which failover is occurring ... Moving latest journalcopy'd journal into place as the active journal ... Updating configuration of the failed-over server ... Restarting this server ...
During this time, if you run commands against the master, you may see:
Server currently in failover mode, try again after failover has completed
Change the DNS entries so downstream replicas (and users) will connect to the new master (that was previously HA)
Validate that your downstream replicas are communicating with your new master
On each replica machine:
p4 pull -lj
Or against the new master:
p4 servers -J
Check output of
: Server address: box-helix-02 : ServerID: master_1 Server services: commit-server :
Make sure the old server spec (
p4d_ha_bos) has correctly had its
Options:field set to
nomandatory(otherwise all replication would stop!)
3. Unplanned Failover
In this case there is no active participation of upstream server, so there is an increase risk of lost data.
We assume we are still failing over to the HA machine, so:
Failover target is
Server spec still has
Original master is not running
The output of
p4 failover on the DR machine might be:
Checking if failover might be possible ... Server ID must be specified in the '-s' or --serverid' argument for a failover without the participation of the server from which failover is occurring. Checking for archive file content not transferred ... Verifying content of recently update archive files ... After addressing any reported issues that might prevent failover, use --yes or -y to execute the failover.
p4 failoverwith the extra parameter to specify server we are failing over from:
p4 failover --serverid master_1 --yes
Expected output is somewhat shorter than for planned failover:
Starting failover process ... Waiting for 'pull -L' to complete its work ... Checking for archive file content not transferred ... Verifying content of recently updated archive files ... Moving latest journalcopy'd journal into place as the active journal ... Updating configuration of the failed-over server ... Restarting this server ...
4. Old style failover
This does not use the
p4 failover command (so is valid for pre-2018.2 p4d versions)