Perforce Helix Server Deployment Package (for Windows)

The following are required (details mentioned below):

Optional (but recommended):

The scripts now use Powershell rather than .BAT files due to improved error handling and options, and code re-use (via a single included module rather than duplication of functionality in every script). This also allows us to keep the scripts more closely aligned with the functionality of the Unix scripts.

It is important to enable local scripts to be run. The following command must be run within an Powershell Administrator prompt:

The result needs to be either RemoteSigned or Unrestricted. If not then set it as below.

For Windows 10, Windows Server 2016 or later, run the following commands as Administrator:

Use get-executionpolicy to check the policy has been updated. You may need to ensure that the various scripts are not "blocked" - right click in Windows Explorer and check Properties options.

Prior to installing the Perforce server, perform the following steps.

The create_env.py script, available in %SDP%\Server\Windows\setup, sets up the basic directory structure used by the SDP. It creates .bat files to register the Perforce service as Windows services. It parses and validates the sdp_master_config.ini file in the same directory.

You need to customize this sdp_master_config.ini file. It contains lots of comments as to how to set the various configuration values.

The following shows a sample config after editing:

Review the contents of template_configure_new_server.bat file which defines the recommended default configurable values for any server, and make any desired changes. This file will be parsed and used to create instance specific configuration files.

After updating the configuration file, run create_env.py from the same directory.

Edit and save changes:

Run the command to create the environment (by default it looks for the config file sdp_master_config.ini but this can be changed with -c option):

The output will look something like this (note the value EC2AMAZ-LJ68A4I is an example of the output of the hostname command - your value will be different):

If the output looks correct then re-run the script with -y parameter to actually perform the copying of files and creation of directories and links.

This file is written to c:\p4\config. It will look something like this (note the value EC2AMAZ-LJ68A4I is an example of the output of the hostname command - your value will be different):

The above are the values for a single master/commit server.

If you configure a replica then these fields should be set to appropriate values:

The above command will create a couple of files in that directory. The first is install_services_<hostname>.bat, so on a machine where the hostname is svrp4master, it will be install_services_svrp4master.bat

Validate the contents of this file and run it if it looks appropriate - this installs the service(s) with appropriate parameters. Please note that it is specific to the hostname that you specified inside sdp_master_config.ini - so it will only run on the correct host server. It should look something like:

Note that if you have defined multiple instances in sdp_master_config.ini to run on this same hostname, then they will all be installed by this .bat file.

Having installed the service, we now test that it will start: c:\p4\common\bin\svcinst start -n p4_<instance name>, e.g.

Or

If the service fails to start, then examine the log file for the reason (e.g. missing license file) in c:\p4\<instance>\logs.

Ensure the server is running (specify appropriate port):

Use c:\p4\common\bin\svcinst stop -n p4_<instance> to stop the service if required.

There are several things to do:

Desktop shortcuts simplify setting up the SDP shell environment, making it easy to interact with the configured Helix Core service.

Create a shortcut for each SDP instance on the Desktop of the user account used to manage the Perforce Helix services, and/or the human administrators who login directly to the server machine.

The shortcut brings up a Windows Command Prompt (cmd.exe) with the SDP environment configured.

For each SDP instance configured, create a separate shortcut named P4Admin N, replacing N with the SDP instance name. For example, if the instance name is 1 (the default), create a shortcut named P4Admin 1.

For each instance defined in sdp_master_config.ini, a configuration .bat file will be created, called configure_<instance>.bat, so for instance master, it will be configure_master.bat.

Review the contents of the file and make any desired changes.

You will only be able to run the .bat file if you have started the server instance as per previous section.

If an instance is a replica (or similar), then you should apply the configurables to the master server and then checkpoint it before seeding the replica - see the Distributing Perforce guide.

To configure the server, perform the following steps:

To verify your installation, perform these steps:

In Windows 2012 or later you should use the Task Scheduler. We recommend that you create a folder called Perforce at the top level in which to create your tasks (otherwise they can be hard to find when you next look in Task scheduler!).

Note that the schtasks command can be useful from command line, or taskschd.msc to get the control panel equivalent.

The recommendation is to run the following scheduled tasks on a master server:

If you are on a replica server, then the recommendation is to run these scheduled tasks:

These are all setup in a similar way to the screenshots below Section 3.1.13.1, “Daily-backup.bat”, with the exception of 'rotate-log-files.bat' which requires an argument to be specified which is the <instance>, e.g. 1 or master.

It is sensible to create a Perforce workspace and to store the configuration files in Perforce.

Typically the depot root might be something like //perforce/sdp. If you have many machines, then you might use //perforce/sdp/<machine> using either a physical or a logical name for the machine.

A typical workspace view (e.g. for workspace called p4admin.sdp and for instance master), might be:

You would have appropriate workspaces for each machine, and appropriate lines for each instance on that machine.

Now that the server is running properly, copy the following configuration files to the depotdata volume for backup:

It is possible to add a new instance to an existing machine.

Edit the sdp_master_config.ini and add a new section for the new instance.

The run create_env.py, specifying to just create the new instance.

If the output looks correct then re-run the script with -y parameter to actually perform the copying of files and creation of directories and links.

The easiest way to upgrade a service instance is:

Simple reporting commands to compare before/after include:

Older versions of the Windows SDP (pre June 2014) stored configuration values for each instance in a p4env.bat file within the p4\common\bat directory.

They also didn’t link all directories from c:\p4, but instead used drives such as E:, F: and G: and paths on those drives.

The easiest way to upgrade (most of the work can be done without stopping the service) is:

After the server is installed and configured, most sites will want to modify server permissions (protections) and security settings. Other common configuration steps include modifying the file type map and enabling process monitoring. To configure permissions, perform the following steps:

A p4broker may be created to run a p4broker process that talks to an upstream p4d instance. It may be deployed at the same or separate host as the upstream p4d.

To create and configure:

A commit server instance is the ultimate store for submitted data, and also for any workspace state (WIP - work in progress) for users directly working with the commit server (part of the same "data set")

An edge server instance maintains its own copy of workspace state (WIP). If you have people connecting to an edge server, then any workspaces they create (and files they open for some action) will be only stored on the edge server. Thus it is normally recommended to have an HA backup server, so that users don’t lose their state in case of failover.

There is a concept of a "build edge" which is an edge server which only supports build farm users. In this scenario it may be deemed acceptable to not have an HA backup server, since in the case of failure of the edge, it can be re-seeded from the commit server. All build farm clients would be recreated from scratch so there would be no problems.

A Failover is the hand off of the role of a master/primary/commit server from a primary server machine to a standby replica (typically on a different server machine). As part of failover processing the secondary/backup is promoted to become the new master/primary/commit server.

As of 2018.2 release, p4d supports a p4 failover command that performs a failover to a standby replica (i.e. a replica with Services: field value set to standby or forwarding-standby). Such a replica performs a journalcopy replication of metadata, with a local pull thread to update its db.* files. After the failover is complete, traffic must be redirected to the server machine where newly promoted standby server operates, e.g. with a DNS change (possibly automated with a post-failover trigger).

See also: Configuring a Helix Core Standby.

You can modify the Options: field of the server spec of a standby or forwarding-standby replica to make it mandatory. This setting affects the mechanics of how failover works.

When a standby server instance is configured as mandatory, the master/commit server will wait until this server confirms it has processed journal data before allowing that journal data to be released to other replicas. This can simplify failover if the master server is unavailable to participate in the failover, since it provides a guarantee that no downstream servers are ahead of the replica.

This guarantee is important, as it ensures downstream servers can simply be re-directed to point to the standby after the master server has failed over to its standby, and will carry on working without problems or need for human intervention on the servers.

Failovers in which the master does not participate are generally referred to as unscheduled or reactive, and are generally done in response to an unexpected situation. Failovers in which the master server is alive and well at the start of processing, and in which the master server participates in the failover, are referred to as scheduled or planned.

If set to nomandatory then there is no risk of delaying downstream replicas, however there is no guarantee that they will be able to switch seamlessly over to the new server in event of an unscheduled failover.

To change a server spec to be mandatory or nomandatory, modify the server spec with a command like p4 server p4d_ha_bos to edit the form, and then change the value in the Options: field to be as desired, mandatory or nomandatory, and the save and exit the editor.

This is recommended, but not a requirement for SDP scripts to implement failover.

Examples:

FQDNs (fully qualified DNS names) of short DNS names used in these examples would also exist, and would be based on the same short names.

These are vital as part of your planning.

To configure a replica server, first configure a machine identically to the master server (at least as regards the link structure such as /p4, /p4/common/bin and /p4/instance/*), then install the SDP on it to match the master server installation. Once the machine and SDP install is in place, you need to configure the master server for replication.

Perforce supports many types of replicas suited to a variety of purposes, such as:

We always recommend first setting up the replica as a read-only replica and ensuring that everything is working. Once that is the case you can easily modify server specs and configurables to change it to a forwarding replica, or an edge server etc.

The checkpoint files are stored in the c:\p4\<instance>\checkpoints directory, and the most recent checkpoint is named p4_<instance>.ckp.<number>.gz. Recreating up-to-date database files requires the most recent checkpoint, from c:\p4\<instance>\checkpoints, and the journal file from c:\p4\<instance>\logs.

To recover the server database manually, perform the following steps from the root directory of the server (c:\p4\<instance>\root). In the examples below we assume the <instance> is 1.

If the Perforce service starts without errors, delete the old database files from c:\p4\1\root\save.

If problems are reported when you attempt to recover from the most recent checkpoint, try recovering from the preceding checkpoint and journal. If you are successful, replay the subsequent journal. If the journals are corrupted, contact Perforce Technical Support. For full details about back up and recovery, refer to the Perforce System Administrator’s Guide.

This section describes how to recover from a tape or other offline backup to a new server machine if the server machine fails. The tape backup for the server is made from the depotdata volume. The new server machine must have the same volume layout and user/group settings as the original server. In other words, the new server must be as identical as possible to the server that failed.

To recover from a tape backup, perform the following steps.

c:\p4\<instance>\bin\p4d_<instance> -r c:\p4\<instance>\offline_db -jr -z last_ckp_file

See SDP Failover Guide (PDF) or SDP Failover Guide (HTML) for detailed steps.

Occasionally modifications are made to the Perforce database. For example, server upgrades and some recovery procedures modify the database.

When upgrading the server, replaying a journal patch, or performing any activity that modifies the db.* files, you must restart the offline checkpoint process so that the files in the offline_db directory match the ones in the live server directory. The easiest way to restart the offline checkpoint process is to run the live-checkpoint script after modifying the db.* files Section 7.3.5, “live-checkpoint.ps1”

Archiving labels is a best practice for large installations, with hundreds of users and Perforce checkpoints that are gigabytes in size. Smaller sites need not necessarily concern themselves with archiving labels to maintain performance, though doing so will minimize database size if labels are used extensively.

To use the p4 unload and p4 reload commands for archiving clients and labels, you must first create an unload depot using the p4 depot command. Run:

Set the type of the depot to unload and save the form.

After the depot is created, you can use the following command to archive all the clients and labels that have been accessed since the given date:

For example, to unload all clients and labels that haven’t been accessed since Jan. 1, 2019, you would run:

Users can reload their own clients/labels using the reload command. They can run:

or

As a super user, you can reload and unloaded item by adding the -f flag to the reload command as follows:

In addition, you can avoid having to unload/reload labels by creating a trigger to set the autoreload option as the default on all new labels. That will cause the server to use the unload depot for storing the labels rather than storing them in db.label. This helps with performance of the server by not increasing the size of the database for label storage.

You can automate these tasks with $SDP/Maintenance/unload_clients.py and $SDP/Maintenance/unload_labels.py

The simplest option is to use create a template client workspace (usual name is template.client) and then set configurable template.client to that name. This will mean that all new client workspaces created after that time will have the same options and view, unless otherwise explicitly updated.

Alternatively the old fashioned way is to install a trigger from the Unsupported/Samples/triggers folder.

The form-out trigger $SDP/Unsupported/Sample/triggers/SetWsOptions.py contains default workspace options, such as leaveunchanged instead of submitunchanged.

To use the trigger, first copy it to /p4/common/bin/triggers

To enable the trigger, first modify the OPTIONS variable in the script, providing the set of desired options. Then insert an entry in the trigger table like the following:

The form-save trigger $SDP/Server/common/p4/common/bin/triggers/PreventWsNonAscii.py enforces the policy that no workspaces may contain non-ASCII characters.

To use the trigger, first copy it to /p4/common/bin/triggers

To enable the trigger, insert an entry in the trigger table like the following:

To delete empty pending changelists, run python remove_empty_pending_changes.py.

To prevent large requests from overwhelming the server, you can limit the amount of data and time allowed per query by setting the maxresults, maxscanrows and maxlocktime parameters to the lowest setting that does not interfere with normal daily activities. As a good starting point, set maxscanrows to maxresults * 3; set maxresults to slightly larger than the maximum number of files the users need to be able to sync to do their work; and set maxlocktime to 30000 milliseconds. These values must be adjusted up as the size of your server and the number of revisions of the files grow. To simplify administration, assign limits to groups rather than individual users.

To prevent users from inadvertently accessing large numbers of files, define their client view to be as narrow as possible, considering the requirements of their work. Similarly, limit users' access in the protections table to the smallest number of directories that are required for them to do their job.

Finally, keep triggers simple. Complex triggers increase load on the server.

For remote users who need to sync large numbers of files, Perforce offers a proxy server. P4P, the Perforce Proxy, is run on a machine that is on the remote users' local network. The Perforce Proxy caches file revisions, serving them to the remote users and diverting that load from the main server.

P4P is included in the Windows installer.

P4P does not require special hardware because it doesn’t use much processing power, and it doesn’t need to be backed up. If the P4P instance isn’t working, users can switch their port back to the main server and continue working until the instance of P4P is fixed.

This script is configured to run seven days a week using the Windows scheduler. The script truncates the journal, replays it into the offline_db directory, creates a new checkpoint from the resulting database files, then recreates the offline_db directory from the new checkpoint.

This procedure rebalances and compresses the database files in the offline_db directory. These can be rotated into the live database directory on an occasional (e.g. monthly) basis using Section 7.3.8, “recreate-live-from-offline-db.ps1”.

This script verifies the integrity of the depot files. This script is run by Windows scheduler, usually on a weekly basis, e.g. Saturday morning. It emails the resulting report - please check for any errors contained.

This script creates a checkpoint from the offline_db files, filtered for use with an edge server. Note restrictions below.

See also partner script Section 7.3.7, “recover-edge.ps1” which replays the checkpoint on the edge server machine.

This script recreates offline db from the latest checkpoint found.

Windows version of Unix grep command. Useful for searching inside files.

There is a Windows equivalent which is findstr (although not as powerful).

Windows version of Unix gzip command - useful for checkpoint (de)compression.

This script stops the server, creates a checkpoint from the live database files, recovers from that checkpoint to rebalance and compress the files, then recovers the checkpoint in the offline_db directory to ensure that the database files are optimized.

This script makes a new checkpoint of the modified database files in the live root directory, then recovers that checkpoint to the offline_db directory so that both directories are in sync. This script can also be used anytime to create a checkpoint of the live database (with above warnings about locking!).

This command may be required to be run when an error occurs during offline checkpointing. It restarts the offline checkpoint process from the live database files to bring the offline copy back in sync. If the live checkpoint script fails, contact Perforce Consulting at consulting-helix-core@perforce.com.

Run this script when creating the server and if an error occurs while replaying a journal during the off-line checkpoint process.

This script logs the standard superuser account in to the server with their stored password, using the details from Section 3.1.5, “sdp_config.ini”.

Examples of using the p4login.bat wrapper and specifying SDP Instance parameter:

This script recreates an edge server from create-filtered-edge-checkpoint, maintaining local data such as workspaces (in edge specific db.have table) plus the other 6+ edge tables.

Partner script to Section 7.3.1, “create-filtered-edge-checkpoint.ps1”

This script can be scheduled to run every few months - it used to be run weekly, but that is no longer best practice since database files are not fragmented as they used to be. It will move the db.* files from offline to live root (so requires stopping the service).

This script sends an email with the results of the latest p4 pull -lj. Useful for basic monitoring services.

This script is intended to be run nightly on a replica which may not have any other scheduled tasks running. It ensures that the log files are appropriately rotated and old logs (and journals) are deleted according the settings of KEEP_CKPS in sdp_config.ini

This script is intended to be run on a commit-server/standard server where it is considered too disruptive to use daily-checkpoint.ps1 (because it takes too long or impacts on working day. It will rotate the journal and apply it to the offline_db.

If using this script then create a scheduled task to run on days when daily-checkpoint.ps1 is not run.

This script is the main repository of all shared functions used by other scripts.

They each source the file and then call individual functions as required. This is not intended by be called directly by the user - just sourced by other scripts.

E.g.

It understands how to parse config files, start/stop instances, rotate log files etc.

This script is useful for debugging the setup of the sending of emails by the various scripts.

It sends a test email using the values found in c:\p4\config\sdp_config.ini

If successful then it shows other script emails will work correctly.

See Section A.2, “Emails not being sent” in Appendix A if having problems.

This is used for the Windows service to:

So example usage would be:

This script copies checkpoint files from master to the current replica.

This script upgrades the p4d.exe and related files (including Windows service p4s.exe), performing the appropriate -xu to upgraded the database.

Before running this script you should ensure the new versions of p4.exe and p4d.exe have been downloaded into c:\p4\common\bin

To send from Gmail, you may need to set the "Less Secure App Access" setting, which is configured in the gmail account. While logged into the gmail account from which you want the SDP scripts to send email, go to this URL: https://myaccount.google.com/lesssecureapps

That page should contain a slider button titled Allow less secure apps:. If the value is OFF, slide the slider button to the right so that it indicates ON.

Problems have been observed with some SMTP providers. For example on port 465 which is implicit SSL. There are 2 possibilities:

If the server is explicit SSL then the script will just work, e.g. the relevant entries in Section 3.1.5, “sdp_config.ini”:

The easiest solution for implicit SSL is to run a local copy of stunnel which configures a local port and knows how to talk to the remote server (for example Rackspace, or Gmail).

The relevant section in the example Windows config file:

Using the above we can set the relevant entries in Section 3.1.5, “sdp_config.ini”:

Since stunnel will forward local port 25 to remote port 465.

The important SDP maintenance and backup scripts generate email notifications when they complete.

For further monitoring, you can consider options such as: