Perforce Helix Server Deployment Package (for UNIX/Linux)

USAGE for mkdirs.sh v4.3.0:

mkdirs.sh <instance> [-s <ServerID>] [-t <server_type>] [-MDD /bigdisk] [-MLG /jnl] [-MDB1 /db1] [-MDB2 /db2 ] [-f] [-p] [-test [-clean]] [-n] [-L <log>] [-d|-D]

or

mkdirs.sh [-h|-man]


DESCRIPTION:

This script initializes an SDP instance on a single machine.

This script is intended to support two scenarios:

* First time SDP installation on a given machine.
* Adding new SDP instances (separate Helix Core data sets) to an existing
  SDP installation on a given machine.

And SDP instance is a single Helix Core data set, with its own unique
set of one set of users, changelist numbers, jobs, labels, versioned
files, etc. An organization may run a single instance or multiple
instances.

This is intended to be run either as root or as the operating system
user account (OSUSER) that p4d is configured to run as, typically
'perforce'.  It should be run as root for the initial install.
Subsequent additions of new instances do not require root.

If an initial install as done by a user other than root, various
directories must exist and be writable and owned by 'perforce' before starting:

* /p4
* /hxdepots
* /hxlogs
* /hxmetadata

This script creates an init script in the /p4/N/bin directory.

After running this script, set up the crontab based on templates
generated in /p4/common/etc/cron.d.  For convenience, a sample cronat
is generated for the current machine in /p4/common/etc/cron.d named

crontab.<osuser>.<host>

where <osuser> is the user that services run as (typically 'perforce'),
and <host> is the short hostname (as returned by a 'hostname -s' command).


Next, put the license file in place in the P4ROOT dir, and launch the server
with the init script.

Then run /p4/common/bin/p4master_run instance /p4/common/bin/live_checkpoint.sh
and then run both the daily_checkpoint.sh and recreate_db_checkpoint.sh to
make sure everything is working before setting up the crontab.

Also run /p4/common/bin/p4master_run <instance> /p4/common/bin/p4review.py <instance>
to make sure the review script is working properly.  If you intend to use
Swarm, you can skip configuration of the review daemon, and instead configure
Swarm to handle review-style email notifications.

REQUIRED PARAMETERS:
 <instance>
	Specify the SDP instance name to add.  This is a reference to the Perforce
	Helix Core data set.

OPTIONS:
 -s <ServerID>
	Specify the ServerID, overriding the REPLICA_ID setting in the configuration
	file.

 -S <TargetServerID>
	Specify the ServerID of the P4TARGET of the server being installed.
	Use this when setting up an edge server.

 -t <server_type>
	Specify the server type, overriding the SERVER_TYPE setting in the config
	file.  Valid values are:
	* p4d_master - A master/commit server.
	* p4d_replica - A replica with all metadata from the master (not filtered in any way).
	* p4d_filtered_replica - A filtered replica or filtered forwarding replica.
	* p4d_edge - An edge server.
	* p4d_edge_replica - Replica of an edge server. If used, '-S <TargetServerID>'
	  is required.
	* p4broker - An SDP host running only a broker, with no p4d.
	* p4proxy - An SDP host running a proxy (maybe with a broker in front), with no p4d.

 -MDD /bigdisk
 -MLG /jnl
 -MDB1 /db1
 -MDB2 /db2
	Specify the '-M*' to specify mount points, overriding DD/LG/DB1/DB2
	settings in the config file.  Sample:

	-MDD /bigdisk -MLG /jnl -MDB1 /fast

	If -MDB2 is not specified, it is set the the same value as -MDB1 if
	that is set, or else it defaults to the same default value as DB1.

 -f	Specify -f 'fast mode' to skip chown/chmod commands on depot files.
	This should only be used when you are certain the ownership and
	permissions are correct, and if you have large amounts of existing
	data for which the chown/chmod of the directory tree would be
	slow.

 -p	Specify '-p' to halt processing after preflight checks are complete,
	and before actual processing starts. By default, procesing starts
	immediately upon successful completion of preflight checks.

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) goes to this file
	in the current directory:

	mkdirs.<instance>.<datestamp>.log

	NOTE: This script is self-logging.  That is, output displayed on the screen
	is simultaneously captured in the log file.  Do not run this script with
	redirection operators like '> log' or '2>&1', and do not use 'tee.'

DEBUGGING OPTIONS:
 -test	Specify '-test' to execute a simulated install to /tmp/p4 as the install
	root (rather than /p4), and with the mount point directories specifed in
	the configuration file prefixed with /tmp/hxmounts, defaulting to:
	* /tmp/hxmounts/hxdepots
	* /tmp/hxmounts/hxlogs
	* /tmp/hxmounts/hxmetadata

 -clean
	Specify '-clean' with '-test' to clean up from prior test installs,
	which will result in removal of files/folders installed under /tmp/hxmounts
	and /tmp/p4.

 	Do not specify '-clean' if you want to test a series of installs.

 -n	No-Op.  In No-Op mode, no actions that affect data or structures are taken.
	Instead, commands that would be run are displayed.  This is an alternative
	to -test. Unlike '-p' which stops after the preflight checks, with '-n' more
	processing logic can be exercised, with greater detail about what commands
	that would be executed without '-n'.

 -d     Increase verbosity for debugging.

 -D     Set extreme debugging verbosity, using bash '-x' mode. Also implies -d.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

FILES:
	The mkdirs.sh script uses a configuration file for many settings.  A
	sample file, mkdirs.cfg, is included with the SDP.  After determining
	your SDP instance name (e.g. '1' or 'abc'), create a configuration
	file for it named mkdirs.<N>.cfg, replacing 'N' with your instance.

	Running 'mkdirs.sh N' will load configuration settings from mkdirs.N.cfg.

UPGRADING SDP:
	This script can be useful in testing and upgrading to new versions of the SDP,
	when the '-test' flag is used.

EXAMPLES:
	Example 1: Setup of first instance

	Setup of the first instance on a machine using the default instance name,
	'1', executed after using sudo to become root:
	$ sudo su -
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ vi mkdirs.cfg

	# Adjust settings as desired, e.g P4PORT, P4BROKERPORT, etc.

	$ ./mkdirs.sh 1

	A log will be generated, mkdirs.1.<timestamp>.log

	Example 2: Setup of additional instance named 'abc'.

	Setup a second instanced on the machine, which will be a seprate Helix Core
	instantce with its own P4ROOT, its own set of users and changelists, and
	its own license file (copied from the master instance).

	Note that while the first run of mkdirs.sh on a given machine should be done
	as root, but subsequent instane additions should be done as the 'perforce'
	user (or whatever operating system user accounts Perforce Helix services run
	as).

	$ sudo su - perforce
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ cp -p mkdirs.cfg mkdirs.abc.cfg
	$ vi mkdirs.abc.cfg

	# Adjust settings in mkdirs.abc.cfg as desired, e.g P4PORT, P4BROKERPORT, etc.

	$ ./mkdirs.sh abc

	A log will be generated, mkdirs.abc.<timestamp>.log

USAGE for mkrep.sh v2.6.1:

mkrep.sh -i <SDP_Instance> -t <Type> -s <Site_Tag> -r <Replica_Host> [-f <From_ServerID>] [-p] [-L <log>] [-v<n>] [-n] [-D]

or

mkrep.sh [-h|-man|-V]


DESCRIPTION:
	This script simplifies the task of creating Helix Core replicas and
	edge servers, and helps ensure they are setup with best practices.

	This script does all the metadata configuration to be executed on the
	master server that must be baked into a seed checkpoint for creating
	the replica/edge. It also provides enough information to create,
	transfer, and load seed checkpoints into the replica/edge.  This
	essentially captures the planning for a new replica, and can be
	done before the physical infratructure (hardware and storage) is
	ready.

	Before using this script, a set of geograpic site tags must be defined.
	See the FILES: below for details on a site tags.

	This script adheres to the these SDP Standards:
	* Server Spec Naming Standard: https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/ServerSpecNamingStandard.html
	* Journal Prefix Standard: https://swarm.workshop.perforce.com/view/guest/perforce_software/sdp/main/doc/JournalPrefixStandard.html

	This script does the following to help create a replica or edge server:
	* Generates the server spec for the the replica.
	* Generates a server spec for master server (if needed).
	* Sets configurables ('p4 configure' settings) for replication.
	* Selects the correct 'Services' based on replica type.
	* Creates service user for the replica, and sets a password.
	* Creates service user for the master (if needed), and sets a password.
	* Addes newly created service users to the group 'ServiceUsers'.
	* Verifies the group ServiceUsers is granted super access in the
	protections table (and with the '-p', updations Protections).

	After these steps are completed, detailed instructions are presented
	to the user through the remaining steps needed to complete the
	deployment of the replica.  This starts with creating a new
	checkpoint to capture all the metadata changes made by this
	script.

SERVICE USERS:
	Service users created by this type are always of type 'service',
	and so will not consume a licensed seat.

	Service users also have an 'AuthMethod' of 'perforce' (not
	'ldap') as is required by 'p4d' for 'service' users.  Passwords
	set for service users are long 32 character random strings
	that are not stored, as they are never needed.  Login tickets for
	service users are generated using: p4login -service -v

OPTIONS:
 -i <SDP_Instance>
	Specify the SDP Instance.

 -t <Type>
	Specify the replica type tag. The type corresponds to the 'Type:' and
	'Services:' field of the server spec, which describes the type of services
	offered by a given replica.

	Valid values are:
	* ha:   High Availability standby replica, for 'p4 failover' (P4D 2018.2+)
	* ham:  High Availability metadata-only mandatory standby replica, for 'p4 failover' (P4D 2018.2+)
	* ro:   Read-Only standby replica.
	* rom:  Read-Only standby replica, Metadata only.
	* fr:   Forwarding Replica (Unfiltered).
	* fs:   Forwarding Standby (Unfiltered).
	* frm:  Forwarding Replica (Unfiltered, Metadata only).
	* fsm:  Forwarding Standby (Unfiltered, Metadata only).
	* ffr:  Filtered Forwarding Replica.  Not a valid failover target.
	* edge: Edge Server. Filtered by definition.

	Replicas with 'standby' are always unfiltered, and use the 'journalcopy'
	method of replication, which copies a byte-for-byte verbatim journal file
	rather than one that is merely logically equivalent.

	The tag has several purposes:
	1. Short Hand. Each tag represents a combination of 'Type:' and fully
	qualified 'Services:' values used in server specs.

	2. Distillation. Only the most useful Type/Services combinations have a
	shorthand form.

	3. For forwarding replicas, the name includes the critical distinction of
	whether any replication filtering is used; as filtering of any kind disqualifies
	a replica from being a potential failover target. (No such distinction is
	needed for edge servers, which are filtered by definition).

 -s <Site_Tag>
	Specify a geographic site tag indicating the location and/or data center where
	the replica will physically be located. Valid site tags are defined in the site
	tags file:

	/p4/common/config/SiteTags.cfg

 -r <Replica_Host>
	Specify the target replica host.

 -f <From_ServerID>
	Specify ServerID of the P4TARGET server from which we are replicating.
	This is used to populate the 'ReplicatingFrom' field of the server
	spec. The value must be a valid ServerID.

	By default, this is determined dynamically by checking the ServerID
	of the master server. This option should be used if the target is
	something other than the master. For example, to create an HA replica
	of an edge server, you might specify something like '-f p4d_edge_syd'.

 -p	This script performs a check to ensure that the Protections table grants
	super access to the group ServiceUsers.

	By default, an error is displayed if the check fails, i.e. if super user
	access for the group ServiceUsers cannot be verified. This is
	because, by default, we want to avoid making changes to the Protections
	table. Some sites have local policies or custom automation that requires
	site-specific procedures to update the Protections table.

	If '-p' is specified, an attempt is made to append the Protections table
	an entry like:

	super group ServiceUsers * //...

 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging. By default, all output (stdout and stderr) goes in the logs
	directory referenced by $LOGS environment variable, in a file named
	mkrep.<timestamp>.log

	NOTE: This script is self-logging. That is, output displayed on the screen
	is simultaneously captured in the log file. Do not run this script with
	redirection operators like '> log' or '2>&1', and do not use 'tee.'

 -n	No-Op. Prints commands instead of running them.

 -D	Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Dispay version info for this script and its libraries.

FILES:
	This Site Tags file defines the list of valid geographic site tags:
	/p4/common/config/SiteTags.cfg

	The contains one-line entries of the form:

	<tag>: <description>

	where <tag> is a short alphanumeric tag name for a geographic location,
	data center, or other useful distinction. This tag is incorporated into
       	the ServerID of replicas or edge servers created by this script.  Tag
	names should be kept short, ideally no more than about 5 characters in
	length.

	The <description> is a one-line text description of what the tag
	refers to, which may contain spaces and ASCII punctuation.

	Blank lines and lines starting with a '#' are considered comments
	and are ignored.

REPLICA SERVER MACHINE SETUP:
	The replica/edge server machine must be have the SDP structure installed,
	either using the mkdirs.sh script included in the SDP, or the Helix
	Installer for 'green field' installations.

	When setting up an edge server, a replica of an edge server, or filtered
	replica, confirm that the JournaPrefix Standard (see URL above) structure
	has the separate checkpoints folder as identified in the 'Second Form' in
	the standard.  A baseline SDP structure can typically be extended by running
	commands like like these samples (assuming a ServerID of p4d_edge_syd or
	p4d_ha_edge_syd):

	   mkdir /hxdepots/p4/1/checkpoints.edge_syd
	   cd /p4/1
	   ln -s /hxdepots/p4/1/checkpoints.edge_syd

EXAMPLES:
	EXAMPLE 1 - Set up a High Availability (HA) Replica of the master.

	Add an HA replica to instance 1 to run on host bos-helix-02:
	mkrep.sh -i 1 -t ha -s bos -r bos-helix-02

	EXAMPLE 2 - Add an Edge Server to the topology.

	Add an Edge server to instance acme to run on host syd-helix-04:

	mkrep.sh -i acme -t edge -s syd -r syd-helix-04

	EXAMPLE 3 - Setup an HA replica of an edge server.

	Add a HA replica of the edge server to instance acme to run on host syd-helix-05:

	mkrep.sh -i acme -t ha -f p4d_edge_syd -s syd -r syd-helix-05

USAGE for get_helix_binaries.sh v1.2.0:

get_helix_binaries.sh [-r <HelixMajorVersion>] [-b <Binary1>,<Binary2>,...]

   or

get_helix_binaries.sh -h|-man

DESCRIPTION:
	This script acquires Perforce Helix binaries from the Perforce FTP server.

	The four Helix binaries that can be acquired are:

	* p4, the command line client
	* p4d, the Helix Core server
	* p4p, the Helix Proxy
	* p4broker, the Helix Broker

	This script gets the latest patch of binaries for the current major Helix
	version.  It is intended to acquire the latest patch for an existing install,
	or to get initial binaries for a fresh new install.  It must be run from
	the /hxdepots/sdp/helix_binaries directory (or simiar; the /hxdepots directory
	is the default but is subject to local configuration).

	The helix_binaries directory is used for staging binaries for later upgrade
	with the SDP 'upgrade.sh' script (documented separately).  This helix_binaries
	directory is used to stage binaries on the current machine, while the
	'upgrade.sh' script updates a single SDP instance (of which there might be
	several on a machine).

	The helix_binaries directory may not be in the PATH. As a safety feature,
	the 'verify_sdp.sh' will report an error if the 'p4d' binary is found outside
	/p4/common/bin in the PATH. The SDP 'upgrade.sh' check uses 'verify_sdp.sh'
	as part of its preflight checks, and will refuse to upgrade if any 'p4d' is
	found outside /p4/common/bin.

	When a newer major version of Helix binares is needed, this script should not
	be modified directly. Instead, the recommended approach is to upgrade the SDP
	to get the latest version of SDP first, which will included a newer version of
	this script, as well as the latest 'upgrade.sh'.  The 'upgrade.sh' script
	is updated with each major SDP version to be aware of any changes in
	the upgrade procedure for the corresponding p4d version.  Upgrading SDP first
	ensures you have a version of the SDP that works with newer versions of p4d
	and other Helix binaries.

OPTIONS:
 -r <HelixMajorVersion>
	Specify the Helix Version, using the short form.  The form is rYY.N, e.g. r20.1
	to denote the 2020.1 release. The default: is r20.1

 -b <Binary1>[,<Binary2>,...]
	Specify a comma-delimited list of Helix binaries. The default is: p4 p4d p4broker p4p

 -n	Specify the '-n' (No Operation) option to show the commands needed
	to fetch the Helix binaries from the Perforce FTP server without attempting
	to execute them.

 -D	Set extreme debugging verbosity using bash 'set -x' mode.

HELP OPTIONS:
 -h	Display short help message
 -man	Display this manual page

EXAMPLES:
	Note: All examples assume the SDP is in the standard location, /hxdepots/sdp.

	Example 1 - Typical Usage with no arguments:

	cd /hxdepots/sdp/helix_binaries
	./get_helix_binaries.sh

	This acquires the latest patch of all 4 binaries for the r20.1
	release (aka 2020.1).

	Example 2 - Specifying the major version:

	cd /hxdepots/sdp/helix_binaries
	./get_helix_binaries.sh -r r19.2

	This gets the latest patch of for the 2019.2 release of all 4 binaries.

	Note: Only supported Helix binares are guaranteed to be available from the
	Perforce FTP server.

	Note: Only the latest patch of any given binary is available from the Perforce
	FTP server.

	Example 3 - Sample getting r20.1 and skipping the proxy binary (p4p):

	cd /hxdepots/sdp/helix_binaries
	./get_helix_binaries.sh -r r20.1 -b p4,p4d,p4broker

DEPENDENCIES:
	This script requires outbound internet access. Depending on your environment,
	it may also require HTTPS_PROXY to be defined, or may not work at all.

	If this script doesn't work due to lack of outbound internet access, it is
	still useful illustrating the locations on the Perforce FTP server where
	Helix Core binaries can be found.  If outbound internet access is not
	available, use the '-n' flag to see where on the Peforce FTP server the
	files must be pulled from, and then find a way to get the files from the
	Perforce FTP server to the correct directory on your local machine,
	/hxdepots/sdp/helix_binaries by default.

EXIT CODES:
	An exit code of 0 indicates no errors were encounted. An
	non-zero exit code indicates errors were encounterd.

USAGE for upgrade.sh v4.3.2:

upgrade.sh <instance> [-p] [-c] [-n] [-L <log>] [-d|-D]

or

upgrade.sh [-h|-man]


DESCRIPTION:

This script upgrades the following Helix Core software:

* p4d, the Perforce Helix Core server
* p4broker, the Helix Broker server
* p4p, the Helix Proxy server
* p4, the command line client

Details of each upgrade are described below. Prior to executing any upgrades, a
preflight check is done to help ensure upgrades will go smoothly.  Also, checks
are done to determine what (if any) of the above software products need to be
updated.

To prepare for an upgrade, new binaries must be update in the /p4/sdp/helix_binaries
directory.  This is generally done using the get_helix_binaries.sh script in
that directory.  Binaries in this directory are not referenced by live running
servers, and so it is safe to upgrade files in this directory to stage for a
future upgrade at any time. Also, the SDP standard PATH does not include this
directory, as verified by the verify_sdp.sh script.

THE INSTANCE BIN DIR

The 'instance bin' directory, /p4/<instance>/bin, (e.g. /p4/1/bin for instance
1), is expected to contain *_init scripts for services that operate on the
given machine.

For example, a typical master machine for instance 1 might have the following
in /p4/1/bin:

* p4broker_1_init script
* p4broker_1 symlink
* p4d_1_init script
* p4d_1 symlink or script
* p4_1 symlink (a reference to the 'p4' command line client)

A server machine for instance 1 that runs only the proxy server would have the
following in /p4/1/bin:

* p4p_1_init script
* p4p_1 symlink
* p4_1 symlink

The instance bin directory is never modified by the 'upgrade.sh' script.
The addition of new binaries and update of symlinks occur in .

The existence of *_init scripts for any given binary determines whether this
script attempts to manage the service on a given machine, stopping it before
upgrades, restarting it afterward, and other processing in the case of p4d.

Note that Phase 2, adding new binaries and updating symlinks, will occur for
all binaries for which new staged versions are available, regardless of
whether they are operational on the given machine.

THE COMMON DIR

This script performs it operations in the SDP common bin dir, .

Unlike the instance bin directory, the  directory is expected
to be identical across all machines in a topology.  Scripts and symlinks
should always be the same, with only temporary differences while global
topology upgrades are in progress.

Thus, all binaries available to be upgraded will be upgraded in Phase 2, even
if the binary does not operate on the current machine. For example, if a new
version of 'p4p' binary is available, a new version will be copied to
 and symlink references updated there. However, the p4p binary will
not be stopped/started.

GENERAL UPGRADE PROCESS

This script determines what binaries need to be upgraded, based on what new
binaries are available in the /p4/sdp/helix_binaries directory compared to what binaries
the current instance uses.

There are 5 potential phases. Which phase execute depend on the what binaries
are being upgraded.  The phases are:

* PHASE 1 - Establish a clean rollback point.
This phase executes on the master if p4d is upgraded.

* PHASE 2 - Install new binaries and update SDP symlinks in .
This phase executes for all upgrades.

* PHASE 3 - Stop services to be upgraded.
This phase executes for all upgrades involving p4d, p4p, p4broker.
Only a 'p4' client only upgrade skips this phase.

* PHASE 4 - Perforce p4d schema upgrades
This step involves the 'p4d -xu' processing. It executes if p4d is upgraded
to a new major version, and occurs on the master as well as all replicas/edge
servers. The behavior of 'p4d -xu' differs depending on whether the server is
the master or a replica.

This phase is skipped if upgrading to a patch of the same major version, as
patches do not require 'p4d -xu' processing.

* PHASE 5 - Start upgraded services.
This phase executes for all upgrades involving p4d, p4p, p4broker.
Only a 'p4' client only upgrade skips this phase.

UPGRADING HELIX CORE - P4D

The p4d process, the Perforce Helix Core Server, is the center of the Perforce
Helix Universe, and the only server with a significant database component.
Most of the upgrade phases above are about performing the p4d upgrade.

This 'upgrade.sh' script requires that the 'p4d' service be running at the
beginning of processing if p4d is to be upgraded, and will abort if p4d is
not running.

ORDER OF UPGRADES

Any given Perforce Helix installation will have least one p4d master server, and
may have several other p4d servers deployed on different machines as replicas and
edge servers.  When upgrading multiple p4d servers for any given instance (i.e.
any given data set, with a unique set of changelist numbers and users), the order
in which upgrades are performed matters.  Upgrades must be done in "outer to
inner" order.

The master server, at the center of the topology, is the innermost server and
must be upgraded last. Any replicas or edge servers connected directly to the
master constitue the next outer circle.  These can be upgraded in any order
relative to each other, but must be done before the master and after any
replicas farther out from the master in the topology.  So this 'upgrade.sh'
script should be run first on the server machines that are "outermost" from
the master from a replication perspective, and moving inward. The last run is
done on the master server machine.

Server machines running only proxies and brokers do not have a strict order
dependency for upgrades. These are commonly done in the same "outer to inner"
methodology as p4d for process consistency rather than strict technical need.

See the SDP_Guide.Unix.html for more information related to performing global
topology upgrades.

MASTER JOURNAL ROTATIONS

This script helps minimize downtime for upgrades by taking advantage of the SDP
offline checkpoint mechanism.  Rather than wait for a full checkpoint, a journal
is rotated and replayed to the offline_db.  This typically takes very little
time compared to a checkpoint, reducing downtime needed for the overall upgrade.

When the master server is upgraded, two rotations of the master server's journal
occur during processing.  The first journal rotation occurs before any upgrade
processing occurs, i.e. before the new binaries are added and symlinks are
updated. This gives a clean rollback point.

Later, after the p4d has started and p4d performs its journaled upgrade
processing, a second journal rotation occurs in Phase 5. This second journal
rotation captures all upgrade-related processing in a seprate numbered journal.

UPGRADING HELIX BROKER

Helix Broker (p4broker) servers are commonly deployed on the same machine as a
Helix Core server, and can also be deployed on stand-alone machines (e.g.
deployed to a DMZ host to provide secure access outside a corporate firewall).

Helix Brokers configured in the SDP environment can use a default configuration
file, and may have other configurations. The default configuration is the done
defined in /p4/common/config/p4_N.broker.cfg (or a host-specific override file
if it exists named /p4/common/config/p4_N.broker.<short_hostname>.cfg).  Other
broker configurations may exist, such as a DFM (Down for Maintenance) broker
config /p4/common/config/p4_N.broker.dfm.cfg.

During upgrade processing, this 'uprade.sh' script only stops and restarts the
broker with the default configuration. Thus, if coordinating DFM brokers, first
manually shutdown the default broker and start the DFM brokers before calling
this script. This script will leave the DFM brokers running while adding the
new binaries and updating the symlinks.  (Note: Depending on how services
are configured, this DFM configuration might not survive a machine reboot.
typically the default broker will come online after a machine reboot).

This 'upgrade.sh' script will stop the p4broker service if it is running at the
beginning of processing.  If it was stopped, it will be restarted after the new
binaries are in place and symlinks are updated. If p4broker was not running at the
start of processing, new binaries are added and symlinks updated, but the
p4broker server will not be started.

UPGRADING HELIX PROXY

Helix Proxy (p4p) are commonly deployed on a machine by themselves, with no p4d
and no broker. It may also be run on the same machine as p4d.

This 'upgrade.sh' script will stop the p4p service if it is running at the
beginning of processing.  If it was stopped, it will be restarted after the new
binaries are in place and symlinks are updated. If p4p was not running at the
start of processing, new binaries are added and symlinks updated, but the p4p
server will not be started.

UPGRADING HELIX P4 COMMAND LINE CLIENT

The command line client, 'p4', is upgraded in Phase 2 by addition of new
binaries and updating of symlinks.

STAGING HELIX BINARIES

If your server can reach the Perforce FTP server over the public internet, a
script can be used from the /p4/sdp/helix_binaries directory to get the latest
binaries:

	$ cd /p4/sdp/helix_binaries
	$ ./get_helix_binaries.sh

If your server cannot reach the Perforce FTP server, perhaps due to outbound
network firewall restrictions or operating on an "air gapped" network,
use the '-n' option to see where Helix binaries can be acquired from:

	$ cd /p4/sdp/helix_binaries
	$ ./get_helix_binaries.sh -n

OPTIONS:
 <instance>
	Specify the SDP instance name to add.  This is a reference to the Perforce
	Helix Core data set. This defaults to the current instance based on the
	$SDP_INSTANCE shell environment variable. If the SDP shell environment is
	not loaded, this option is required.

 -p	Specify '-p' to halt processing after preflight checks are complete,
	and before actual processing starts. By default, procesing starts
	immediately upon successful completion of prelfight checks.

 -c	Specify '-c' to execute a command to upgrade the Protections table
	comment format after the p4d upgrade, by using a command like:

        p4 protect --convert-p4admin-comments -o | p4 -s protect -i

	By default, this Protections table conversion is not performed.  In some
	environments with custom policies related to update of the protections
	table, this command may not work.

	The new style of comments and the '--convert-p4admin-comments' option
	was introduced in P4D 2016.1.

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) goes to this file
	in the /p4/N/logs directory (where N is the SDP instance name):

	upgrade.p4_N.<datestamp>.log

	NOTE: This script is self-logging.  That is, output displayed on the
	screen is simultaneously captured in the log file.  Do not run this
	script with redirection operators like '> log' or '2>&1', and do not
	use 'tee.'

	Logging can only be disabled with '-L off' if the '-n' or '-p' flags
	are used. Disabling logging for actual upgrades is not allowed.

DEBUGGING OPTIONS:
 -n	No-Op.  In No-Op mode, no actions that affect data or structures are
	taken.  Instead, commands that would be run are displayed.  This
	command can also be educational, showing various steps that will occur
	during an upgrade.

 -d     Increase verbosity for debugging.

 -D     Set extreme debugging verbosity, using bash '-x' mode. Also implies -d.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	EXAMPLE 1: Typical Usage

	Typical usage is with just the SDP instance name as an argument, e.g.
	instance '1', and no other parameters, as in this example:

	$ cd /p4/common/bin
	$ ./upgrade.sh 1

	This executes the upgrade after successful completion of preflight checks,
	and aborts if preflight checks detected any issues.

	EXAMPLE 2: Preflight Only

	To see if an upgrade is needed for this instance, based on binaries
	staged in /p4/sdp/helix_binaries, use the '-p' flag to execute only the preflight
	checks, and disable logging, as in this example:

	$ cd /p4/common/bin
	$ ./upgrade.sh 1 -p -L off

	EXAMPLE 3: Simplified

	If the standard SDP shell environment is loaded, upgrade.sh will be in
	the path, so the 'cd' command to /p4/common/bin is not needed. Also,
	the SDP_INSTANCE shell environment variable will be defined, so the
	'instance' paramter can be dropped, with simply a call to the script
	needed:

	$ upgrade.sh

SEE ALSO:
	The /verify_sdp.sh script is used for preflight checks.

	The /p4/sdp/helix_binaries/get_helix_binaries.sh script acquires new binaries
	for upgrades.

	Both scripts sport the same '-h' (short help) and '-man' (full manual)
	usage options as this script.

LIMITATIONS:
	This script does not handle upgrades of 'p4dtg', Helix Swarm,
	Helix4Git, or any other software.

USAGE for  v5.3.0:

p4verify.sh [<instance>] [-nu] [-nr] [-ns] [-nS] [-a] [-recent] [-L <log>] [-v] [-D]

   or

p4verify.sh -h|-man

DESCRIPTION:

	This script performs a 'p4 verify' of all submitted and shelved versioned
	files in depots of all types except 'remote' and 'archive' type depots.

	If run on a replica, it schedules archive failures for transfer to the
	replica.

OPTIONS:
<instance>
	Specify the SDP instances.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined, p4verify.sh
	exists immediately with an error message.

 -nu	Specify '-nu' (No Unload) to skip verification of the singleton depot
	of type 'unload' (if created).  The 'unload' depot is verified
	by default.

 -nr	Specify '-nr' (No Regular) to skip verification of regular submitted
	archive files.  The '-nr' option is not compatible with '-recent'.
	Regular submitted archive files are verified by default.

 -ns	Specify '-ns' (No Spec Depot) to skip verification of singleton depot
	of type 'spec' (if created). The 'spec' depot is verified by default.

 -nS	Specify '-nS' (No Shelves) to skip verification of shelved archive
	files, i.e. to skip the 'p4 verify -qS'.

 -a	Specify '-a' (Archive Depots) to do verification of depots of type
	'archive'.  Depots of type 'archive' are not verified by default, as
	archive depots are often physicially removed from the server's
	storage subsystem for long-term cold storage.

 -recent
	Specify that only recent changelists should be verified.
	The $SDP_RECENT_CHANGES_TO_VERIFY variable defines how many
	changelists are considered recent; the default is 200.

	If the default is not appropriate for your site, add
	"export SDP_RECENT_CHANGES_TO_VERIFY" to /p4/common/config/p4_N.vars to
	change the default for an instance, or to /p4/common/bin/p4_vars to
	change it globally.  If $SDP_RECENT_CHANGES_TO_VERIFY is unset, the
	default is 200.

	When -recent is used, neither shelves nor files in the unload depot
	are verified.

 -v     Verbose.  Show output of verify attempts, which is suppressed by default.
	Setting SDP_SHOW_LOG=1 in the shell environment has the same effect as -v.

	The default behavior of this script is to generate no terminal outpout,
	but instead to write output into a log file -- see LOGGING below.  If
	'-v' is specified, the generated log is sent to stdout at the end of
	processing.  This flag is not recommended for routine cron operation or
	for large data sets.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/p4verify.log

	Log rotation and old log cleanup logic does not apply to log files
	specified with -L.  Thus, using -L is not recommended for routine scheduled
	operation, e.g. via crontab.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	This script is typically called via cron with only the instance
	paramter as an argument, e.g.:
	p4verify.sh N

LOGGING:
	This script generates no output by default.  All (stdout and stderr) is
	logged to /p4/N/logs/p4verify.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

	If the '-v' flag is used, the contents of the log are displayed to
	stdout at the end of processing.

EXIT CODES:
	An exit code of 0 indicates no errors were encounted attempting to
	perform verifications, AND that all verifications attempted
	reported no problems.

	A exit status of 1 indicates that verifications could not be
	attempted for some reason.

	A exit status of 2 indicates that verifications were successfully
	performed, but that problems such as BAD or MISSING files
	were detected, or else system limits prevented verification.

USAGE for p4login v4.4.1:

p4login [<instance>] [-p <port> | -service] [-automation] [-all]

   or

p4login -h|-man

DESCRIPTION:

	In its simplest form, this script simply logs in P4USER to P4PORT
	using the defined password access mechanism.

	It generates a login ticket for the SDP super user, defined by
	P4USER when sourcing the SDP standard shell environment.  It is
	called from cron scripts, and so does not normally generate any
	output.

	If run on a replica with the -service option, the serviceUser defined
	for the given replica is logged in.

	The $SDP_AUTOMATION_USERS variable can be defined in
	/p4_N.vars. If defined, this should contain a
	comma-delimited list of automation users to be logged in when the
	-automation option is used.  A definition might look like:

	export SDP_AUTOMATION_USERS=builder,trigger-admin,p4review

	Login behaviour is affected by external factors:
	1. P4AUTH, if defined, affects login behavior on replicas.

	2. The auth.id setting, if defined, affects login behaviors (and
	generally simplifies them).

	3. The $SDP_ALWAYS_LOGIN variable.  If set to 1, this causes p4login
	to always execute a 'p4 login' command to generate a login ticket,
	even if a 'p4 login -s' test indicates none is needed.  By default,
	the login is skipped if a 'p4 login  -s' test indicates a long-term
	ticket is available that expires 31+days in the future.
	Add "export SDP_ALWYAYS_LOGIN=1" to /p4_N.vars to
	change the default for an instance, or to /p4/common/bin/p4_vars to
	change it globally.  If unset, the default is 0.

	4. If the P4PORT contains an ssl: prefix, the P4TRUST relationship
	is checked, and if necessary, a p4 trust -f -y is done to establish
	trust.

OPTIONS:
<instance>
	Specify the SDP instances.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined, p4login
	exists immediately with an error message.

 -service
	Specify -service when run on a replica or edge server to login
	the super user and the replication service user.

	This option is not compatible with '-p <port>'.

 -p <port>
	Specify a P4PORT value to login to, overriding the default
	defined by P4PORT setting in the environment.  If operating
	on a host other than the master, and auth.id is set, this
	flag is ignored; the P4TARGET for the replica is used
	instead.

	This option is not compatible with '-service'.

 -automation
	Specify -automation to login external automation users defined
	by the $SDP_AUTOMATION_USERS variable.

 -v     Show ouptput of login attempts, which is suppressed by default.
	Setting SDP_SHOW_LOG=1 in the shell environment has the same
	effect as -v.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/p4login.log

 -d     Set debugging verbosity.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	1. Typical usage for automation, with instance SDP_INSTANCE defined
	in the environment by sourcing p4_vars, and logging in only the super
	user P4USER to P4PORT:
	source /p4/common/bin/p4_vars abc
	p4login

	Login in only P4USER to the specified port, P4MASTERPORT in this example:
	p4login -p $P4MASTERPORT

	Login the super user P4USER, and then login the replication serviceUser
	for the current ServerID:
	p4login -service

	Login external automation users (see SDP_AUTOMATION_USERS above):
	p4login -automation

	Login all users:
	p4login -all

	Or: p4login -service -automation

LOGGING:
	This script generates no output by default.  All (stdout and stderr) is
	logged to /p4/N/logs/p4login.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

	If the '-v' flag is used, the contents of the log are displayed to
	stdout at the end of processing.

EXIT CODES:
	An exit code of 0 indicates a valid login ticket exists, while a
	non-zero exit code indicates a failure to login.

USAGE for verify_sdp.sh v5.12.0:

verify_sdp.sh [<instance>] [-online] [-skip <test>[,<test2>,...]] [-si] [-L <log>|off ] [-D]

   or

verify_sdp.sh -h|-man

DESCRIPTION:

	This script verifies the current SDP setup for the specified instance,
	and also performs basic health checks of configured servers.

	This uses the SDP instance bin directory /p4/N/bin to determine
	what server binaries (p4d, p4broker, p4p) are expected to be configured
	on this machine.

	Existence of the '*_init' script indicates the given binary is
	expected. For example, for instance 1, if /p4/1/bin/p4d_1_init
	exists, a p4d server is expected to run on this machine.

	Checks may be executed or skipped depending on what servers are
	configured. For example, if a p4d is configured, the $P4ROOT/server.id
	file should exist. If p4p is configured, the 'cache' directory
	should exist.

OPTIONS:
 <instance>
	Specify the SDP instances.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined,
	exits immediately with an error message.

 -online
	Online mode.  Does additional checks that require P4D to be online.

 -skip <test>[,<test2>,...]

	Specify a comma-delimited list of test names to skip.

	Valid test names:

	* crontab: Skip crontab check. Use this if you do not expect crontab to
	be configured, perhaps if you use a different scheduler.
	* license: Skip license related checks.
	* version: Skip version checks.
	* excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH.
	* masterid: Skip check to ensure ServerID of master starts with 'master'.

	As an alternative to using the '-skip' command, the shell environment
	variable VERIFY_SDP_SKIP_TEST_LIST can be set to a comma-separated
	list of test names to skip.  Using the command line parameter is the
	best choice for temporarily skipping tests, while specifying the
	environment variable is better for making permanent exceptions (e.g.
	always excluding the crontab check if crontabs are not used at this
	site).  The variable should be set in /p4/common/config/p4_N.vars.

	If the '-skip' option is provided, the VERIFY_SDP_SKIP_TEST_LIST
	variable is ignored (not appended to). So it may make sense to
	reference the variable on the command line. For example, if the
	value of the variable is 'crontab', to skip crontab and license
	checks, you could specify:

	-skip $VERIFY_SDP_SKIP_TEST_LIST,license

 -si	Silent mode, useful for cron operation.  Both stdout and stderr
	are still captured in the log.  The '-si' option cannot be used
	with '-L off'.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/verify_sdp.log
	The special value 'off' disables logging to a file.

	Note that '-L off' and '-si' are mutually exclusive.

 -D	Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	Example 1: Typical usage:

	This script is typically called after SDP update with only the instance
	name or number as an argument, e.g.:

	verify_sdp.sh 1

	Example 2: Skipping some checks.

	verify_sdp.sh 1 -skip crontab

	Example 3: Automation Usage

	If used from automation already doing its own logging, use -L off:

	verify_sdp.sh 1 -L off

LOGGING:
	This script generates a log file and also displays it to stdout at the
	end of processing.  By default, the log is:
	/p4/N/logs/verify_sdp.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

	If the '-si' (silent) flag is used, the log is generated, but its
	contents are not displayed to stdout at the end of processing.

EXIT CODES:
	An exit code of 0 indicates no errors were encounted attempting to
	perform verifications, and that all checks verified cleanly.

USAGE for load_checkpoint.sh v2.3.6:

load_checkpoint.sh <checkpoint> [-i <instance>] [-s <ServerID>] [-c] [-l] [-r] [-b] [-y] [-L <log>] [-si] [-v<n>] [-D]

or

load_checkpoint.sh [-h|-man|-V]


DESCRIPTION:
	This script loads a specified checkpoint into /p4/N/root and /p4/N/offline_db,
	where 'N' is the SDP instance name.

	At the start of processing, preflight checks are done. Preflight checks
	include:
	* The specified checkpoint and corresponding *.md5 file must exist.
	* The $P4ROOT/server.id file must exist, unless '-s' is specified.
	* The $P4ROOT/license file must exist, unless '-l' is specified.
	* Basic SDP structure and key files must exist.

  	If the preflight passes, the p4d_N service is shutdown, and also the
	p4broker_N service is shutdown if configured.

	Next, the specified checkpoint is loaded. Upon completion, the Helix Core
	server process, p4d_N, is started.

	If the server to be started is a replica, the serviceUser configured for
	the replica is logged into the P4TARGET server.  Any needed 'p4 trust' and
	'p4 login' commands are done to enable replication.

	Note that this part of the processing will fail if the correct super user
	password is not stored in the standard SDP password file,

	/p4/common/config/.p4passwd.p4_N.admin

	After starting the server, a local 'p4 trust' is done if needed, and then
	a 'p4login -service -v' and 'p4login -v'.

	By default, the p4d_N service is started, but the p4broker_N service is not.
	Specify '-b' to restart both services.

ARGUMENTS AND OPTIONS:
 <checkpoint>
	Specify the path to the checkpoint file to load.

	The file may be a compressed or uncompressed checkpoint, and it may bea case
	sensitive or case-insensitive checkpoint.  The checkpoint file must have a
	corresponding *.md5 checksum file in the same directory, with one of two name
	variations: If the checkpoint file is /somewhere/foo.gz, the checksum file may
	be named /somewhere/foo.gz.md5 or /somewhere/foo.md5.

 -i <instance>
	Specify the SDP instance. This can be omitted if SDP_INSTANCE is already
	defined.

 -s <ServerID>
	Specify the ServerID.  This value is written into $P4ROOT/server.id file.

	If no $P4ROOT/server.id file exists, this flag is required.

	If the $P4ROOT/server.id file exists, this argument is not needed. If this
	'-s <ServerID>' is given and a $P4ROOT/server.id file exists, the value in
	the file must match the value specified with this argument.

 -c	Specify that SSL certificates are required, and not to be generated with
	'p4d_N -Gc'.

	By default, if '-c' is not supplied and SSL certs are not availalbe, certs
	are generated automatically with 'p4d_N -Gc'.

 -l	Specify that the server is to start without a license file. By default, if
	there is no $P4ROOT/license file, this script will abort.  Note that if '-l'
	is specified and a license file is actually needed, the attempt this script makes
	to start the server after loading the checkpoint will fail.

 -r	Specify '-r' to replay only to P4ROOT.  By default, this script replays both
	to P4ROOT and the offline_db.

 -b	Specify '-b' to start the a p4broker process (if configured).  By default
	the p4d process is started after loading the checkpoint, but the p4broker
	process is not. This can be useful to ensure the human administrator has
	an opportunity to do sanity checks before enabling the broker to allow
	access by end users (if the broker is deployed for this usage).

 -y	Use the '-y' flag to bypass an interactive warning and confirmation
	prompt.

 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).  The default is 5.

 -L <log>
	Specify the path to a log file.  By default, all output (stdout and stderr)
	goes to:
	/p4/<instance>/logs/load_checkpoint.<timestamp>.log

	NOTE: This script is self-logging.  That is, output displayed on the screen
	is simultaneously captured in the log file.  Do not run this script with
	redirection operators like '> log' or '2>&1', and do not use 'tee.'

 -si	Operate silently.  All output (stdout and stderr) is redirected to the log
	only; no output appears on the terminal.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Dispay version info for this script and its libraries.

EXAMPLES:
	Sample non-interactive usage (bash syntax):
	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -y -si < /dev/null > /dev/null 2>&1 &

	Then, monitor with:
	tail -f $(ls -t $LOGS/load_checkpoint.*.log|head -1)

Usage:

   p4pcm.pl [-d "proxy cache dir"] [-tlow <low_threshold>] [-thigh <high_threshold>] [-n]
or
   p4pcm.pl -h

This utility removes files in the proxy cache if the amount of free disk space
falls below the low threshold (default 10GB).  It removes files (oldest first)
until the high threshold is (default 20GB) is reached. Specify the thresholds
in kilobyte units (kb).

The '-d "proxy cache dir"' argument is required unless $P4PCACHE is defined,
in which case it is used.

The log is $LOGS/p4pcm.log if $LOGS is defined, else p4pcm.log in the current
directory.

Use '-n' to show what files would be removed.

# Valid Geographic site tags.

# Each is intended to indciate a geography, and optionally a specific Data
# Center (or Computer Room, or Computer Closet) within a given geographic
# location.
#
# The format is:
# Name:Description
# The Name must be alphanumeric only. The Description may contain spaces.
# Lines starting with # and blank lines are ignored.

bej: Beijing, China
bos: Boston, MA, USA
blr: Bangalore, India
chi: Chicago greater metro area
cni: Chennai, India
pune: Pune, India
lv: Las Vegas, NV, USA
mlb: Melbourne, Australia
syd: Sydney, Australia