Perforce Helix Core Server Deployment Package (for UNIX/Linux)

    // this block should be a peer of 'p4'
    'mail' => array(
        // 'sender' => 'swarm@my.domain',   // defaults to 'notifications@hostname'
        'transport' => array(
            'name' => 'localhost',          // name of SMTP host
            'host' => 'localhost',          // host/IP of SMTP host
            ),
        ),
    ),

USAGE for get_helix_binaries.sh v1.7.3:

get_helix_binaries.sh [-r <HelixMajorVersion>] [-b <Binary1>,<Binary2>,...] [-api] [-sbd <StageBinDir>] [-n] [-d|-D]

   or

get_helix_binaries.sh -h|-man

DESCRIPTION:
	This script acquires Perforce Helix binaries from the Perforce FTP server.

	The four Helix binaries that can be acquired are:

	* p4, the command line client
	* p4d, the Helix Core server
	* p4p, the Helix Proxy
	* p4broker, the Helix Broker

	In addition, P4API, the C++ client API, can be downloaded.

	This script gets the latest patch of binaries for the current major Helix
	version.  It is intended to acquire the latest patch for an existing install,
	or to get initial binaries for a fresh new install.  It must be run from
	the /p4/sdp/helix_binaries directory in order for the upgrade.sh script
	to find the downloaded binaries.

	The helix_binaries directory is used for staging binaries for later upgrade
	with the SDP 'upgrade.sh' script (documented separately).  This helix_binaries
	directory is used to stage binaries on the current machine, while the
	'upgrade.sh' script uses the downloaded binaries to upgrade a single SDP
	instance (of which there might be several on a machine).

	The helix_binaries directory must NOT be in the PATH. As a safety feature,
	the 'verify_sdp.sh' will report an error if the 'p4d' binary is found outside
	/p4/common/bin in the PATH. The SDP 'upgrade.sh' check uses 'verify_sdp.sh'
	as part of its preflight checks, and will refuse to upgrade if any 'p4d' is
	found in the PATH outside /p4/common/bin.

	When a newer major version of Helix binaries is needed, this script should not
	be modified directly. Instead, get the latest version of SDP first, which will
	included a newer version of this script, as well as the latest 'upgrade.sh'
	The 'upgrade.sh' script is updated with each major SDP version to be aware of
	any changes in the upgrade procedure for the corresponding p4d version.
	Upgrading SDP first ensures you have a version of the SDP that works with
	newer versions of p4d and other Helix binaries.

PLATFORM DETECTION
	The 'uname' command is used to determine the architecture for the current
	machine on which this script is run.

	This script and supporting P4*.json release list files know what platforms
	for which builds are available for each Helix Core binary (p4, p4d, p4broker,
	p4p).  If the 'jq' utility is available, this script uses the P4*.json files
	to verify that a build is available for the current platform, and in some
	cases selects an alternate compatible platform if needed. For example, if
	the detected platform is for OSX 12+ for the x86_64 architecture, no build
	is available for binaries such as p4d for that platform, so a compatible
	alternative is used instead, in this case macosx1015x86_64.

	This script handles only the UNIX/Linux platforms (to include OSX).

RELEASE LIST FILES:
	For each binary, there is a corresponding release list file (in json format)
	that indicates the platforms available for the given binary.  These files are:

	P4.json (for the 'p4' binary)
	P4D.json (for the 'p4d' binary)
	P4Broker.json (for the 'p4broker' binary)
	P4Proxy.json (for the 'p4p' binary)

	These P4*.json release list files are aware of a wide list of supported
	platforms for a range of Helix Core binaries.

	These release list files are packaged with the SDP, and updated for each
	major release.

OPTIONS:
 -r <HelixMajorVersion>
	Specify the Helix Version, using the short form.  The form is rYY.N, e.g. r21.2
	to denote the 2021.2 release. The default: is r24.2

	The form of 'rYY.N', e.g. 'r24.1', is the default form of the version, matching
	what is used in URLS on the Perforce Helix FTP server.  For flexibility, similar
	forms that clearly convent the intended version are also accepted.  For example:

	'-r 23.1' is implicitly converted to '-r r23.1'.
	'-r 2023.1' is implicitly converted to ' -r r23.1'.

 -b <Binary1>[,<Binary2>,...]
	Specify a comma-delimited list of Helix binaries. The default is: p4 p4d p4broker p4p

	Alternately, specify '-b none' in conjunction with '-api' to download only APIs
	and none of the p4* binaries.

 -api
	Specify '-api' to download P4API, the C++ client API.  This will acquire one or
	more client API tarballs, depending on the current platform.  The API files will
	look something like these examples:

	* p4api-glibc2.3-openssl1.1.1.tgz
	* p4api-glibc2.3-openssl3.tgz
	* p4api-glibc2.12-openssl1.1.1.tgz
	* p4api-glibc2.12-openssl3.tgz

	* p4api-openssl1.1.1.tgz
	* p4api-openssl3.tgz

	All binaries that match 'p4api*tgz' in the relevant directory on the Perforce
	FTP server for the current architecture and Helix Core version are downloaded.

	Unlike binary downloads, the old versions are not checked, because file names are
	fixed as they are with binares.

	APIs are not needed for normal operations, and are only downloaded if requested
	with the '-api' option. They may be useful for developing custom automation such
	as custom triggers.  Be warned, custom triggers are not supported by Perforce Support.

 -sbd <StageBinDir>
 	Specify the staging directory to install downloaded binaries.

	By default, this script downloads files into the current directory, which
	is expected and required to be /p4/sdp/helix_binaries.  Documented workflows
	for using this script involve first cd'ing to that directory.  Using this
	option disables the expected directory check and allows binaries to be
	installed in any directory, which may be useful if this script is used
	as a standalone script outside the SDP (e.g. for setting up test
	environments or enabling Helix native DVCS features by installing binaries
	into /usr/local/bin on a non-SDP machine.

	This option also sets the location in which this script searches for the
	P4*.json release list files.

 -n	Specify the '-n' (No Operation) option to show the commands needed
	to fetch the Helix binaries from the Perforce FTP server without attempting
	to execute them.

 -d	Set debugging verbosity.

 -D	Set extreme debugging verbosity using bash 'set -x' mode. Implies '-d'.

HELP OPTIONS:
 -h	Display short help message
 -man	Display this manual page

EXAMPLES:
	Example 1 - Typical Usage with no arguments:

	cd /p4/sdp/helix_binaries
	./get_helix_binaries.sh

	This acquires the latest patch of all 4 binaries for the r24.2
	release (aka 2024.2).

	This will not download APIs, which are not needed for general operation.

	Example 2 - Specify the major version:

	cd /p4/sdp/helix_binaries
	./get_helix_binaries.sh -r r21.2

	This gets the latest patch of for the 2021.2 release of all 4 binaries.

	Note: Only supported Helix binaries are guaranteed to be available from the
	Perforce FTP server.

	Note: Only the latest patch of any given binary is available from the Perforce
	FTP server.

	Example 3 - Get r22.2 and skip the proxy binary (p4p):

	cd /p4/sdp/helix_binaries
	./get_helix_binaries.sh -r r22.2 -b p4,p4d,p4broker

	Example 4 - Download r23.1 binaries in a non-default directory.

	cd /any/directory/you/want
	./get_helix_binaries.sh -r r23.1 -sbd .

	or:

	./get_helix_binaries.sh -r r23.2 -sbd /any/directory/you/want

	Example 5 - Download C++ client API only:

	./get_helix_binaries.sh -r r24.1 -b none -api

DEPENDENCIES:
	This script requires outbound internet access. Depending on your environment,
	it may also require HTTPS_PROXY to be defined, or may not work at all.

	If this script doesn't work due to lack of outbound internet access, it is
	still useful illustrating the locations on the Perforce FTP server where
	Helix Core binaries can be found.  If outbound internet access is not
	available, use the '-n' flag to see where on the Perforce FTP server the
	files must be pulled from, and then find a way to get the files from the
	Perforce FTP server to the correct directory on your local machine,
	/p4/sdp/helix_binaries by default.

EXIT CODES:
	An exit code of 0 indicates no errors were encountered. An
	non-zero exit code indicates errors were encountered.

USAGE for upgrade.sh v4.12.4:

upgrade.sh <instance> [-p|-I] [-M] [-Od] [-Osp] [-c] [-y] [-L <log>] [-d|-D]

or

upgrade.sh [-h|-man]


DESCRIPTION:

    This script upgrades the following Helix Core software:

    * p4d, the Perforce Helix Core server
    * p4broker, the Helix Broker server
    * p4p, the Helix Proxy server
    * p4, the command line client

    The preferred process for using this script is to start with the services
    to be upgraded (p4d, p4broker, and/or p4p ) up and running at the
    start of processing. The p4d service must be online if it is to be
    upgraded.

    Details of each upgrade are described below. Prior to executing any upgrades, a
    preflight check is done to help ensure upgrades will go smoothly.  Also, checks
    are done to determine what (if any) of the above software products need to be
    updated.

    To prepare for an upgrade, new binaries must be update in the /p4/sdp/helix_binaries
    directory.  This is generally done using the get_helix_binaries.sh script in
    that directory.  Binaries in this directory are not referenced by live running
    servers, and so it is safe to upgrade files in this directory to stage for a
    future upgrade at any time. Also, the SDP standard PATH does not include this
    directory, as verified by the verify_sdp.sh script.

THE INSTANCE BIN DIR

    The 'instance bin' directory, /p4/<instance>/bin, (e.g. /p4/1/bin for instance
    1), is expected to contain *_init scripts for services that operate on the
    current machine.

    For example, a typical commit server machine for instance 1 might have the
    following in /p4/1/bin:

    * p4broker_1_init script
    * p4broker_1 symlink
    * p4d_1_init script
    * p4d_1 symlink or script
    * p4_1 symlink (a reference to the 'p4' command line client)

    A server machine for instance 1 that runs only the proxy server would have the
    following in /p4/1/bin:

    * p4p_1_init script
    * p4p_1 symlink
    * p4_1 symlink

    The instance bin directory is never modified by the 'upgrade.sh' script.
    The addition of new binaries and update of symlinks occur in .

    The existence of *_init scripts for any given binary determines whether this
    script attempts to manage the service on a given machine, stopping it before
    upgrades, restarting it afterward, and other processing in the case of p4d.

    Note that Phase 2, adding new binaries and updating symlinks, will occur for
    all binaries for which new staged versions are available, regardless of
    whether they are operational on the given machine.

THE COMMON DIR

    This script performs it operations in the SDP common bin dir, .

    Unlike the instance bin directory, the  directory is expected
    to be identical across all machines in a topology.  Scripts and symlinks
    should always be the same, with only temporary differences while global
    topology upgrades are in progress.

    Thus, all binaries available to be upgraded will be upgraded in Phase 2, even
    if the binary does not operate on the current machine. For example, if a new
    version of 'p4p' binary is available, a new version will be copied to
     and symlink references updated there. However, the p4p binary will
    not be stopped/started.

GENERAL UPGRADE PROCESS

    This script determines what binaries need to be upgraded, based on what new
    binaries are available in the /p4/sdp/helix_binaries directory compared to what binaries
    the current instance uses.

    There are 5 potential phases. Which phases execute depend on the set of binaries
    being upgraded.  The phases are:

    * PHASE 1 - Establish a clean rollback point.
    This phase executes on the master if p4d is upgraded.

    * PHASE 2 - Install new binaries and update SDP symlinks in .
    This phase executes for all upgrades.

    * PHASE 3 - Stop services to be upgraded.
    This phase executes for all upgrades involving p4d, p4p, p4broker.
    Only a 'p4' client only upgrade skips this phase.

    * PHASE 4 - Perforce p4d schema upgrades
    This step involves the 'p4d -xu' processing. It executes if p4d is upgraded
    to a new major version, and occurs on the master as well as all replicas/edge
    servers. The behavior of 'p4d -xu' differs depending on whether the server is
    the master or a replica.

    This phase is skipped if upgrading to a patch of the same major version, as
    patches do not require 'p4d -xu' processing.

    * PHASE 5 - Start upgraded services.
    This phase executes for all upgrades involving p4d, p4p, p4broker.
    Only a 'p4' client only upgrade skips this phase.

SPECIAL CASE - TO OR THRU P4D 2019.1

    If you are upgrading from a version that is older than 2019.1, services
    are NOT restarted after the upgrade in Phase 5, except on the master.
    Services must be restarted manually on all other servers.

    For these 'to-or-thru' 2019.1 upgrades, after ensuring all replicas/edges
    are caught up (per 'p4 pull -lj'), shutdown all servers other than the
    master.

    Proceeding outer-to-inner, execute this script like so on all machines
    except the master:

    1. Deploy new executables in /p4/sdp/helix_binaries
    2. Stop p4d.
    3. Run 'verify_sdp.sh -skip cron,version'; fix problems if needed until it reports clean.
    4. Run 'upgrade.sh -M' to update symlinks.
    5. Do the upgrade manually with:  p4d -xu
    6. Leave the server offline.

    On the master, execute like this:

    1. Deploy new executables in /p4/sdp/helix_binaries
    2. Run 'verify_sdp.sh -skip cron,version'; fix problems if needed until it reports clean.
    3. upgrade.sh

    When the script completes (it will wait for 'p4 storage' upgrades),
    restart services manually after the upgrade in the 'inner-to-outer'
    direction.  Restart services on replicas/edges going inner-to-outer

    This procedure requiring extra steps is specific to 'to-or-thru' P4D 2019.1
    upgrades.  For upgrades starting from P4D 2019.1 or later, things are simpler.

UPGRADES FOR P4D 2019.1+

    For upgrades where the P4D start version is 2019.1 and going to any subsequent
    version, run this script going outer-to-inner. On each machine, it leaves the
    services online and running.  Going in the outer-to-inner direction an all
    servers, do:

    1. Deploy new executables in /p4/sdp/helix_binaries
    2. Run 'verify_sdp.sh -skip cron,version'; fix problems if needed until it reports clean.
    3. upgrade.sh

UPGRADE PREPARATION

   The steps for deploying new binaries to server machines and running verify_sdp.sh
   (and potentially correcting any issues it discovers) can and should be done before
   the time or even day of any planned upgrade.

UPGRADING HELIX CORE - P4D

    The p4d process, the Perforce Helix Core Server, is the center of the Perforce
    Helix universe, and the only server with a significant database component.
    Most of the upgrade phases above are about performing the p4d upgrade.

    This 'upgrade.sh' script requires that the 'p4d' service be running at the
    beginning of processing if p4d is to be upgraded, and will abort if p4d is
    not running.

ORDER OF UPGRADES

    Any given Perforce Helix installation will have least one p4d master server, and
    may have several other p4d servers deployed on different machines as replicas and
    edge servers.  When upgrading multiple p4d servers for any given instance (i.e.
    any given data set, with a unique set of changelist numbers and users), the order
    in which upgrades are performed matters.  Upgrades must be done in "outer to
    inner" order.

    The master server, at the center of the topology, is the innermost server and
    must be upgraded last. Any replicas or edge servers connected directly to the
    master constitute the next outer circle.  These can be upgraded in any order
    relative to each other, but must be done before the master and after any
    replicas farther out from the master in the topology.  So this 'upgrade.sh'
    script should be run first on the server machines that are "outermost" from
    the master from a replication perspective, and moving inward. The last run is
    done on the master server machine.

    Server machines running only proxies and brokers do not have a strict order
    dependency for upgrades. These are commonly done in the same "outer to inner"
    methodology as p4d for process consistency rather than strict technical need.

    See the SDP_Guide.Unix.html for more information related to performing global
    topology upgrades.

COMMIT SERVER JOURNAL ROTATIONS

    This script helps minimize downtime for upgrades by taking advantage of the SDP
    offline checkpoint mechanism.  Rather than wait for a full checkpoint, a journal
    is rotated and replayed to the offline_db.  This typically takes very little
    time compared to a checkpoint, reducing downtime needed for the overall upgrade.
    It also prepares the offline_db in case a rollback is needed.

    When the commit server is upgraded, two rotations of the commit server's journal
    occur during processing for major upgrades, and a single journal rotation is done
    for patch upgrades.  The first journal rotation occurs before any upgrade
    processing occurs, i.e. before the new binaries are added and symlinks are
    updated. This gives a clean rollback point.  This journal is immediately replayed
    to the offline_db.

    Later, after p4d has started performs its journaled upgrade processing, a second
    journal rotation occurs in Phase 5 if a major upgrade was done. This second
    journal rotation captures all upgrade-related processing in a separately numbered
    journal.  This second journal is not applied to the offline_db by this script.
    Instead, the replay of the second journal to the offline_db will occur the next
    time a call is made to the daily_checkpoint.sh or rotate_journal.sh, e.g. via
    routine crontab.  For a p4d patch upgrade, there will not be any upgrade
    processing.

    In the very unlikely event that a rollback were to ever be needed, the
    offline_db is left in a state that it could be used for a fast rollback
    on the commit server.

MULTI-SERVER OUTER-TO-INNER UPGRADES
    Before starting an outer-to-inner upgrade involving multiple p4d servers,
    (standby, edge, and other p4d replica servers), a manual journal rotation
    should be done on the commit server before starting to call upgrade.sh
    on each of the p4d servers in outer-to-inner order.  Take note of the
    journal counter used for this pre-start journal rotation.  It can be
    useful in event of a rollback.  That journal may need to be replayed
    to the offline_db on all servers other than the commit in a rollback
    scenario.

    In prepartion in days or weeks before an upgrade, every p4d server in
    the topology should be checked to ensure its offline_db is healthy
    and current.

ROLLBACK
    In the very unlikely event that a rollback is needed, bear in mind the
    following:
    * There is no standard procedure for rolling back, because a procedure
    would need to take into account the reason a decision was made to
    rollback.  Presumably the decision would be driven by some kind of
    failure. A large factor in determining whether rollback is practical
    is the point in the process at which a rollback is needed.  In some
    situations, a 'Fix and Roll Forward' approach may be more pragmatic
    than a rollback, and should always be considered.
    * This script and supporting documentation will help prepare your data
    for as smooth a rollback as possible should it ever become necessary.
    * To best prepare for a rollback, it is essential to manage user
    lockout as part of your overall maintenance procedure.  Then let users
    back in after you have confirmed you are moving forward.  User lockout
    is outside the scope of this script, but can be managed using several
    possible methods such as:
      - Crafting a special Protections table to be used during maintenance,
      - Using "Down for Maintenance" brokers,
      - Using network and/or on-host firewall rules,
      - Using temporary ports for maintenance.
    * If Phase 2 (update of symlinks and binaries) completed and must be
    undone, than can be achieved by putting the pre-upgrade binaries in
    place in the directory /p4/sdp/helix_binaries, named simply p4, p4d,
    p4broker, and p4p.  Then run a command like this example for Instance 1:

    upgrade.sh 1 -M -I -y

    This will change symlinks back to reference the older versions.  The
    new binaries will still exist in /p4/common/bin, but will no longer
    be referenced for Instance 1.

UPGRADING HELIX BROKER

    Helix Broker (p4broker) servers are commonly deployed on the same machine as a
    Helix Core server, and can also be deployed on stand-alone machines (e.g.
    deployed to a DMZ host to provide secure access outside a corporate firewall).

    Helix Brokers configured in the SDP environment can use a default configuration
    file, and may have other configurations. The default configuration is the done
    defined in /p4/common/config/p4_N.broker.cfg (or a host-specific override file
    if it exists named /p4/common/config/p4_N.broker.<short_hostname>.cfg).  Other
    broker configurations may exist, such as a DFM (Down for Maintenance) broker
    config /p4/common/config/p4_N.broker.dfm.cfg.

    During upgrade processing, this 'upgrade.sh' script only stops and restarts the
    broker with the default configuration. Thus, if coordinating DFM brokers, first
    manually shutdown the default broker and start the DFM brokers before calling
    this script. This script will leave the DFM brokers running while adding the
    new binaries and updating the symlinks.  (Note: Depending on how services
    are configured, this DFM configuration might not survive a machine reboot.
    typically the default broker will come online after a machine reboot).

    This 'upgrade.sh' script will stop the p4broker service if it is running at the
    beginning of processing.  If it was stopped, it will be restarted after the new
    binaries are in place and symlinks are updated. If p4broker was not running at the
    start of processing, new binaries are added and symlinks updated, but the
    p4broker server will not be started.

UPGRADING HELIX PROXY

    Helix Proxy (p4p) are commonly deployed on a machine by themselves, with no p4d
    and no broker. It may also be run on the same machine as p4d.

    This 'upgrade.sh' script will stop the p4p service if it is running at the
    beginning of processing.  If it was stopped, it will be restarted after the new
    binaries are in place and symlinks are updated. If p4p was not running at the
    start of processing, new binaries are added and symlinks updated, but the p4p
    server will not be started.

UPGRADING HELIX P4 COMMAND LINE CLIENT

    The command line client, 'p4', is upgraded in Phase 2 by addition of new
    binaries and updating of symlinks.

STAGING HELIX BINARIES

    If your server can reach the Perforce FTP server over the public internet, a
    script can be used from the /p4/sdp/helix_binaries directory to get the latest
    binaries:

    $ cd /p4/sdp/helix_binaries
    $ ./get_helix_binaries.sh

    If your server cannot reach the Perforce FTP server, perhaps due to outbound
    network firewall restrictions or operating on an "air gapped" network,
    use the '-n' option to see where Helix binaries can be acquired from:

    $ cd /p4/sdp/helix_binaries
    $ ./get_helix_binaries.sh -n

OPTIONS:
 <instance>
    Specify the SDP instance name to add.  This is a reference to the Perforce
    Helix Core data set. This defaults to the current instance based on the
    $SDP_INSTANCE shell environment variable. If the SDP shell environment is
    not loaded, this option is required.

 -p Specify '-p' to halt processing after preflight checks are complete,
    and before actual processing starts. By default, processing starts
    immediately upon successful completion of preflight checks.

 -Od
    Specify '-Od' to override the rule preventing downgrades.

    WARNING: This is an advanced option intended for use by or with the
    guidance of Perforce Support or Perforce Consulting.

 -Osp
    Specify '-Osp' to override the sudo preflight, skipping that check.

    WARNING: This is an advanced option intended for use by or with the
    guidance of Perforce Support or Perforce Consulting.

 -I
    Specify '-I' to ignore preflight errors. Use of this flag is STRONGLY
    DISCOURAGED, as the preflight checks are essential to ensure a safe
    and smooth migration. If used, preflight checks are still done so
    their errors are recorded in the upgrade log, and then the migration
    will attempt to proceed.

    WARNING: This is an advanced option intended for use by or with the
    guidance of Perforce Support or Perforce Consulting.

 -M
    Specify '-M' if you plan to do a manual upgrade. With this option,
    only Phase 2 processing, adding new staged binaries and updating
    symlinks, is done by this script.

    If '-M' is used, this script does not check that services to
    be upgraded are online at the start of processing, nor does it
    attempt to start to stop services.  If '-M' is used, the
    services should be stopped manually before calling this script,
    and then started manually after.

    WARNING: This is an advanced option intended for use by or with the
    guidance of Perforce Support or Perforce Consulting.

 -c
    Specify '-c' to execute a command to upgrade the Protections table
    comment format after the p4d upgrade, by using a command like:

        p4 protect --convert-p4admin-comments -o | p4 -s protect -i

    By default, this Protections table conversion is not performed.  In some
    environments with custom policies related to update of the protections
    table, this command may not work.

    The new style of comments and the '--convert-p4admin-comments' option
    was introduced in P4D 2016.1.

 -L <log>
    Specify the path to a log file, or the special value 'off' to disable
    logging.  By default, all output (stdout and stderr) goes to this file
    in the /p4/N/logs directory (where N is the SDP instance name):

    upgrade.p4_N.<datestamp>.log

    NOTE: This script is self-logging.  That is, output displayed on the
    screen is simultaneously captured in the log file.  Redirection
    operators like '> log' and '2>&1' are not required, nor is 'tee'.

    Logging can only be disabled with '-L off' if the '-n' or '-p' flags
    are used. Disabling logging for actual upgrades is not allowed.

 -y
    Specify the '-y' option to confirm that the upgrade should be done.

    By default, this script operates in No-Op mode, meaning no actions
    that affect data or structures are taken.  Instead, commands that would
    be run are displayed.  This mode can be educational, showing various
    steps that will occur during an actual upgrade.

DEBUGGING OPTIONS:
 -d     Increase verbosity for debugging.

 -D     Set extreme debugging verbosity, using bash '-x' mode. Also implies -d.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
    EXAMPLE 1: Preflight Only

    To see if an upgrade is needed for this instance, based on binaries
    staged in /p4/sdp/helix_binaries, use the '-p' flag to execute only the preflight
    checks, and disable logging, as in this example:

    $ cd /p4/common/bin
    $ ./upgrade.sh 1 -p -L off

    EXAMPLE 2: Typical Usage

    Typical usage is with just the SDP instance name as an argument, e.g.
    instance '1', and no other parameters, as in this example:

    $ cd /p4/common/bin
    $ ./upgrade.sh 1

    This first runs preflight checks, and aborts if preflight checks
    detected any issues. The it gives a preview of the upgrade. A
    successful preview completes with a line near the end that looks like
    this sample:

    Success: Finished p4_1 Upgrade.

    If the preview is successful, then proceed with the real upgrade using
    the -y flag:

    $ ./upgrade.sh 1 -y

    EXAMPLE 3: Simplified

    If the standard SDP shell environment is loaded, upgrade.sh will be in
    the path, so the 'cd' command to /p4/common/bin is not needed. Also,
    the SDP_INSTANCE shell environment variable will be defined, so the
    'instance' parameter can be dropped, with simply a call to the script
    needed.  First do a preview:

    $ upgrade.sh

    Review the output of the preview, looking for the 'Success: Finished'
    message near the end of the output. If that exists, then execute again
    with the '-y' flag to execute the actual migration:

    $ upgrade.sh -y

CUSTOM PRE- AND POST- UPGRADE AUTOMATION HOOKS:
    This script can execute custom pre- and post- upgrade scripts. This
    can be useful to incorporate site-specifc elements of an upgrade.

    If the file /p4/common/site/upgrade/pre-upgrade.sh exists and is
    executable, it will be executed as a pre-upgrade script. If the file
    /p4/common/site/upgrade/post-upgrade.sh exists and is executable,
    it will be executed as a post-upgrade script.

    Pre- and post- upgrade scripts are called with an SDP instance
    parameter, and an optional '-y' flag to confirm actual processing is
    to be done.  Custom scripts are expected to operate in preview mode
    by default, taking no actions that affect data (just as this script
    behaves).  If this upgrade.sh script is given the '-y' flag, that
    option is passed to the custom script as well, indicating active
    processing should occur.

    Pre- and post- upgrade scripts are expected to exit with a zero exit
    code to indicate success, and non-zero to indicate failure.

    The custom pre-upgrade script is executed after standard preflight
    checks complete successfully.  If the '-I' flag is used to ignore the
    status of preflight checks, the custom pre-upgrade script is
    executed regardless of the status of preflight checks. Preflight
    checks are executed before actual upgrade processing commences. If a
    custom pre-upgrade script indicates a failure, the overall upgrade
    process aborts.

    The post-upgrade custom script is executed after the main upgrade
    is successful.

    Success or failure of pre- and post- upgrade scripts is reported in
    the log.  These scripts do not require independent logging, as all
    standard and error output is captured in the log of this upgrade.sh
    script.

    TIP: Be sure to fully test custom scripts in a test environment
    before incorporating them into an upgrade on production systems.

SEE ALSO:
    The /verify_sdp.sh script is used for preflight checks.

    The /p4/sdp/helix_binaries/get_helix_binaries.sh script acquires new binaries
    for upgrades.

    Both scripts sport the same '-h' (short help) and '-man' (full manual)
    usage options as this script.

LIMITATIONS:
    This script does not handle upgrades of 'p4dtg', Helix Swarm,
    Helix4Git, or any other software. It only handles upgrades of
    p4d, p4p, p4broker, and the p4 client binary on the SDP-managed
    server machine on which it is executed.

USAGE for sdp_upgrade.sh v2.2.0:

   sdp_upgrade.sh [-y] [-p] [-L <log>|off] [-D]

   or

sdp_upgrade.sh -h|-man

This script must be executed from the 'sdp_upgrade' directory in the extracted
SDP tarball, which can be in one of two locations.  If the /opt/perforce/helix-sdp/sdp
directory exists, start the upgrade process like this:

  cd /opt/perforce/helix-sdp/sdp/Server/Unix/p4/common/sdp_upgrade
  ./sdp_upgrade.sh

Otherwise, start the upgrade like this:

  cd /hxdepots/downloads/new/sdp/Server/Unix/p4/common/sdp_upgrade
  ./sdp_upgrade.sh

Running sdp_upgrade.sh without '-y' will do a dry run.  Review the output and check
for a SUCCESS indication at the end.  If you get the success message, proceedd
with:

  ./sdp_upgrade.sh -y


DESCRIPTION:

	This script upgrades Perforce Helix Server Deployment Package (SDP) from
	SDP 2020.1 to the version included in the latest SDP version, SDP
	2024.2.

	== Pre-Upgrade Planning ==

	This script will upgrade the SDP if the pre-upgrade starting SDP version
	is SDP 2020.1 or later, including any/all patches of SDP
	2020.1.

	If the current SDP version is older than 2020.1, it must first be upgraded
	to SDP 2020.1 using the SDP Legacy Upgrade Guide.  For upgrading from
	pre-20.1 versions dating back to 2007, in-place or migration-style upgrades
	can be done.  See:

	https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Legacy_Upgrades.Unix.html

	The SDP should always be upgraded to the latest version first before
	Helix Core binaries p4d/p4broker/p4p are upgraded using the SDP
	upgrade.sh script.

	Upgrading the SDP first ensures the version of the SDP you have
	is compatible with the latest versions of p4d/p4broker/p4p/p4, and
	will always be compatible with all supported versions of these
	Helix Core binaries.

	When this script is used, i.e. when the current SDP version is 2020.1
	or newer, the SDP upgrade procedure does not require downtime for any
	running Perforce Helix services, such as p4d, p4broker, or p4p.  This
	script is safe to run in environments where live p4d instances are running,
	and does not require p4d, p4broker, p4p, or any other services to be stopped
	or upgraded.  Upgrade of the SDP is cleanly separate from the upgrade the
	Helix Core binaries. The upgrade of the SDP can be done immediately prior to
	Helix Core upgrades, or many days prior.

	There can be multiple SDP instances on a given server machine. This script
	will upgrade the SDP on the machine, and thus after the upgrade all
	instances will immediately use new SDP scripts and updated instance
	configuration files, e.g. the /p4/common/config/p4_N.vars files.  However,
	all instances will continue running the same Helix Core binaries.  Any live
	running Helix Core server process on the machine are unaffected by the
	upgrade of SDP.

	This script will upgrade the SDP on a single machine. If your Perforce
	Helix topology has multiple machines, the SDP should be upgraded on all
	machines. The upgrade of SDP on multiple machines can be done in any order,
	as there is no cross-machine dependency requiring the SDP to be the same
	version.  (The order of upgrade of Helix Core services and binaries such as
	p4d in global topologies with replicas and edge servers does matter, but is
	outside the scope of this script).

	Planning Recap:
	1. The SDP can be upgraded without downtime when this script is used, i.e.
	when the starting SDP version is 2020.1 or later.
	2. Upgrade SDP on all machines, in any order, before upgrading p4d and other
	Helix binaries.

	== Diretory Structure Changes for /p4/sdp and /p4/common ==

	There is a structure change with SDP affecting where the /p4/sdp and
	/p4/common symlinks are targeted. This change is part of a phased
	rollout of a new structure to be used by a future helix-sdp OS package.
	Nothing in the structural changes affects behaviors of routine
	SDP daily scripts. The changes affect how SDP upgrades work, and on
	what volumes files like Helix Core server binaries exist on.  The gist
	of the change is that that /hxdepots/p4/commnon and /hxdepots/sdp
	folders (which are on NFS if /hxdepots is NFS-mounted) are changeed from
	being actively used folders to become backup directories,

	The active directories are moved to local storage on the machine
	in the new /opt/perforce/helix-sdp structure.

	Prior to SDP 2024.2, this was the typical symlink structure is:

	* /hxdepots/sdp, symlinked as /p4/sdp
	* /hxdepots/p4/common, symlinked as /p4/common
	* /hxdepots/downloads, no symlnk

	If /hxdepots is NFS mounted, then the active folders are on NFS.

	Starting with 2024.2, a new structure is available, and will be
	used if the install_sdp.sh script was used for the initial SDP
	install.  In that structure, we have the following:

	* /opt/perforce/helix-sdp/p4/sdp, symlinked as /p4/sdp
	* /opt/perforce/helix-sdp/p4/common, symlinked as /p4/common
	* /opt/perforce/helix-sdp/downloads, no symlink
	* /opt/perforce/helix-sdp/sdp, immutable root-owned structure,
	updated only by SDP upgrades (i.e. this script).

	During upgrades, the legacy structure is changed in ways that
	are safe even if /hxdepots is NFS-shared with machines not
	being upgraded. For safety, some structures are abandoned and
	untouched.

	* A new /hxdepots/sdp/backup folder is created, and contain
	/hxdepots/sdp/backup/opt/perforce/helix-sdp
	* Dirs other than backup under /helix/sdp are unusued/untouched
	* /hxdepots/p4/common is unused/untouched
	* /hxdepots/downloads is moved to /hxdepots/sdp/backup/downloads,
	and the /hxdepots/sdp becomes a symlink to /opt/perforce/helix-sdp/downloads.

	== NFS Sharing of HxDepots ==

	In some environments, the HxDepots volume is shared across multiple server
	machines with NFS, typically mounted as /hxdepots.  This script updates the
	/hxdepots/p4/common and /hxdepots/sdp directories, both of which are on the
	NFS mount.  Thus upgrading SDP on a single machine will effectively and
	immediately upgrade the SDP on all machines that share /hxdepots from the
	same NFS-mounted storage. This is a safe and valid configuration, as
	upgrading the SDP does not affect any live running p4d servers.

	== Acquiring the SDP Package - OS Package Structure ==

	If the /opt/perforce/helix-sdp structure exists on your machine, then
	upgrade using the procedure in this section.  Otherwise see the section
	below "Acquiring the SDP Package - Classic Structure".

	Become the root user first:
	  sudo su -
	  cd /opt/perforce/helix-sdp/downloads
	  [[ -e sdp.Unix.tgz ]] && mv -f sdp.Unix.tgz sdp.Unix.$(date +'%Y-%m-%d-%H%M%S').tgz
	  curl -L -O https://swarm.workshop.perforce.com/download/guest/perforce_software/sdp/downloads/sdp.Unix.tgz
	  tar -tzf sdp.Unix.tgz 2>&1 | grep -q sdp/Version && echo OK

	If this does not display an OK message, the tarball is not valid. Investigate
	and resolve this issue before proceeding. As root:

	  cd /opt/perforce/helix-sdp
	  [[ -d backup ]] || mkdir backup
	  mv sdp backup/sdp.old.$(date +'%Y-%m-%d-%H%M%S')
	  tar -xzf downloads/sdp.Unix.tgz

	  cd /opt/perforce/helix-sdp/sdp/Server/Unix/p4/common/sdp_upgrade
	  ./sdp_upgrade.sh -man

	== Acquiring the SDP Package - Classic Structure ==

	If the /opt/perforce/helix-sdp structure exists on your machine, then
	upgrade using the procedure above in the OS Package Structure section.

	This script is part of the SDP package (tarball). It must be run from an
	extracted tarball directory.  Acquiring the SDP tarball is a manual operation.

	The SDP tarball must be extracted such that the 'sdp' directory appears as
	<HxDepots>/downloads/new/sdp, where <HxDepots> defaults to /hxdepots. To
	determine the value for <HxDepots> at your site you can run the following:

     bash -c 'cd /p4/common; d=$(pwd -P); echo ${d%/p4/common}'

   On this machine, that value is: /hxdepots

	Following are sample commands to acquire the latest SDP, to be executed
	as the user perforce:

	  cd /hxdepots
	  [[ -d downloads ]] || mkdir downloads
	  cd downloads
	  [[ -d new ]] && mv new old.$(date +'%Y%m%d-%H%M')
	  curl -L -O https://swarm.workshop.perforce.com/download/guest/perforce_software/sdp/downloads/sdp.Unix.tgz
	  ls -l sdp.Unix.tgz
	  mkdir new
	  cd new
	  tar -xzf ../sdp.Unix.tgz

	After extracting the SDP tarball, cd to the directory where this
	sdp_upgrade.sh script resides, and execute it from there.

	  cd /hxdepots/downloads/new/sdp/Server/Unix/p4/common/sdp_upgrade
	  ./sdp_upgrade.sh -man

	== Preflight Checks ==

	Prior to upgrading, preflight checks are performed to ensure the
	upgrade can be completed successfully. If the preflight checks
	fail, the upgrade will not start.

	Sample Preflight Checks:

	* The existing SDP version is verified to be SDP 2020.1+.
	* Various basic SDP structural checks are done.
	* The /p4/common/bin/p4_vars is checked to confirm it can be upgraded.
	* All /p4/common/config/p4_N.vars files are checked to confirm they can be upgraded.

	== Automated Upgrade Processing ==

	Step 1: Backup /p4/common.

	The existing <HxDepots>/p4/common structure is backed up to:

	<HxDepots>/p4/common.bak.<YYYYMMDD-hhmm>

	Step 2: Update /p4/common.

	The existing SDP /p4/common structure is updated with new
	versions of SDP files.

	Step 3: Generate the SDP Environment File.

	Regenerate the SDP general environment file,
	/p4/common/bin/p4_vars.

	The template is /p4/common/config/p4_vars.template.

	Step 4: Generate the SDP Instance Files.

	Regenerate the SDP instance environment files for all instances based on
	the new template.

	The template is /p4/common/config/instance_vars.template.

	For Steps 3 and 4, the re-generation logic will preserve current
	settings. If upgrading from SDP r20.1, any custom logic that
	exists below the '### MAKE LOCAL CHANGES HERE' tag will be
	split into separate files.  Custom logic in p4_vars will be moved
	to /p4/common/site/config/p4_vars.local. Custom logic in
	p4_N.vars files will be moved to /p4/common/site/config/p4_N.vars.local.

	Note: Despite these changes, the mechanism for loading the SDP shell
	environment remains unchanged since 2007, so it looks like:

	$ source /p4/common/bin/p4_vars N

	Changes to the right-side of assignments for specific are preserved
	for all defined SDP settings.  For p4_vars, preserved settings are:
	  - OSUSER (determined by current owner of /p4/common)
	  - KEEPLOGS
	  - KEEPCKPS
	  - KEEPJNLS

	For instance_vars files, preserved settings are:
	  - MAILTO
	  - MAILFROM
	  - P4USER
	  - P4MASTER_ID
	  - SSL_PREFIX
	  - P4PORTNUM
	  - P4BROKERPORTNUM
	  - P4MASTERHOST
	  - PROXY_TARGET
	  - PROXY_PORT
	  - PROXY_MON_LEVEL
	  - PROXY_V_FLAGS
	  - P4DTG_CFG
	  - SNAPSHOT_SCRIPT
	  - SDP_ALWAYS_LOGIN
	  - SDP_AUTOMATION_USERS
	  - SDP_MAX_START_DELAY_P4D
	  - SDP_MAX_START_DELAY_P4BROKER
	  - SDP_MAX_START_DELAY_P4P
	  - SDP_MAX_STOP_DELAY_P4D
	  - SDP_MAX_STOP_DELAY_P4BROKER
	  - SDP_MAX_STOP_DELAY_P4P
	  - VERIFY_SDP_SKIP_TEST_LIST
	  - The 'umask' setting.
	  - KEEPLOGS (if set)
	  - KEEPCKPS (if set)
	  - KEEPJNLS (if set)

	Note that the above list excludes any values that are calculated.

	Step 5: Remove Deprecated Files.

	Deprecated files will be purged from the SDP structure.  The list of
	files to be cleaned are listed in this file:

	/downloads/new/sdp/Server/Unix/p4/common/sdp_upgrade/deprecated_files.txt

	Paths listed in this file are relative to the '/p4' directory (or
	more accurately the SDP Install Root directory, which is always
	'/p4' except in SDP test production environments).

	Step 6: Update SDP crontabs.

	No crontab updates are required for this SDP upgrade.

	== Post-Upgrade Processing ==

	This script provides guidance on any post-processing steps. For some
	releases, this may include upgrades to crontabs.

OPTIONS:
-y	Specify the '-y' option to confirm that the SDP upgrade should be done.

	By default, this script operates in No-Op mode, meaning no actions
	that affect data or structures are taken.  Instead, commands that would
	be run are displayed.  This mode can be educational, showing various
	steps that will occur during an actual upgrade.

 -p	Specify '-p' to halt processing after preflight checks are complete,
	and before actual processing starts. By default, processing starts
	immediately upon successful completion of preflight checks.

 -Od	Specify '-Od' to override the rule preventing downgrades.

	WARNING: This is an advanced option intended for use by or with the
	guidance of Perforce Support or Perforce Consulting.

 -L <log>
	Specify the log file to use.  The default is /tmp/sdp_upgrade.<timestamp>.log

	The special value 'off' disables logging to a file. This cannot be
	specified if '-y' is used.

 -d	Enable debugging verbosity.

 -D	Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

FILES AND DIRECTORIES:
	Name: SDPCommon
	Path: /p4/common
	Notes: This sdp_upgrade.sh script updates files in and under this folder.

	Name: HxDepots
	Default Path: /hxdepots
	Notes: The folder containing versioned files, checkpoints, and numbered
	journals, and the SDP itself. This is commonly a mount point.

	Name: DownloadsDir
	Default Path: /hxdepots/downloads

	Name: SDPInstallRoot
	Path: /p4

	EDITME - Add new structure dirs /opt/perforce/helix-sdp

EXAMPLES:
	This script must be executed from 'sdp_upgrade' directory in the extracted
	SDP tarball.  Typical operation starts like this:

	cd /hxdepots/downloads/new/sdp/Server/Unix/p4/common/sdp_upgrade
	./sdp_upgrade.sh -h

	All following examples assume operation from that directory.

	Example 1: Prelight check only:

	sdp_upgrade.sh -p

	Example 2: Preview mode:

	sdp_upgrade.sh

	Example 3: Live operation:

	sdp_upgrade.sh -y

LOGGING:
	This script generates a log file, ~/sdp_upgrade.<timestamp>.log
	by default. See the '-L' option above.

CUSTOM PRE- AND POST- UPGRADE AUTOMATION HOOKS:
	This script can execute custom pre- and post- upgrade scripts. This
	can be useful to incorporate site-specific elements of an SDP upgrade.

	If the file /p4/common/site/upgrade/pre-sdp_upgrade.sh exists and is
	executable, it will be executed as a pre-upgrade script. If the file
	/p4/common/site/upgrade/post-sdp_upgrade.sh exists and is executable,
	it will be executed as a post-upgrade script.

	Pre- and post- upgrade scripts are passed the '-y' flag to confirm
	actual processing is to be done.  Custom scripts are expected to
	operate in preview mode by default, taking no actions that affect data
	(just as this script behaves).  If this sdp_upgrade.sh script is given
	the '-y' flag, that option is passed to the custom script as well,
	indicating active processing should occur.

	Pre- and post- upgrade scripts are expected to exit with a zero exit
	code to indicate success, and non-zero to indicate failure.
	The custom pre-upgrade script is executed after standard preflight
	checks complete successfully.  Preflight checks are executed before
	actual upgrade processing commences. If a custom pre-upgrade script
	indicates a failure, the overall upgrade process aborts.

	The post-upgrade custom script is executed after the main SDP upgrade
	is successful.

	Success or failure of pre- and post- upgrade scripts is reported in
	the log.  These scripts do not require independent logging, as all
	standard and error output is captured in the log of this sdp_upgrade.sh
	script.

	TIP: Be sure to fully test custom scripts in a test environment
	before incorporating them into an upgrade on production systems.

EXIT CODES:
	An exit code of 0 indicates no errors were encountered during the
	upgrade. A non-zero exit code indicates the upgrade was aborted
	or failed.

USAGE for clear_depot_Map_fields.sh v1.2.0:

clear_depot_Map_fields.sh [-i <instance>] [-L <log>] [-v<n>] [-p|-n] [-D]

or

clear_depot_Map_fields.sh [-h|-man|-V]


DESCRIPTION:
	This script obsoletes the SetDefaultDepotSpecMapField.py trigger.

	It does so by following a series of steps.  First, it ensures that
	the configurable server.depot.root is set correctly, setting it
	if it is not already set.

	Next, the Triggers table is checked to ensure the call to the
	SetDefaultDepotSpecMapField.py is not called; it is deleted from
	the Triggers table if found.
	Last, it resets the 'Map:' field of depot specs for depot
	types where that is appropriate, setting it to the default value of
	'<DepotName>/...', so that it honors the server.depot.root
	configurable.  This is done for depots of these types:

	* stream
	* local
	* spec
	* unload
	* graph

	but not these:
	* archive
	* remote

	If an unknown depot type is encountered, the 'Map:' field is reset
	as well if it is set.

	This script does a preflight check first, reporting any cases
	where the starting conditions are not as expected.  These conditions
	are treated as Errors, and will abort processing:

	* Depot Map field set to something other than the default.
	* Configurable server.depot.root is set, but to something other
	than what it should be.

	The following are treated as Warnings, and will be reported but
	will not prevent processing.

	* Configurable server.depot.root is already set.
	* SetDefaultDepotSpecMapField.py not found in triggers.
	* Depot already has 'Map:' field set to the default value:
	<DepotName>/...

OPTIONS:
 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) goes to
	EDITME_DEFAULT_LOG

	NOTE: This script is self-logging.  That is, output displayed on the screen
	is simultaneously captured in the log file.  Do not run this script with
	redirection operators like '> log' or '2>&1', and do not use 'tee.'

 -p	Run preflight checks only, and then stop. By default, actual changes
	occur if preflight checks find no issues.

 -n	No-Op.  No actions are taken that would affect data significantly;
	instead commands are displayed rather than executed.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Display version info for this script and its libraries.

EXAMPLES:
	A typical flow for this script is to do a preflight first, and then
	a live run, for any given instance:
	clear_depot_Map_fields.sh -i 1 -p
	clear_depot_Map_fields.sh -i 1

	Note that if using '-n', the '-v5' flag should also be used.

USAGE for mkrep.sh v3.4.2:

mkrep.sh -t <Type> -s <Site_Tag> -r <Replica_Host> [-f <From_ServerID>] [-os] [-p] [-p4config <PathToFile>] [-N <N>] [-i <SDP_Instance>] [-L <log>] [-v<n>] [-n] [-D]

or

mkrep.sh [-h|-man|-V]


DESCRIPTION:
	This script simplifies the task of creating Helix Core replicas and
	edge servers, and helps ensure they are setup with best practices.

	This script executes as two phases. In Phase 1, this script does all
	the metadata configuration to be executed on the master server that
	must be baked into a seed checkpoint for creating the replica/edge.
	This essentially captures the planning for a new replica, and can be
	done before the physical infrastructure (e.g. hardware, storage, and
	networking) is ready. Phase 1, fully automated by this script, takes
	only seconds to run.

	In Phase 2, this script provides information for the manual steps
	needed to create, transfer, and load seed checkpoints onto the
	replica/edge. The guidance is specific to type of replica created,
	based on the command line flags provided to this script. This
	processing can take a while for large data sets, as it involves
	creating and transporting checkpoints.

	Before using this script, a set of geographic site tags must be defined.
	See the FILES: below for details on a site tags.

	This script adheres to the these SDP Standards:
	* Server Spec Naming Standard: https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.html#_server_spec_naming_standard
	* Journal Prefix Standard: https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.html#_the_journalprefix_standard

	In Phase 1, this script does the following to help create a replica or
	edge server:
	* Generates the server spec for the new replica.
	* Generates a server spec for master server (if needed).
	* Sets configurables ('p4 configure' settings) for replication.
	* Selects the correct 'Services' based on replica type.
	* Creates service user for the replica, and sets a password.
	* Creates service user for the master (if needed), and sets a password.
	* Adds newly created service user(s) to the group 'ServiceUsers'.
	* Verifies the group ServiceUsers is granted super access in the
	protections table (and with '-p', also updates Protections).

	After these steps are completed, in Phase 2, detailed instructions are
	presented to guide the user through the remaining steps needed to complete
	the deployment of the replica.  This starts with creating a new
	checkpoint to capture all the metadata changes made by this
	script in Phase 1.

SERVICE USERS:
	Service users created by this script are always of type 'service',
	and so will not consume a licensed seat.

	Service users also have an 'AuthMethod' of 'perforce' (not
	'ldap') as is required by 'p4d' for 'service' users.  Passwords
	set for service users are long 32 character random strings
	that are not stored, as they are never needed.  Login tickets for
	service users are generated using: p4login -service -v

OPTIONS:
 -t <Type>[N]
	Specify the replica type tag. The type corresponds to the 'Type:' and
	'Services:' field of the server spec, which describes the type of services
	offered by a given replica.

	Valid type values are:
	* ha:   High Availability standby replica, for 'p4 failover' (P4D 2018.2+)
	* ham:  High Availability metadata-only standby replica, for 'p4 failover' (P4D 2018.2+)
	* ro:   Read-Only standby replica. (Discouraged; Use 'ha' instead for 'p4 failover' support.)
	* rom:  Read-Only standby replica, Metadata only. (Discouraged; Use 'ham' instead for 'p4 failover' support.)
	* fr:   Forwarding Replica (Unfiltered).
	* fs:   Forwarding Standby (Unfiltered).
	* frm:  Forwarding Replica (Unfiltered, Metadata only).
	* fsm:  Forwarding Standby (Unfiltered, Metadata only).
	* ffr:  Filtered Forwarding Replica.  Not a valid failover target.
	* edge: Edge Server. Filtered by definition.

	Replicas with 'standby' are always unfiltered, and use the 'journalcopy'
	method of replication, which copies a byte-for-byte verbatim journal file
	rather than one that is merely logically equivalent.

	The tag has several purposes:
	1. Short Hand. Each tag represents a combination of 'Type:' and fully
	qualified 'Services:' values used in server specs.

	2. Distillation. Only the most useful Type/Services combinations have a
	shorthand form

	3. For forwarding replicas, the name includes the critical distinction of
	whether any replication filtering is used; as filtering of any kind disqualifies
	a replica from being a potential failover target. (No such distinction is
	needed for edge servers, which are filtered by definition).

 -s <Site_Tag>
	Specify a geographic site tag indicating the location and/or data center where
	the replica will physically be located. Valid site tags are defined in the site
	tags file:

	/p4/common/config/SiteTags.cfg

	A sample SiteTags.cfg file that is here:

	/p4/common/config/SiteTags.cfg.sample

 -r <Replica_Host>
	Specify the DNS name of the server machine on which the new replica will
	run. This is used in the 'ExternalAddress:' field of the replica's
	ServerID, and also used in instructions to the user for steps after
	metadata configuration is done by this script.

 -f <From_ServerID>
	Specify ServerID of the P4TARGET server from which we are replicating.
	This is used to populate the 'ReplicatingFrom' field of the server
	spec. The value must be a valid ServerID.

	This option should be used if the target is something other than the
	master. For example, to create an HA replica of an edge server, you might
	specify something like '-f p4d_edge_syd'.

 -os	Specify the '-os' option to overwrite an exising server spec.  By
	default, this script will abort of the server spec to be generated
	already exists on the Helix Core server.  Specify this option to
	overwrite the existing server spec.

 -p	This script always performs a check to ensure that the Protections table
	grants super access to the group ServiceUsers.

	By default, an error is displayed if the check fails, i.e. if super user
	access for the group ServiceUsers cannot be verified. This is
	because, by default, we want to avoid making changes to the Protections
	table. Some sites have local policies or custom automation that requires
	site-specific procedures to update the Protections table.

	If '-p' is specified, an attempt is made to append the Protections table
	an entry like:

	super group ServiceUsers * //...

	This option may not be suitable for use on servers that have custom
	automation managing the Protections table.

 -p4config <PathToFile>
	Use the '-p4config' option use this SDP mkrep.sh script to create a
	replica spec on an arbitrary p4d server.  That arbitrary server can be
	any p4d version, operating on any platform, and need not be managed with SDP.

	To use this option, first create a P4CONFIG file that defines settings
	needed to access the other server.  As a convention, identify a short tag
	name for the other server to use in the P4CONFIG file.  In the example
	below, we use 'mot' for "my other server".  Create a P4CONFIG file text
	named /p4/common/site/config/.p4config.mot that contains these settings:

	P4PORT=ssl:my_other_server:1666
	P4USER=p4admin
	P4TICKETS=/p4/common/site/config/.p4tickets.mot
	P4TRUST=/p4/common/site/config/.p4trust.mot

	The P4TRUST setting is only needed if the port is SSL-enabled. If it
	is enabled, next trust the port:

	p4 -E P4CONFIG=/p4/common/site/config/.p4config.mot trust -y

	Next, generate a ticket on that connection:
	p4 -E P4CONFIG=/p4/common/site/config/.p4config.mot login -a

	Provide the password if prompted.

	Finally, call mkrep.sh and specify the config file.  When using this
	option, using '-L' to specify a non-default log file name
	is useful to keep logs from external servers cleanly separated.

 	mkrep.sh -p4config /p4/common/site/config/.p4config.mot -L /mkrep.mot.log

	This will run the mkrep against the server specify in that P4CONFIG
	file.


-N <N>
	Specify '-N <N>', where N is an integer.  This is used to indicate that
	multiple replicas of the same type are to be created at the same site.
	The value specified with '-N' must be a numeric value. Left-padding with
	zeroes is allowed. For example, '-N 04' is allowed, and 'N A7' is not
	(as it is not numeric).

	This affects the ServerID to be generated.  For example, the options
	'-t edge -s syd' would result in a ServerID of p4d_edge_syd.  To
	create a second edge in the same site, use '-t edge -s syd -N 2' to
	generate p4d_edge2_syd.

 -i <SDP_Instance>
	Specify the SDP Instance. If not specified and the SDP_INSTANCE environment
	is defined, that value is used. If SDP_INSTANCE is not defined, the
	'-i <SDP_Instance>' argument is required.

 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging. By default, all output (stdout and stderr) goes in the logs
	directory referenced by $LOGS environment variable, in a file named
	mkrep.<timestamp>.log

	NOTE: This script is self-logging. That is, output displayed on the screen
	is simultaneously captured in the log file. Using redirection operators like
	'> log' or '2>&1' are not necessary, nor is using 'tee.'

 -n	No-Op. Prints commands instead of running them.

 -D	Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Display version info for this script and its libraries.

FILES:
	This Site Tags file defines the list of valid geographic site tags:
	/p4/common/config/SiteTags.cfg

	The contains one-line entries of the form:

	<tag>: <description>

	where <tag> is a short alphanumeric tag name for a geographic location,
	data center, or other useful distinction. This tag is incorporated into
	the ServerID of replicas or edge servers created by this script.  Tag
	names should be kept short, ideally no more than about 5 characters in
	length.

	The <description> is a one-line text description of what the tag
	refers to, which may contain spaces and ASCII punctuation.

	Blank lines and lines starting with a '#' are considered comments
	and are ignored.

REPLICA SERVER MACHINE SETUP:
	The replica/edge server machine must be have the SDP structure installed,
	either using the mkdirs.sh script included in the SDP, or the Helix
	Installer for 'green field' installations.

	When setting up an edge server, a replica of an edge server, or filtered
	replica, confirm that the JournaPrefix Standard (see URL above) structure
	has the separate checkpoints folder as identified in the 'Second Form' in
	the standard.  A baseline SDP structure can typically be extended by running
	commands like like these samples (assuming a ServerID of p4d_edge_syd or
	p4d_ha_edge_syd):

	   mkdir /hxdepots/p4/1/checkpoints.edge_syd
	   cd /p4/1
	   ln -s /hxdepots/p4/1/checkpoints.edge_syd

CUSTOM PRE- AND POST- OPERATION AUTOMATION HOOKS:
	This script can execute custom pre- and post- processing scripts. This
	can be useful to incorporate site-specific elements of replica setup.

	If the file /p4/common/site/mkrep/pre-mkrep.sh exists and is
	executable, it will be executed before mkrep.sh processing. If the file
	/p4/common/site/mkrep/post-mkrep.sh exists and is executable,
	it will be executed after mkrep.sh processing.

	Pre- and post- processing scripts are called with the same command line
	arguments passed to this mkrep.sh script.

	The pre- and post- processing scripts can use or ignore arguments as
	needed, though it is required to implement the '-n' flag to operate in
	preview mode, taking no actions that affect data (just as this script
	behaves).

	Pre- and post- processing scripts are expected to exit with a zero exit
	code to indicate success, and non-zero to indicate failure.

	The custom pre-processing script is executed after standard preflight
	checks complete successfully.  If a custom pre-processing script
	indicates a failure, processing is aborted before standard mkrep.sh
	processing occurs.

	The post-processing custom script is executed after the standard
	mkrep.sh processing is successful.  If a post-processing custom script
	is detected, the instructions that would be provided to the user in
	Phase 2 are not displayed, as it is expected that the custom post-
	processing will alter or handle these steps.

	Success or failure of pre- and post- processing scripts is reported in
	the log.  These scripts do not require independent logging, as all
	standard and error output is captured in the log of this mkrep.sh
	script.

	TIP: Be sure to fully test custom scripts in a test environment
	before incorporating them into production systems.

EXAMPLES:
	EXAMPLE 1 - Set up a High Availability (HA) Replica of the master.

	Add an HA replica to instance 1 to run on host bos-helix-02:
	mkrep.sh -i 1 -t ha -s bos -r bos-helix-02

	EXAMPLE 2 - Add an Edge Server to the topology.

	Add an Edge server to instance acme to run on host syd-helix-04:

	mkrep.sh -i acme -t edge -s syd -r syd-helix-04

	EXAMPLE 3 - Setup an HA replica of an edge server.

	Add a HA replica of the edge server to instance acme to run on host syd-helix-05:

	mkrep.sh -i acme -t ha -f p4d_edge_syd -s syd -r syd-helix-05

	EXAMPLE 4 - Add a second edge server in the same site as another edge.

	mkrep.sh -i acme -t edge -N 2 -s syd -r syd-helix-04

	EXAMPLE 5 - Set up a High Availability (HA) Replica of the master *from* the
	replica server via -p4config.

	Add an HA replica to instance 1 to run on host bos-helix-02:
	mkrep.sh -i 1 -t ha -s bos -r bos-helix-02 -p4config /p4/common/site/config/.p4config.mot -L /mkrep.mot.log

USAGE for  v5.20.1:

p4verify.sh [<instance>] [-N] [-nu] [-nr] [-ns] [-nS] [-a] [-nt] [-nz] [-no_z] [-o BAD|MISSING] [-p4config <PathToFile>] [-chunks <ChunkSize>|-paths <paths_file>] [-w <Wait>] [-q <MaxActivePullQueueSize>] [-Q MaxTotalPullQueueSize] [-recent | -recent=N] [-dlf <depot_list_file>] [-I|-ignores <regex_ignores_file>] [-Ocache] [-n] [-L <log>] [-v] [-d] [-D]

   or

p4verify.sh -h|-man

DESCRIPTION:

	This script performs a 'p4 verify' of all submitted and shelved versioned
	files in depots of all types except 'remote' and 'archive' type depots.

	The singular Extensions depot is also verified, if present.

	The singular Traits depot is also verified, if present.

	Archive depots are not verified by defefault, but can be with the '-a'
	option.

	If run on a replica, it schedules archive failures for transfer to the
	replica.

OPTIONS:
<instance>
	Specify the SDP instance.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined, p4verify.sh
	exists immediately with an error message.

 -N	Specify '-N' (Notify Only On Failure) to disable the default behavior
	which will always send a notification which includes a report of the p4
	verify status.  Specifying '-N' which change the behavior to only send
	a notification if there is an error during the p4 verify execution.
	Notification methods are email, AWS SNS, and PagerDuty. Details on
	configuration can be found in the SDP documentation.  Providing the
	environment variable NOTIFY_ONLY_ON_FAILURE=1 is equivalent to the '-N'
	command line argument.

 -nu	Specify '-nu' (No Unload) to skip verification of the singleton depot
	of type 'unload' (if created).  The 'unload' depot is verified
	by default.

 -nr	Specify '-nr' (No Regular) to skip verification of regular submitted
	archive files.  The '-nr' option is not compatible with '-recent'.
	Regular submitted archive files are verified by default.

	This option also causes Extension and Traits depots (if present)
	not to be verified.

 -ns	Specify '-ns' (No Spec Depot) to skip verification of singleton depot
	of type 'spec' (if created). The 'spec' depot is verified by default.

 -nS	Specify '-nS' (No Shelves) to skip verification of shelved archive
	files, i.e. to skip the 'p4 verify -qS'.

 -a	Specify '-a' (Archive Depots) to do verification of depots of type
	'archive'.  Depots of type 'archive' are not verified by default, as
	archive depots are often physically removed from the server's
	storage subsystem for long-term cold storage.

-nt	Specify the '-nt' option to avoid passing the '-t' flag to 'p4 verify'
	on a replica. By default, p4verify.sh detects if it is running on a
	replica, and if so automatically applies the '-t' flag to 'p4 verify'.
	That causes the replica to attempt to self-heal, as files that fail
	verification are scheduled for transfer from the P4TARGET server. This
	default behavior results in 'Transfer scheduled' messages in the log,
	and MISSING/BAD files are listed as 'info:' rather than 'error:'. There
	is no clear indication of whether or which of the scheduled transfers
	complete successfully, and so there may be a mix of transient/correctable
	and "real"/persistent transfer errors for files that are also BAD/MISSING
	on the master server.  Specify '-nt' to ensure the log contains a list
	of files that currently fail a 'p4 verify' without attempting to transfer
	them from the master.

 -nz	Specify '-nt' to skip the gzip of the old log file.  By default, if a
	log with the default name or the name specified with '-L' exists at the
	start of processing, the old log is rotated and gzipped.  With this option
	the old log is not zipped when rotated.

 -no_z	Specify '-no_z' to avoid passing the '-z' option 'p4 verify' commands.
	Typically, verifies are done with '-qz'; with this option, '-q' is used
	instead.

	See 'p4 help verify' for more information.

 -o BAD|MISSING
	Specify '-o MISSING' to check only whether expected archive files exist
	or not, skipping the checksum calculation of existing files.  This results
	in dramatically faster, if less comprehensive, verification.  This
	is particularly well suited when verification is being used to schedule
	archive file transfers of missing files on replicas.  This translates into
	passing the '--only MISSING' option to 'p4 verify'.

	Specify '-o BAD' to check only for BAD revisions.  This translates into
	passing the '--only BAD' option to 'p4 verify'.

	This option requires p4d to be 2021.1 or newer. For older p4d versions,
	this option is silently ignored.

 -p4config <PathToFile>
	Use the '-p4config' option use this SDP p4verify.sh script to verify
	an arbitrary p4d server.  That arbitrary server can be any p4d version,
	operating on any platform, and need not be managed with SDP.

	To use this option, first create a P4CONFIG file that defines settings
	needed to access the other server.  As a convention, identify a short tag
	name for the other server to use in the P4CONFIG file.  In the example
	below, we use 'mot' for "my other server".  Create a P4CONFIG file text
	named /p4/common/site/config/.p4config.mot that contains these settings:

	P4PORT=ssl:my_other_server:1666
	P4USER=p4admin
	P4TICKETS=/p4/common/site/config/.p4tickets.mot
	P4TRUST=/p4/common/site/config/.p4trust.mot

	The P4TRUST setting is only needed if the port is SSL-enabled. If it
	is enabled, next trust the port:

	p4 -E P4CONFIG=/p4/common/site/config/.p4config.mot trust -y

	Next, generate a ticket on that connection:
	p4 -E P4CONFIG=/p4/common/site/config/.p4config.mot login -a

	Provide the password if prompted.

	Finally, call p4verify.sh and specify the config file.  When using this
	option, using '-L' to specify a non-default log file name
	is useful to keep logs from external servers cleanly separated.

 	p4verify.sh -p4config /p4/common/site/config/.p4config.mot -L /p4verify.mot.log

	This will run the varify against the server specify in that P4CONFIG
	file.

 -chunks <ChunkSize>
	Specify the maximum amount of content by size to verify at once.  If
	this is specified, the depot_verify_chunks.py script is used to
	break up depots into chunks of a given size, e.g. 100M or 4G.

	The <ChunkSize> parameter must be a size value valid to pass to the
	depot_verify_chunks.py script with the '-m' option.  That is,
	specifying '-chunks 200M' translates to calling depot_chunks_verify.sh
	with '-m 200M'.

	This requires the perforce-p4python3 module to be installed and the
	python3 in the PATH must be the correct one that uses the P4 module.

	Using '-chunks' is likely to result in a significantlly slower overall
	verify operation, though chuking can make it less impactful when it
	runs.  Using the '-chunks' option may be necessary on very large data
	sets, e.g. if there insufficient resources to process the largest
	depots.

	The '-recent' and '-chunks' options are mutually exclusive.

	The '-chunks' and '-paths' options can be used together; see the
	description of the '-paths' option below.

	Chunking logic applies only in depots of type 'stream' or 'local'.

 -paths <paths_file>

	Specify a file containing a list of depot paths to verify, with one
	line per entry.  Valid entries in the file start with '//', e.g.

	//mydepot/main/src/...

	In this example, when //mydepot depot is processed, only specified
	paths will be verified.  All other depots will be processed in full.
	To verify only specified paths, combine '-paths <paths_file>' with
	'-dlf <depot_list_file>' where the depot list file contains only
	'mydepot' (per the example above).

	The '-chunks' and '-paths' options can be used together for combined
	effects. If both options are specified, depots that contain specified
	paths are chunked based on the specified paths rather than the entire
	depot, and other paths in that depot are not processed.  Depots that
	do not have any specified paths listed in the <paths_file> are
	chunked at the top/depot level directory.

	The '-paths' option can be combined with '-recent' to verify only
	recent changelists in the specified paths.

	This option disables processing of the Extensions and Traits
	depots by default, though '-paths' can specify paths in those depots.

	Paths specified must be in depots of type 'stream' or 'local', or
	the singular Extensions or Traits depots.

 -w <Wait>
	Specify the '-w' option, where <Wait> is a positive integer
	indicating the number of seconds to sleep between individual calls
	to 'p4 verify' commands.  For example, specifying '-w 300' results
	in a delay of 5 minutes between verify commands.

	This can be used with '-chunks' to inject a delay between chunked
	depot paths. Otherwise, the delay is injected between each depot
	processed. This can significantly lengthen the overall duration
	of 'p4verify.sh' operation, but can also spread out the resource
	consumption load on a server machine.

	If shelves are procossed (regardless of whether '-chunks' is used),
	the delay is injected between each individual shelved changelist, as
	shelved changes are verified one changelist at a time.  For data sets
	with a large number of shelves, it may be be wise to process shelves
	separately from submitted files if '-w' is used, a delay value to
	apply between depots may be different from that applied to
	individual changelists.

	See the '-q' option for a description of how '-q' and '-w' can be
	used together.

-q <MaxActivePullQueueSize>
	Specify the '-q' option, where <MaxActivePullQueueSize> is a positive
	integer indicating the maximum number of active pulls allowed before
	a 'p4 verify' command will be executed to transfer archives.

	The absolute maximum number of possible active pulls is affected by
	the number of 'startup.N' threads configured to pull archives files,
	and whether those threads indicate batching.

	The threads that pull archive files are those that configured to use
	the 'pull' command the '-u' option. Typically, a small number of pull
	threads are configured, between 2 and 10 or perhaps 20.

	If '-q 1' is specified, new 'p4 verify' commands will only be run
	when the active pull queue is quiet.  Specifying a too-high value,
	e.g. '-q 50' if only 3 'pull -u' archive pull threads are configured,
	will be ineffective, as the active pull threads will never exceed
	3 (let alone 50).

	The current active pull queue on a replica is reported by:

	p4 -ztag -F %replicaTransfersActive% pull -ls

	This option can be useful if using this p4verify.sh script to pull
	many or even all archives on a new replica server machine from its
	target server.  The injected delays can give the server time to transfer
	archives scheduled in one call to 'p4 verify' before calling it again
	with the goal of avoidng overloading the pull queue.

	If '-w' and '-q' options are both used, the delay specified by '-w'
	is ignored unless the active pull queue size is greater than or equal
	to the specified maximum active pull queue size.  The '-w' then
	essentially determines how frequently the 'p4 pull -ls' is run to
	check the active pull queue size.  A reasonable set of values might
	be '-q 1 -w 10'.

	The '-q' option in mutually exclusive with '-nt'.

	The '-q' option in mutually exclusive with '-Q'.

-Q <MaxTotalPullQueueSize>
	Specify the '-Q' option, where <MaxTotalPullQueueSize> is a positive
	integer indicating the maximum number of total pulls allowed before
	a 'p4 verify' command will be executed to transfer archives.

	In certain scenarios, the pull queue can become quite massive. For
	example, if a fresh standby replica is seeded from a checkpoint
	but has no archive files, and then a 'p4verify.sh' is run, the
	verify will schedule all files to be transferred, perhaps millions.

	If the pull queue gets too large, it can impact metadata replication.
	Setting this value may help mitigate issues related to scheduling
	too many archives pulls at once, by delaying scheduling new archive
	pulls until enough previously scheduled pulls are completed.

	This option can be useful in such scenarios, if this p4verify.sh script
	is used to pull many or even all archives on a new replica server machine
	from its target server.  The injected delays can give the server time to
	transfer archives scheduled in one call to 'p4 verify' before calling it
	again with the goal of avoidng overloading the pull queue.

	If individual depots contain large numbers of files, such that
	a verify on a single depot will schedule too many files to be
	transferred at once, it may be necessary to combine this option with
	the '-chunks' option to avoid overloading the transfer queue.

	**WARNING**: If there are files that cannot be tranferred from the
	replica's target server, the value of '-Q' must be set to higher than
	that number, or an infinite loop may occur.  For example, if there are
	500 permanent "legacy" verify errors on the commit server from 10
	years ago that have long since been abandoned, those files can never
	be transferred to any replica. Running p4verify.sh on the replica will
	cause those files to be scheduled, but as they cannot be pulled, they
	will land in the total pull queue.  In this scenario, the value set
	with '-Q' must be greater than 500, or an infinite loop is possible.

	Specify '-Q 0' to disable checking the total pull queue.

	The current total pull queue on a replica is reported by:

	p4 -ztag -F %replicaTransfersTotal% pull -ls

	This option can be useful if using this p4verify.sh script to pull
	many or even all archives on a new replica server machine from its
	target server.  The injected delays can give the server time to transfer
	archives scheduled in one call to 'p4 verify' before calling it again
	with the goal of avoidng overloading the pull queue.

	If '-w' and '-Q' options are both used, the delay specified by '-w'
	is ignored unless the total pull queue size is greater than or equal
	to the specified maximum total pull queue size.  The '-w' then
	essentially determines how frequently the 'p4 pull -ls' is run to
	check the total pull queue size.  A reasonable set of values might
	be '-q 50000 -w 10'.

	The '-Q' option in mutually exclusive with '-nt'.

	The '-Q' option in mutually exclusive with '-q'.

 -recent[=N]
	Specify that only recent changelists should be verified.

	This can be specified as '-recent' or '-recent=N', where N is an integer
	indicating the number of recent changelists to verify.

	If '-recent' is used without the optional '=N' syntax, the
	$SDP_RECENT_CHANGES_TO_VERIFY variable defines how many changelists
	are considered recent; the default is 200.

	If the default is not appropriate for your site, add
	"export SDP_RECENT_CHANGES_TO_VERIFY" to /p4/common/site/config/p4_N.vars.local to
	change the default for an instance, or to /p4/common/site/config/p4_vars.local to
	change it globally.  If $SDP_RECENT_CHANGES_TO_VERIFY is unset, the
	default is 200.

	When -recent is used, files in the unload depot are not verified.

 -dlf <depot_list_file>
	Specify a file containing a list of depots to process in the desired
	order.  By default, all depots in the order reported by reported by
	'p4 depots' are processed, which effectively results in depots being
	processed in alphabetical order, with the singleton Extensions and
	Traits depots (if present) being processed after other depots.

	This '-dlf' option can be useful in time-sensitive situations where the
	order of processing can be prioritized, and/or to prevent processing
       	certain depots.

	The format fo the depot list file is straighforward, one line per
	depot, without the '//' nor trailling /..., so a list might look
	like this sample:

	ProjA
	ProjB
	spec
	.swarm
	unload
	archive
	ProjC

	Blank lines and lines starting with a '#' are treated as comments and
	ignored.

	WARNING: This is not intended to be the primary method of verification,
	because it would be easy to forget to add new depots to the list file.

	If the depot list file is not readable, processing aborts.

	This option disables processing of the singleton Extensions and Traits
	depots unless those depots are explicitly included in the depot list
	file.

 -ignores <regex_ignores_file>
 	Specify the 'verify ignores' file, a file containing a series of
	regular expression patterns representing files or file revisions
	to ignore when scanning for verify errors.  Errors matching the
	pattern will be suppressed from the output captured in the log,
	and will not be considered a verification error.

	If the '-ignores' is not specified, the default verify ignores
	file is:

	/p4/common/config/p4verify.N.ignores

	where 'N' is the SDP instance name. If this file exists, it is
	considered the 'verify ignores' file.

	Specify '-ignores none' to avoid processing the standard ignores
	file.

	The patterns can be specific files, specific file paths, or broader
	patterns (e.g. in the case of entirely abandoned depots).  The file
	provided is passed as the '-f <file>' option to the 'grep' utility,
	and is expected to contain a series of one-line entries, each
	containing an expression to exclude from being considered as verify
	errors reported by this script.

	You can test your expression by first using it with grep to
	ensure it suppresses errors by using a command like this sample,
	providing an older log from this script that contains errors to
	be suppressed:

	grep -Ev -f /path/to/regex_file /path/to/old/p4verify.log

	If your server is case-sensitive, change that command to use '-i':

	grep -a -Evi -f /path/to/regex_file /path/to/old/p4verify.log

	This sample entry ignores a single file revision:

	//Alpha/main/docs/Expenses from February 1999.xls#3

	This sample entry ignores all revisions of a single file:

	//Alpha/main/docs/Expenses from February 1999.xls

	This sample entry ignores all entries in the spec depot
	related to client specs:

	//spec/client

	This sample uses the MD5 checksum from the verify error, just
	to illustrate that this can be used as an alternative to
	specifying file paths:

	D34989BFB8D9B0FB9866C4A604A05410

	This sample ignores BAD! (but not MISSING!) errors under the
	//Beta/main/src directory tree:

	//Beta/main/src/.* BAD!

	WARNING: Ensure that the regex file provided does NOT contain
	any blank lines or comments. The file should contain only tested
	regex patterns.

	This option is intended to provide a way to ignore unrecoverably lost
	file revisions from things like past infrastructure failures, for
	which search and recovery efforts have been abandoned. This option
	subtly changes the question answered by this script from "Are there any
	verify errors?" to "Are there any new verify errors, errors we don't
	already know about?"

	WARNING: This option is not intended to be incorporated into the primary
	method of verification, because ignoring archive errors in this script
	does not solve the problem at its source.  Ideally, the root cause of
	the verify errors should be addressed by recovering lost archives,
	injecting replacement content, or other means. So long as verify errors
	remain, even if ignored by this option, users attempting to access the
	revisions will still see Librarian errors, and replicas will encounter
	errors trying to pull the missing archives. This option could increase
	the risk that such revisions are never dealt with.

 -Ocache
	Specify '-Ocache' to attempt a verification on a replica configured
	with a 'lbr.replication' replication configuration setting value
	of 'cache'.  By default, if the 'lbr.replication' configurable is
	set to 'cache', this script aborts, as replication of such a depot
	will schedule transfers that are likely unintended.  This is a
	safety feature.

	The 'cache' mode is generally used on replicas or edge servers with
	limited disk space.  Because running a verify will cause transfers
	of any missing files, this could result in filling up the disk.

	Use of '-Ocache' is strongly discouraged unless combined with
	other options to ensure that only targeted paths are scheduled
	for transfer.

 -v     Verbose.  Show output of verify attempts, which is suppressed by default.
	Setting SDP_SHOW_LOG=1 in the shell environment has the same effect as -v.

	The default behavior of this script is to generate no terminal output,
	but instead to write output into a log file -- see LOGGING below.  If
	'-v' is specified, the generated log is sent to stdout at the end of
	processing.  This flag is not recommended for routine cron operation or
	for large data sets.

	The -chunks and -recent options are mutually exclusive.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/p4verify.log

	Log rotation and old log cleanup logic does not apply to log files
	specified with -L.  Thus, using -L is not recommended for routine
	scheduled operation, e.g. via crontab.

DEBUGGING OPTIONS:
 -n	No-Operation (NO_OP) mode, for debugging.

 	Display certain commands that would be executed without executing
	them. When '-n' is used, commands that might take a long time to
	run or affect data are only displayed.

	Even in '-n' mode, some information-gathering commands such as
	listing shelved CLs are executed, which may cause the script to take
	a bit of time to run on a large data set even in dry run mode.

 -d	Specify that debug messages should be displayed.

 -D	Use bash 'set -x' extreme debugging verbosity, and imply '-d'.

 -L off
	The special value '-L off' disables logging. This can only be used
	with '-n' for debugging.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

USAGE TIPS:
	On a p4d server machine on which this script runs, the P4USER
	usually has an unlimited ticket in the P4TICKETS file.  If this
	is not the case, ensure that the ticket duration is sufficient
	for the verify operation to complete. If the '-p4config' option
	is used, ensure the defined P4USER references a P4TICKETS file
	with sufficiently far out expiration to prevent issues with ticket
	expiration.

	Depending on scale of data and system resources, this p4verify.sh
	script may run for hours or even days. A ticket duration of less
	than a defined minimum results in an warning being displayed in
	the log (but does not prevent the script from attempting the verify).

	The minimum ticket duration is 31 days 0 hours 0 minutes 0 seconds.

EXAMPLES:
	Example 1: Full Verify

	This script is typically called via cron with only the instance
	parameter as an argument, e.g.:

	p4verify.sh 1

	Example 2: Fast Verify

	A "fast" verify is one in which only the check for MISSING archives
	is done, while the resource-intensive checksum calculation of
	potentially BAD existing archives is skipped.  This is especially
	useful when used on a replica.

	p4verify.sh 1 -o MISSING

	Example 3: Fast and Recent Verify

	The '-o MISSING' and '-recent' flags can be combined for a very
	fast check.  This check might be incorporated into a failover
	procedure.

	p4verify.sh 1 -o MISSING -recent

	Example 4: Submitted Files Only

	This will verify only use submitted files, ignoring shelves and the
	spec and unload depots, putting the results in a specified log:

	p4verify.sh 1 -ns -nS -nu -L  -L /p4/1/logs/p4verify.submitted.log

	Example 5: Shelved Files Only

	This will verify only use submitted files, ignoring shelves and the
	spec and unload depots, putting them in a specified log:

	p4verify.sh 1 -nr -ns -nu -L /p4/1/logs/p4verify.shelved.log

	Example 6: A Dry Run

	The '-n' option can be used for a dry run.  Output may also be
	displayed to the screen ('-v') for a dry run and the log file optionally
	discarded:

	p4verify.sh 1 -n -nS -L off -v

	Example 7: Archive File Load for New Replica

	The p4verify.sh script can be used to schedule transfers of a large
	number of files from a replica.  When doing so, however, overloading
	the new replicas pull queue with too many files may impact metadata
	replication.  This can be addressed by combining a variety of
	options, such as '-chunks' and '-Q'.  For example:

	p4verify.sh 1 -chunks 200M -Q 10000 -w 20 -o MISSING

NOHUP USAGE:
	Because archive verification is typically a long running task,
	it is advisable to use 'nohup' to call each command, and combine
	that by running the command as a background process.  Alternately,
	use 'screen' or similar.

	Any of the examples above can be used with 'nohup', without output
	redirected to /dev/null (i.e. to "the void", as this script handles
	logging and output redirection).

	To use 'nohup', start the command line with 'nohup', and then after
	the command, add this text exactly:

	< /dev/null > /dev/null 2>&1 &

	As a example, Example 2 above, called with nohup, would look like:

	nohup /p4/common/bin/p4verify.sh 1 -o MISSING < /dev/null > /dev/null 2>&1 &

	With the ampersand '&' at the end, the command will appear to return
	immediately as the process continues to run in the background.

	Then optionally monitor the log:

	tail -f /p4/1/logs/p4verify.log

LOGGING:
	This script generates no output by default.  All (stdout and stderr) is
	logged to /p4/N/logs/p4verify.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

NOTIFICATIONS:
	In addition to logging, a short summary of the verify is sent as a
	notification.  The summary is reliably short even if the output of the
	verifications done by this script results in a large log file.

	There are two notification schemes with this script:

	* Email notification is always attempted.

	* AWS SNS notification is attempted if the SNS_ALERT_TOPIC_ARN custom
	setting is defined. This is typically set in:

	/p4/common/site/config/p4_N.vars.local

TIMING:
	The log file captures various timing information, including the
	time required to verify each depot, or each chunk or path if
	'-paths' or '-chunks' are used. The time to verify shelves
	in all depots is reported separately from submitted files.

	Timing indications all start with the text 'Time: ' on the beginning
	of a line of output in the log file, and can be extracted with a
	command like this example (adjusting the log file name as needed):

	grep -a ^Time: /p4/1/logs/p4verify.log

EXIT CODES:
	An exit code of 0 indicates no errors were encountered attempting to
	perform verifications, AND that all verifications attempted
	reported no problems.

	A exit status of 1 indicates that verifications could not be
	attempted for some reason.

	A exit status of 2 indicates that verifications were successfully
	performed, but that problems such as BAD or MISSING files
	were detected, or else system limits prevented verification.

USAGE for p4login v4.4.4:

p4login [<instance>] [-p <port> | -service] [-automation] [-all]

   or

p4login -h|-man

DESCRIPTION:

	In its simplest form, this script simply logs in P4USER to P4PORT
	using the defined password access mechanism.

	It generates a login ticket for the SDP super user, defined by
	P4USER when sourcing the SDP standard shell environment.  It is
	called from cron scripts, and so does not normally generate any
	output.

	If run on a replica with the -service option, the serviceUser defined
	for the given replica is logged in.

	The $SDP_AUTOMATION_USERS variable can be defined in
	/p4_N.vars. If defined, this should contain a
	comma-delimited list of automation users to be logged in when the
	-automation option is used.  A definition might look like:

	export SDP_AUTOMATION_USERS=builder,trigger-admin,p4review

	Login behavior is affected by external factors:
	1. P4AUTH, if defined, affects login behavior on replicas.

	2. The auth.id setting, if defined, affects login behaviors (and
	generally simplifies them).

	3. The $SDP_ALWAYS_LOGIN variable.  If set to 1, this causes p4login
	to always execute a 'p4 login' command to generate a login ticket,
	even if a 'p4 login -s' test indicates none is needed.  By default,
	the login is skipped if a 'p4 login  -s' test indicates a long-term
	ticket is available that expires 31+days in the future.
	Add "export SDP_ALWYAYS_LOGIN=1" to /p4_N.vars to
	change the default for an instance, or to /p4/common/bin/p4_vars to
	change it globally.  If unset, the default is 0.

	4. If the P4PORT contains an ssl: prefix, the P4TRUST relationship
	is checked, and if necessary, a p4 trust -f -y is done to establish
	trust.

OPTIONS:
<instance>
	Specify the SDP instances.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined, p4login
	exists immediately with an error message.

 -service
	Specify -service when run on a replica or edge server to login
	the super user and the replication service user.

	This option is not compatible with '-p <port>'.

 -p <port>
	Specify a P4PORT value to login to, overriding the default
	defined by P4PORT setting in the environment.  If operating
	on a host other than the master, and auth.id is set, this
	flag is ignored; the P4TARGET for the replica is used
	instead.

	This option is not compatible with '-service'.

 -automation
	Specify -automation to login external automation users defined
	by the $SDP_AUTOMATION_USERS variable.

 -v     Show output of login attempts, which is suppressed by default.
	Setting SDP_SHOW_LOG=1 in the shell environment has the same
	effect as -v.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/p4login.log

 -d     Set debugging verbosity.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	1. Typical usage for automation, with instance SDP_INSTANCE defined
	in the environment by sourcing p4_vars, and logging in only the super
	user P4USER to P4PORT:
	source /p4/common/bin/p4_vars abc
	p4login

	Login in only P4USER to the specified port, P4MASTERPORT in this example:
	p4login -p $P4MASTERPORT

	Login the super user P4USER, and then login the replication serviceUser
	for the current ServerID:
	p4login -service

	Login external automation users (see SDP_AUTOMATION_USERS above):
	p4login -automation

	Login all users:
	p4login -all

	Or: p4login -service -automation

LOGGING:
	This script generates no output by default.  All (stdout and stderr) is
	logged to /p4/N/logs/p4login.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

	If the '-v' flag is used, the contents of the log are displayed to
	stdout at the end of processing.

EXIT CODES:
	An exit code of 0 indicates a valid login ticket exists, while a
	non-zero exit code indicates a failure to login.

USAGE for sdp_health_check.sh v1.13.2:

sdp_health_check.sh

   or

sdp_health_check.sh -h|-man

DESCRIPTION:

	This script does a health check of the SDP. It generates a
	report log, which can be emailed to support@perforce.com.
	It identifies SDP instances and reports on general SDP health.

	It must be run as the OS user who owns the /p4/common/bin
	directory.  This should be the user account which runs the
	p4d process, and which owns the /p4/common/bin directory
	(often 'perforce' or 'p4admin').

	Characteristics of this script:
	* It is always safe to run.  It does only analysis and reporting.
	* It does only fast checks, and has no interactive prompts.
	  Some log files are captured such as checkpoint.log, but not
	  potentially large ones such as the p4d server log.
	* It requires no command line arguments.
	* It works for any and all UNIX/Linux SDP version since 2007.

	Assumptions:
	* The SDP has always used /p4/common/bin/p4_vars as the shell
	  environment file.  This is consistent across all SDP versions.

OPTIONS:
 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	This script is typically called with no arguments.

LOGGING:
	This script generates a log file and also displays it to stdout at the
	end of processing.  By default, the log is:

	/tmp/sdp_health_check.<datestamp>.log

	or

	/tmp/sdp_health_check.log

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

EXIT CODES:
	An exit code of 0 indicates no errors or warnings were encountered.

USAGE for verify_sdp.sh v5.27.1:

verify_sdp.sh [<instance>] [-online] [-skip <test>[,<test2>,...]] [-warn <test>[,<test2>,...]] [-c] [-si] [-L <log>|off ] [-D]

   or

verify_sdp.sh -h|-man

DESCRIPTION:

	This script verifies the current SDP setup for the specified instance,
	and also performs basic health checks of configured servers.

	This uses the SDP instance bin directory /p4/N/bin to determine
	what server binaries (p4d, p4broker, p4p) are expected to be configured
	on this machine.

	Existence of the '*_init' script indicates the given binary is
	expected. For example, for instance 1, if /p4/1/bin/p4d_1_init
	exists, a p4d server is expected to run on this machine.

	Checks may be executed or skipped depending on what servers are
	configured. For example, if a p4d is configured, the $P4ROOT/server.id
	file should exist. If p4p is configured, the 'cache' directory
	should exist.

OPTIONS:
 <instance>
	Specify the SDP instances.  If not specified, the SDP_INSTANCE
	environment variable is used instead.  If the instance is not
	defined by a parameter and SDP_INSTANCE is not defined,
	exits immediately with an error message.

 -online
	Online mode.  Does additional checks that expect p4d, p4broker,
	and/or p4p to be online. Any servers for which there are
	*_init scripts in the Instance Bin directory are checked.  An
	error is reported if p4d is expected to be online and is not;
	warnings are displayed if p4broker or p4p are not online.
	The Instance Bin directory is the /p4/N/bin directory, where N
	is the SDP instance name.

 -c	Specify '-c' to call ccheck.sh to compare configurables, using
	the default config file: /configurables.cfg

	See 'ccheck.sh -man' for more information.

	This option can only be used in Online mode; if '-c' is specified,
	'-online' is implied.

 -skip <test>[,<test2>,...]

	Specify a comma-delimited list of named tests to skip.

	Valid test names are:

	* cron|crontab: Skip crontab check. Use this if you do not expect crontab to
	be configured, perhaps if you use a different scheduler.
	* excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH.
	* init: Skip compare of init scripts w/templates in /p4/common/etc/init.d
	* license: Skip license related checks.
	* commitid: Skip check ensuring ServerID of commit starts with 'commit' or 'master'.
	* masterid: Synonym for commitid.
	* offline_db: Skip checks that require a healthy offline_db.
	* owner: Skip checks that require ownership of specific files/folders.
	* p4root: Skip checks that require healthy P4ROOT db files.
	* p4t_files: Skip checks for existence of P4TICKETS and P4TRUST files.
	* passwd|password: Skip SDP password checks.
	* version: Skip version checks.

	As an alternative to using the '-skip' option, the shell environment
	variable VERIFY_SDP_SKIP_TEST_LIST can be set to a comma-separated
	list of named tests to skip.  Using the command line parameter is the
	best choice for temporarily skipping tests, while specifying the
	environment variable is better for making permanent exceptions (e.g.
	always excluding the crontab check if crontabs are not used at this
	site).  The variable should be set in /p4/common/config/p4_N.vars.

	If the '-skip' option is provided, the VERIFY_SDP_SKIP_TEST_LIST
	variable is ignored (not appended to). So it may make sense to
	reference the variable on the command line. For example, if the
	value of the variable is 'crontab', to skip crontab and license
	checks, you could specify:

	-skip $VERIFY_SDP_SKIP_TEST_LIST,license

 -warn <test>[,<test2>,...]
	Specify a comma-delimited list of named tests that will be reported
	as warnings rather than errors.

	The list of valid test names as the same as for the '-skip' option.

	As an alternative to using the '-warn' option, the shell environment
	variable VERIFY_SDP_WARN_TEST_LIST can be set to a comma-separated
	list of name tests to skip.  Using the command line parameter is the
	best choice for temporarily converting errors to warnings, while
	specifying the environment variable is better for making the
	conversion to warnings permanent.  The variable should be set in
	/p4/common/config/p4_N.vars file.

	If the '-warn' option is provided, the VERIFY_SDP_WARN_TEST_LIST
	variable is ignored (not appended to). So it may make sense to
	reference the variable on the command line. For example, if the
	value of the variable is 'crontab', to convert to warnings for
	crontab and excess binaries tests, you could specify:

	-warn $VERIFY_SDP_WARN_TEST_LIST,excess

 -si	Silent mode, useful for cron operation.  Both stdout and stderr
	are still captured in the log.  The '-si' option cannot be used
	with '-L off'.

 -L <log>
	Specify the log file to use.  The default is /p4/N/logs/verify_sdp.log
	The special value 'off' disables logging to a file.

	Note that '-L off' and '-si' are mutually exclusive.

 -D	Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

EXAMPLES:
	Example 1: Typical usage:

	This script is typically called after SDP update with only the instance
	name or number as an argument, e.g.:

	verify_sdp.sh 1

	Example 2: Skipping some checks.

	verify_sdp.sh 1 -skip crontab

	Example 3: Automation Usage

	If used from automation already doing its own logging, use -L off:

	verify_sdp.sh 1 -L off

LOGGING:
	This script generates a log file and also displays it to stdout at the
	end of processing.  By default, the log is:
	/p4/N/logs/verify_sdp.log.

	The exception is usage errors, which result an error being sent to
	stderr followed usage info on stdout, followed by an immediate exit.

	If the '-si' (silent) flag is used, the log is generated, but its
	contents are not displayed to stdout at the end of processing.

EXIT CODES:
	An exit code of 0 indicates no errors were encountered attempting to
	perform verifications, and that all checks verified cleanly.

USAGE for ccheck.sh v1.2.0:

ccheck.sh [<SDPInstance>] [-p <Profile>] [-c <CfgFile>] [-y] [-v] [-d|-D]

or

ccheck.sh [-h|-man|-V]


DESCRIPTION:
	This script compares configurables set on the current server with best
	practices defined a data file.

OPTIONS:
 -p <Profile>
 	Specify a profile defined in the config file, such as 'demp' or 'hcc'.  A profile
	defines a set of expected configurable values that can differ from the expected
	values in other profiles.  For example, for a demo environment, the filesys.P4ROOT.min
	might have an expected value of 128M, while the expected value in a prod (production)
	profile might be 5G, and the same value might be 30G for 'prodent', the profile for
	production at large enterprise scale.

	The 'always' profile defines settings that always apply whether '-p' is specified
	or not. The profile specified with '-p' applies in addition to the 'always'
	configuration, adding to and possibly overriding settings from the 'always'
	configuration.

	The defaut profile is 'prod', the production profile.

	Specify the special value '-p none' to use only the settings defined in the
	'always' profile.

 -c <CfgFile>
	Specify an alternate config file that defines best practice configurables. This
	is intended for testing.

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) goes to
	$LOGS/ccheck.log

	NOTE: This script is self-logging.  That is, output displayed on the screen
	is simultaneously captured in the log file. Using redirection operators like
	'> log' or '2>&1' are unnecessary, nor is using 'tee'.

 -y	Live operation mode.  By default, any commands that affect data, such as
	setting configurables, are displayed, but not executed.  With the '-y' option,
	commands may be executed.

	This option is included for future needs. This current version of
	ccheck.sh does not execute any commands that affect data.

 -d	Display debug messages.

 -D     Set extreme debugging verbosity using bash 'set -x' mode. Implies -d.

 -si	Silient Mode.  No output is displayed to the terminal (except for usage errors
	on startup). Output is captured in the log.  The '-si' cannot be used with
	'-L off'.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

FILES:
	The standard configurables config file is:

	/p4/common/config/configurables.cfg

EXAMPLES:
	Example 1: Check configurables with the default profile, and no logging:

	ccheck.sh -L off

	Example 2: Check configurables with the 'prod' (Production) profile:

	ccheck.sh -p prod

	Example 3: Check configurables with the 'demo' profile, doing a verbose comparison:

	ccheck.sh -p demo -v

FUTURE ENHANCEMENTS:
	Presently, this ccheck.sh v1.2.0 only reports configurables. It does not
	support changing configurables.

	As the script is currently only capable of reporting, the '-y' option has no
	effect.

	Some possible future enhancements are:
	* Extend reporting to suggesting configuration changes.

	* Provide an option to make changes to configurables that are safe to change
	  immediately, and provide guidance on those configurables that are best set with
	  guidance and plannning.

	* Provide a way to specify custom exemptions for certain configurables.

	* Added multi-version support for backward compatibility. This version assumes
	  P4D 2023.1+ (though will be useful for older versions).

USAGE for load_checkpoint.sh v3.2.6:

load_checkpoint.sh {<checkpoint> [<jnl.1> <jnl.2> ...] | -latest | -latest_jnls | -jo <jnl.1> [<jnl.2> ...] | -jo_latest } [-R|-F <SafetyFactor>] [-i <instance>] [-s <ServerID>] [-t <Type>] [-no_start | [-no_xu] [-verify {default|"Verify Options"} [-delay <delay>]]] [-c] [-l] [-r] [-b] [-y] [-L <log>] [-si] [-d|-D]

or

load_checkpoint.sh [-h|-man]


DESCRIPTION:
	This script can load a specified checkpoint and/or numbered journals
	into P4ROOT (/p4/N/root) and/or /p4/N/offline_db (where 'N' is the SDP
	instance name). It supports a variety of use cases for replaying
	checkpoints and journals, including:

	* Seeding or Reseeding a replica or edge server.
	* Loading a checkpoint on the commit, e.g. in a recovery scenario.

	Checkpoints and/or journals can be specified in one of two ways: they can
	be specified as parameters to this script, or they can be determined by
	this script if they appear in the SDP standard location according to the
	journalPrefix standard.  They key methods are:

	* Specify the path to the checkpoint to replay. The checkpoint can be in the
	form of a compressed .gz file, an uncompressed checkpoint file, or a directory
	(for parallel checkpoints).
	* Use '-latest' to have this script find the latest checkpoint available. For
	a commit server, /p4/N/checkpoints/p4_N is searched. For other servers, their
	journalPrefix is used. The timestamp on the latest available *.md5 file is
	used to determine what checkpoint is the latest available, regardless of
	checkpoint form (compressed or uncompressed file, or a directory for parallel
	checkpoints).
	* Use '-latest_jnls' to find the latest checkpoint as with '-latest', and then
	also find and replay any available subsequent numbered journals.
	* Use '-jo' ("journal only") to specify path(s) to one or more numbered
	journals to be supplied as parameters to this script. Journal files provided
	may be compressed or uncompressed.
	* Use '-jo_latest' to find any numbered journals available to be replayed
	based on the journal counter of the data set.

	At the start of processing, preflight checks are done. Preflight checks
	include:
	* The specified checkpoint and corresponding *.md5 file must exist.
	* The specified checkpoint can be a compressed or uncompressed file or a
	directory (for parallel checkpoints).
	* All journal files to replay (if any are specified) must exist.
	* The $P4ROOT/server.id file must exist, unless '-s' is specified.
	* If the $P4ROOT/server.id file exists and '-s' is specified, the values
	must match.
	* The $P4ROOT/license file must exist, unless '-l' is specified or if
	the replica type does not require a license (such as an edge server).
	* The SDP structure and key files must exist.
	* Disk space checks are done to attempt to determine if sufficient space
	is available to replay the checkpoint.

  	If the preflight passes, the p4d_N service is shutdown. The p4broker_N
	service is shutdown if it is configured.

	If a P4LOG file exists, it is moved aside so there is a fresh p4d server
	log corresponding to operation after the checkpoint load.

	If a P4JOURNAL file exists, it is moved aside as the old journal data is
	no longer relevant after a checkpoint replay. (Exception: If the P4JOURNAL
	is speciffed in a list of journals to reply, then it is not moved aside).

	Next, any existing state* files in P4ROOT are removed.

	Next, any existing database it is files in P4ROOT are preserved and moved aside,
	unless '-R' is specified to remove them.

	Next, the specified checkpoint is loaded. Upon successful completion,
	'p4d -xu' is executed (by default) to help ensure the service can be started
	with the p4d binary used to replay the checkpoint.  Then the Helix Core
	service is started with the current p4d binary.

	If the server to be started is a replica, the serviceUser configured for
	the replica is logged into the P4TARGET server.  Any needed 'p4 trust' and
	'p4 login' commands are done to enable replication.

	Note that this part of the processing will fail if the correct super user
	password is not stored in the standard SDP password file,

	/p4/common/config/.p4passwd.p4_N.admin

	After starting the server, a local 'p4 trust' is done if needed, and then
	a 'p4login -service -v' and 'p4login -v'.

	By default, the p4d_N service is started, but the p4broker_N service is not.
	Specify '-b' to restart both services.

	Finally, the offline_db is rebuilt using the same specified checkpoint and
	journals.

ARGUMENTS AND OPTIONS:
 <checkpoint>
	Specify the path to the checkpoint file or directory to load. Exactly one
	checkpoint must be specified. If a checkpoint file is specified, a serial
	checkpoint replay will be done. If a checkpoint directory is specified,
	a parallel replay will be done using the individual files in the
	directory.

	For checkpoint files:

	The file may be a compressed or uncompressed checkpoint, and it may be a case
	sensitive or case-insensitive checkpoint.  The checkpoint file must have a
	corresponding *.md5 checksum file in the same directory, with one of two name
	variations: If the checkpoint file is /somewhere/foo.gz, the checksum file may
	be named /somewhere/foo.gz.md5 or /somewhere/foo.md5.

	For checkpoint directories:

	This option is required unless the '-latest' option is used.

 <jnl.1> [<jnl.2> ...]
	Specify the path to the one or more journal files to replay after the
	checkpoint, in the correct sequence order.

 -latest
	Specify this as an alternative to providing a specific checkpoint file or
	directory.  The script will then search for the latest *.md5 file in the
	standard checkpoints directory and use that to replay.

	The standard checkpoints directory search is one of the following:

	Commit servers:   /p4/N/checkpoints
	Standby servers:  /p4/N/checkpoints

	Edge servers:     /p4/N/checkpoints.<ShortServerID>

	For standby servers that target an edge server, where the ServerID
	starts with p4d_ha_edge, p4d_ham_edge, p4d_fs_edge, or p4d_fsm_edge,
	the directory for the target edge server is searched.  (If NFS sharing,
	this directory will naturally exist. Otherise, the directory should
	be created and populated as needed on the standby of the edge for
	seeing with checkpoints from the edge.

	The most recent *.md5 file found in the standard checkpoints directory
	determines which checkpoint to load.  The actual checkpoint can be a
	file (gzipped or not) or directory (for parallel checkpoints).

	This option is mutually exclusive with '-latest_jnls'.

 -latest_jnls
	This option is similar to '-latest'.  However, with '-latest_jnls', in
	addition to replaying the latest checkpoint, any subsequent numbered
	journals available in the standard checkpoints directory are also
	replayed.

	This option will only replay numbered journals, not the live P4JOURNAL
	file. However, if the $P4JOURNAL is provided, then it will be
	replayed after all available numbered journals are replayed.

	This option is mutually exclusive with '-latest'.

	If used with '-jo', where the checkpoint and possibly some numbered
	journals will already have been replayed into P4ROOT, then the meaning
	of this option changes.  It will replay needed numbered journal up to
	the latest available, so long as those journals appear in the standard
	checkpoints directory with the usual naming convention. With this option,
	the journal needed are calculated based on the journal counter stored in
	database in the P4ROOT dir.

 -R	Specify '-R' to remove db.* files in P4ROOT rather than moving them aside.

	By default, databases are preserved for possible future for investigation.
 	A folder named 'MovedDBs.<datestamp>' is created under the P4ROOT directory,
	and databases are moved there.

	Keeping an extra set of databases requires sufficient disk space to hold
	the extra set of db.* files.

	If -R specified, old databases in P4ROOT are removed, along with state*
	and other files, and the server.locks directory.

 -F <SafetyFactor>
	When replacing an existing set of db.* files, a safety factor
	is used. This is simply the factor by which the size of pre-existing
	databases is multiplied when comparing against available disk space.

	Specify '-F 0' to disable the safety factor check.

	The disk space safety check is only meaningful if P4ROOT was previously
	populated with a full set of data.

	Specifying a nubmer greater than 1, say 1.2 (the default) gives more
	breathing room.

	Specifying a value lower than 1, say 0.95, may be OK if you are certain
	the expanded-from-a-checkpoint db.* files are significantly smaller
	than size the prior set of db.* files.

	This option is mutually exclusive with '-R'. If '-R' is used, databases
	are removed, and there is no need to calculate disk space.

 -i <instance>
	Specify the SDP instance. This can be omitted if SDP_INSTANCE is already
	defined.

 -s <ServerID>
	Specify the ServerID.  This value is written into $P4ROOT/server.id file.

	If no $P4ROOT/server.id file exists, this flag is required.

	If the $P4ROOT/server.id file exists, this argument is not needed. If this
	'-s <ServerID>' is given and a $P4ROOT/server.id file exists, the value in
	the file must match the value specified with this argument.

 -t <Type>
	Specify the replica type tag if the checkpoint to be loaded is for an edge
	server or replica. The set of valid values for the replica type
	tag are defined in the documentation for mkrep.sh.  See: mkrep.sh -man

	If the type is specified, the '-s <ServerID>' is required.

	If the SDP Server Spec Naming Standard is followed, the ServerID
	specified with '-s' will start with 'p4d_'. In that case, the
	value for '-t edge' value is inferred, and '-t' is not required.

	If the type is specified or inferred, certain behaviors change based
	on the type:
	* If the type is edge, only the correct edge-specific subset of database
	tables are loaded.
	* The P4ROOT/license file check is suppressed unless the type is
	ha, ham, fs, for fsm (standby replicas usable with 'p4 failover').

	Do not use this '-t <Type>' option if the checkpoint being loaded is
	for a commit server.

	For an edge server, an edge seed checkpoint created with edge_dump.sh
	must be used if the edge is filtered, e.g. if any of the *DataFilter
	fields in the server spec are used. If the edge server is not filtered
	by means other than being an edge server (for which certain tables are
	filtered by nature), a standard full checkpoint from the commit can be
	used.

	For a filtered forwarding replica, a proper seed checkpoint must be
	loaded.  This can be created on the commit using key options to p4d,
	including '-P <ServerID> -jd <SeedCkp' on the commit (possibly using
	the 'offline_db' to avoid downtime, similar to how edge_dump.sh
	works for edge servers).

	WARNING: While this script is useful for seeding a new edge server, this
	script is NOT to be used for recovering or reseeding an existing edge server,
	because all edge-local database tables (mostly workspace data) would be lost.
	To recover an existing edge server, see the recover_edge.sh script.

	Warning: If this option is specified with the incorrect type for the
	checkpoint specified, results will be unpredictable.

 -verify default [-delay <delay>]
 -verify "Verify Options" [-delay <delay>]

	Specify '-verify' to initiate a call to 'p4verify.sh' after the server
	is online. On a replica, this can be useful to cause the server to pull
	missing archive files from its P4TARGET server. If this load_checkpoint.sh
	script is used in a recovery situation for a commit server, this '-verify'
	option can be used to discover if archive files are missing after the
	metadata is recovered.

	The 'p4verify.sh' script has a rich set of options. See 'p4verify.sh -man'
	for more info.  The options to pass to p4verify.sh can be passed in a
	quoted list, or '-verify default' can be used to indicate these default
	options:

	-o MISSING

	By default, a fast verify is used if the p4d version is new enough (2021.1+).
	See 'p4verify.sh -man' for more information, specifically the description
	of the '-o MISSING' option.

	In all cases, p4verify.sh is invoked as a background process; this
	load_checkpoint.sh script does not wait for it to complete. The p4verify.sh
	script will email as per normal when it completes.

	The optional delay option specifies how long to wait until kicking off the
	p4verify.sh command, in seconds. The default is 600 seconds.
	This is intended to give the replica time get get caught up with metadata
	before the archive pulls are scheduled. The delay is a workaround for job079842.

	This option is cannot be used with '-no_start'.

 -c	Specify that SSL certificates are required, and not to be generated with
	'p4d_N -Gc'.

	By default, if '-c' is not supplied and SSL certs are not available, certs
	are generated automatically with 'p4d_N -Gc'.

 -l	Specify that the server is to start without a license file. By default, if
	there is no $P4ROOT/license file, this script will abort.  Note that if '-l'
	is specified and a license file is actually needed, the attempt this script makes
	to start the server after loading the checkpoint will fail.

	If '-t <type>' is specified, the license check is skipped unless the type is
	'ha', 'ham', 'fs,' or 'fsm'.  Replicas that are potential targets for a 'p4
	failover' need a license file for a failover to work.

 -r	Specify '-r' to replay only to P4ROOT.  By default, this script replays both
	to P4ROOT and the offline_db.

 -no_start
	Specify '-no_start' to avoid starting the p4d service after loading the
	checkpoint.

	This option is cannot be used with '-verify'.

 -no_xu
	Specify '-no_xu' to skip the 'p4d -xu' step that upgrade the database schema.

	By default, a 'p4d -xu' is done to help ensure the service can be started
	with the current p4d binary after the checkpoint is replayed.

	If the p4d binary used to replay the checkpoint is a newer major version
	than the one used to create the checkpoint, the service will not start
	after the replay until the 'p4d -xu' step is done.  If this '-no_xu' option
	is used and the p4d binary is a newer major version, have a plan to get
	the 'p4d -xu' done before the service is started.

	In EXAMPLES below, see the example titled "Multi Pass Replay of Checkpoints
	and Journals" for an example of using this option as part of a migration
	procedure.

 -jo <jnl.1> [<jnl.2> ...]
	Specify '-jo' to replay only one or more numbered journals without
	first replaying a full checkpoint.  With this option, the cleanup that
	normally occurs before the replay is disabled. The db.* and state* files
	in P4ROOT, as well as P4LOG and P4JOURNAL files, etc. are left in place.

	With '-jo', the paths to journal files must be specified.

	This option is mutually exclusive to the similar option '-jo_latest'.

	This option implies '-r'.

 -jo_latest
	Specify '-jo_latest' to replay only one or more numbered journals without
	first replaying a full checkpoint.  With this option, the cleanup that
	normally occurs before the replay is disabled. The db.* and state* files
	in P4ROOT, as well as P4LOG and P4JOURNAL files, etc. are left in place.

	With '-jo_latest', numbered journals to replay are calculated and determined,
	not specified as parameters.

	This option is mutually exclusive to the similar option '-jo'.

	This option implies '-r'.

 -b	Specify '-b' to start the a p4broker process (if configured).  By default
	the p4d process is started after loading the checkpoint, but the p4broker
	process is not. This can be useful to ensure the human administrator has
	an opportunity to do sanity checks before enabling the broker to allow
	access by end users (if the broker is deployed for this usage).

 -y	Use the '-y' flag to bypass an interactive warning and confirmation
	prompt.

 -L <log>
	Specify the path to a log file.  By default, all output (stdout and stderr)
	goes to:
	/p4/<instance>/logs/load_checkpoint.<timestamp>.log

	NOTE: This script is self-logging.  That is, output displayed on the screen
	is simultaneously captured in the log file.  Do not run this script with
	redirection operators like '> log' or '2>&1', and do not use 'tee.'

 -si	Operate silently.  All output (stdout and stderr) is redirected to the log
	only; no output appears on the terminal.

 -d     Set debugging verbosity.

 -D     Extreme debugging verbosity using bash 'set -x' mode.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

USAGE TIP:
	All the non-interactive examples below illustrate the practice of using
	redirects to create an extra log file named 'load.log' in the $LOGS
	directory for the instance. This load.log file is identical to, and in
	addition to, the standard timestamped log generated by this script.  The
	intent of this practice is to make it easier to find the log for the
	last checkpoint loaded on any given server machine. This convention is only
	useful if used consistently.

	Several examples below illustrate the instance option, '-i' option to
	specify the SDP instance. This is optional and can safely be omitted in an
	environment where the standard SDP shell environment is sourced on login,
	and where there is only a single instance on the server machine.

EXAMPLES:
	EXAMPLE 1: Non-interactive Usage

	Non-interactive usage (bash syntax) to load a checkpoint:

	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	Then, monitor with:
	tail -f $(ls -t $LOGS/load_checkpoint.*.log|head -1)


	EXAMPLE 2: Checkpoint Load then Verify, for the SDP Instance alpha.

	Non-interactive usage (bash syntax) to load a checkpoint followed by a full
	verify of recent archives files only with other options passed to verify.sh:

	nohup /load_checkpoint.sh /p4/alpha/checkpoints/p4_alpha.ckp.95442.gz -i alpha -verify -recent -nu -ns -y < /dev/null > /p4/alpha/logs/load.log 2>&1 &

	EXAMPLE 3: Load Checkpoint and Journals

	Non-interactive usage (bash syntax) to loading a checkpoint and subsequent journals:

	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz /p4/1/checkpoints/p4_1.jnl.4025 /p4/1/checkpoints/p4_1.jnl.4026 -i 1 -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	Then, monitor with:
	tail -f $(ls -t $LOGS/load_checkpoint.*.log|head -1)


	EXAMPLE 4: Interactive usage.

	Interactive usage to load a checkpoint with no license file.

	/load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -l

	With interactive usage, logging still occurs; all output to the screen is
	captured.

	Note that non-interactive usage with nohup is recommended for checkpoints
	with a long replay duration, to make operation more reliable in event of a
	shell session disconnect.  Alternately, running interactively in a 'screen'
	session (if 'screen' is available) provides similar protection against
	shell session disconnects.

	EXAMPLE 5: Seed New Edge

	Seeding a new edge server.

	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -s p4d_edge_syd < /dev/null > /p4/1/logs/load.log 2>&1 &

	WARNING: While this script is useful for seeding a new edge server, this
	script is NOT to be used for recovering or reseeding an existing edge server,
	because all edge-local database tables (mostly workspace data) would be lost.
	To recover an existing edge server, see the recover_edge.sh script.

	EXAMPLE 6: Seed New Edge and Verify

	Seeding a new edge server and then do a verify with default options.

	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025.gz -i 1 -s p4d_edge_syd -verify default < /dev/null > /p4/1/logs/load.log 2>&1 &

	EXAMPLE 7: Load a Parallel Checkpoint on an Edge and Verify Recent

	This non-interactive example loads a parallel checkpoint directory.  The
	usage difference is that the checkpoint path provided is a parallel
	checkpoint directory rather than a single checkpoint file.  This example
	loads the checkpoint for a new edge server, and verifes only the most
	recent 3 changes in each depot.  The delay before calling p4verify.sh,
	10 minutes (600) by default, is shortened to 5 seconds in this example.

	nohup /load_checkpoint.sh /p4/1/checkpoints/p4_1.ckp.4025 -i 1 -s p4d_edge_syd -verify "-o MISSING -recent=3 -ns -L /p4/1/logs/p4verify.fast_and_recent.log" -delay 5 -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	EXAMPLE 8: Multi Pass Replay of Checkpoints and Journals

	In this example, we want to use a multi-pass procedure involving replay of
	a checkpoint at one point in time, and then later replay subsequent numbered
	journals later. This method can be useful to reduce downtime required for
	migration procedures involving a checkpoint replay if the checkpoint replay
	takes a while, e.g. a few hours or more.  The gist of the approach is to replay
	the checkpoint a day or so a head of the scheduled maintenance.  Then replay
	subsequent numbered journals each day after. Then in the maintenance
	window, replay just the last numbered journal from the old environment in
	the new environment.

	This approach involves a few options:
	* When the checkpoint is replayed, '-no_start' and '-no_xu'. Either specify
	the path to the checkpoint, or use '-latest'.
	* When the numbered journals are replayed in days leading up to the maintenance
	window, use the '-jo_latest' option to replay only a numbered journal.
	* During the maintenance window, load any final numbered journals, then start the
	service.

	Pass 1, 3 days before maintenance:
	nohup load_checkpoint.sh -latest -no_start -no_xu -r -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	Pass 2, 2 days before maintenance:
	nohup load_checkpoint.sh -jo_latest -no_start -no_xu -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	Pass 3, 1 day before maintenance:
	nohup load_checkpoint.sh -jo_latest -no_start -no_xu -y < /dev/null > /p4/1/logs/load.log 2>&1 &

	Pass 4, during the maintenance window:
	nohup load_checkpoint.sh -jo_latest -y < /dev/null > /p4/1/logs/load.log 2>&1 &

USAGE for mkdirs.sh v7.1.1:

mkdirs.sh <instance> [-s <ServerID>] [-t <ServerType>] [-tp <TargetPort>] [-lp <ListenPort>] [-I <svc>[,<svc2>]] [-MDD /bigdisk] [-MCD /ckps] [-MLG /jnl] [-MDB1 /db1] [-MDB2 /db2] [-f] [-p] [-no_init|-no_systemd|-no_enable] [-fs|-ls] [-no_cron] [-no_firewall] [-test [-clean]] [-n] [-L <log>] [-d|-D]

OR

mkdirs.sh <instance> [-c <CfgFile>] [-f] [-p] [-no_init|-no_systemd] [-test [-clean]] [-n] [-L <log>] [-d|-D]

or

mkdirs.sh [-h|-man]


DESCRIPTION:

== Overview ==

This script initializes an SDP instance on a single machine.

This script is intended to support two scenarios:

* First time SDP installation on a given machine.  In this case, the
  user calls the install_sdp.sh script, which in turn calls this script.
  See 'install_sdp.sh -man' for more information.

* Adding new SDP instances (separate Helix Core data sets) to an existing
  SDP installation on a given machine. For this scenario, this mkdirs.sh
  script is called directly.

An SDP instance is a single Helix Core data set, with its own unique
set of one set of users, changelist numbers, jobs, labels, versioned
files, etc. An organization may run a single instance or multiple
instances.

This is intended to be run either as root or as the operating system
user account (OSUSER) that p4d is configured to run as, typically
'perforce'.  It should be run as root for the initial install.
Subsequent additions of new instances do not require root.

== Directory Structure ==

If an initial install as done by a user other than root, various
directories must exist and be writable and owned by 'perforce' before starting:

* /p4
* /hxcheckpoints
* /hxdepots
* /hxlogs
* /hxmetadata
* /hxmetadata2
* /opt/perforce/helix-sdp (optional; used for package installations)

The directories starting with '/hx' are configurable, and can be changed by
settings in the mkdirs.cfg file (or mkdirs.N.cfg), or with command line
options as illustrated here:

 -MDD /bigdisk
 -MCD /ckps
 -MLG /jnl
 -MDB1 /db1
 -MDB2 /db2

This script creates an init script in the /p4/N/bin directory.

== Crontab ==

Crontabs are generated for all server types.

After running this script, set up the crontab based on templates
generated as /p4/common/etc/cron.d.  For convenience, a sample crontab
is generated for the current machine as /p4/p4.crontab.<SDPInstance>
(or /p4/p4.crontab.<SDPInstance>.new if the former name exists).

These files should be copied or merged into any existing files named
with this convention:

/p4/common/etc/cron.d/crontab.<osuser>.<host>

where <osuser> is the user that services run as (typically 'perforce'),
and <host> is the short hostname (as returned by a 'hostname -s' command).

== Init Mechanism ==

If this script is run as root, the init mechanism (Systemd or SysV) is
configured for installed services.

The Systemd mechanim is used if the the /etc/systemd/system folder exists and
systemctl is in the PATH of the root user. Otherwise, the SysV init mechanism
is used.

== Firewall Configuration ==

This script checks to see if a known firewall type is available.  The
firewalld is checked using the command 'firewall-cmd --state' command,
and the ufw firewall is checked using the 'ufw status'.  If either firewall
is detected, the ports required for Helix Core applications installed are
opened in the firewall.  For more information, see the templates in these folders:

/p4/common/etc/firewalld
/p4/common/etc/ufw

If the firewall service is not online, no firewall coniguration is performed.

== SELinux Configuration ==

If Systemd is used and the semanage and restorecon utilities are available in
the PATH of the root user, then SELinux configuration for the installed
services is done.

REQUIRED PARAMETERS:
 <instance>
	Specify the SDP instance name to add.  This is a reference to the Perforce
	Helix Core data set.

OPTIONS:
 -s <ServerID>
	Specify the ServerID, overriding the REPLICA_ID setting in the configuration
	file.

 -S <TargetServerID>
	Specify the ServerID of the P4TARGET of the server being installed.
	Use this only when setting up an HA replica of an edge server.

 -t <ServerType>
	Specify the server type, overriding the SERVER_TYPE setting in the config
	file.  Valid values are:
	* p4d_master - A master/commit server.
	* p4d_replica - A replica with all metadata from the master (not
	  filtered in any way).
	* p4d_filtered_replica - A filtered replica or filtered forwarding
	  replica.
	* p4d_edge - An edge server.
	* p4d_edge_replica - Replica of an edge server. If used,
	  '-S <TargetServerID>' is required.
	* p4broker - An SDP host running only a standalone p4broker, with no p4d.
	* p4proxy - An SDP host running only a standalone p4p with no p4d.

 -tp <TargetPort>
	Specify the target port. Use only if ServerType is p4proxy and p4broker.

 -lp <ListenPort>
	Specify the listen port. Use only if ServerType is p4proxy and p4broker.

 -I [<svc>[,<svc2>]]
	Specify additional init scripts to be added to /p4/<instance>/bin
	for the instance.

	By default, the p4p service is installed only if '-t p4proxy' is
	specified, and p4dtg is never installed by default.  Valid values
	to specify are 'p4p' and 'dtg' (for the P4DTG init script).

	If services are not installed by default, they can be added later
	using templates in /p4/common/etc/init.d. Also, templates for
	systemd service files are supplied in /p4/common/etc/systemd/system.

 -MDD /bigdisk
 -MCD /ckps
 -MLG /jnl
 -MDB1 /db1
 -MDB2 /db2
	Specify the '-M*' optons to specify mount points, overriding
	DD/CD/LG/DB1/DB2 settings in the config file.  Sample:

	-MDD /bigdisk -MLG /jnl -MDB1 /fast

	If -MDB2 is not specified, it is set the the same value as -MDB1 if
	that is set, or else it defaults to the same default value as DB1.

 -c <CfgFile>
	Specify the path to the configuration file to use, overriding the
	default logic of finding the file based on naming convention.

 -f	Specify -f 'fast mode' to skip chown/chmod commands on depot files.
	This should only be used when you are certain the ownership and
	permissions are correct, and if you have large amounts of existing
	data for which the chown/chmod of the directory tree would be
	slow.

 -p	Specify '-p' to halt processing after preflight checks are complete,
	and before actual processing starts. By default, processing starts
	immediately upon successful completion of preflight checks.

 -no_init
	Specify '-no_init' to avoid any service configuration, which
	is done by default if running as root (using systemd if available,
	otherwise SysV).  If '-no_init' is used, then neither systemd nor
	SysV init mechanism is configured for installed services.

	This option is implied if not running as root.

	This option is implied if '-test' is used.

 -no_systemd
	Specify '-no_systemd' to avoid using systemd, even if it
	appears to be available. By default, systemd is used if it
	appears to be available.

	This is helpful in operating in containerized test environments
	where systemd does not work even if it appears to be available.

	This option is implied if the systemctl command is not available
	in the PATH of the root user.

	This option is implied if '-no_init' is used.

 -no_enable
	Specify '-no_enable' to avoid enabling systemd services to start
	automatically after a reboot. If this option is used, systemd
	services will still be created, allowing services to be manually
	started and stopped.

	Specifically, this options means the 'systemctl enable' command
	is not run for generated services.

 -no_cron
 	Specify '-no_cron' to avoid loading the crontab.

	A crontab file is generated in the /p4 directory, but
	but with '-no_cron, this file is not loaded as the active crontab.

 -no_firewall
 	Specify '-no_firewall' to avoid attempting firewall configuration.

	By default, if the firewalld service is found to be running,
	it is configured so that the ports for p4d and p4broker are
	open.

 -fs	Specify '-full' when calling gen_sudoers.sh to install a new, full
	sudoers file. This option is only available if running as root.

	This option is mutually exclusive with '-ls'.

	See 'gen_sudoers.sh -man' for more info.

 -ls	Specify '-limited' when calling gen_sudoers.sh to install a new,
	limited sudoers file. This option is only available if running as
	root.

	This option is mutually exclusive with '-fs'.

	See 'gen_sudoers.sh -man' for more info.

 -L <log>
	Specify the path to a log file, or the special value 'off' to disable
	logging.  By default, all output (stdout and stderr) goes to this file
	in the current directory:

	mkdirs.<instance>.<datestamp>.log

	NOTE: This script is self-logging.  That is, output displayed on the
	screen is simultaneously captured in the log file.  Do not run this
	script with redirection operators like '> log' or '2>&1', and do not
	use 'tee'.

DEBUGGING OPTIONS:
 -test
 	Specify '-test' to execute a simulated install to /tmp/p4 as the install
	root (rather than /p4), and with the mount point directories specified in
	the configuration file prefixed with /tmp/hxmounts, defaulting to:
	* /tmp/hxmounts/hxdepots
	* /tmp/hxmounts/hxlogs
	* /tmp/hxmounts/hxmetadata

	This option implies '-no_init'.

 -clean
	Specify '-clean' with '-test' to clean up from prior test installs,
	which will result in removal of files/folders installed under /tmp/hxmounts
	and /tmp/p4.

 	Do not specify '-clean' if you want to test a series of installs.

 -n	No-Op.  In No-Op mode, no actions that affect data or structures are
 	taken.  Instead, commands that would be run are displayed.  This is
	an alternative to -test. Unlike '-p' which stops after the preflight
	checks, with '-n' more processing logic can be exercised, with greater
	detail about what commands that would be executed without '-n'.

 -d     Increase verbosity for debugging.

 -D     Set extreme debugging verbosity, using bash '-x' mode. Also implies -d.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message

FILES:
	The mkdirs.sh script uses a configuration file for many settings.  A
	sample file, mkdirs.cfg, is included with the SDP.  After determining
	your SDP instance name (e.g. '1' or 'abc'), create a configuration
	file for it named mkdirs.<N>.cfg, replacing 'N' with your instance.

	Running 'mkdirs.sh N' will load configuration settings from mkdirs.N.cfg.

UPGRADING SDP:
	This script can be useful in testing and upgrading to new versions of
	the SDP, when the '-test' flag is used.

EXAMPLES:
	Example 1: Setup of first instance

	Setup of the first instance on a machine using the default instance name,
	'1', executed after using sudo to become root:

	$ sudo su -
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ vi mkdirs.cfg

	Adjust settings as desired, e.g P4PORT, P4BROKERPORT, etc.

	$ ./mkdirs.sh 1

	A log will be generated, mkdirs.1.<timestamp>.log

	Example 2: Setup of additional instance named 'abc'.

	Setup a second instance on the machine, which will be a separate Helix
	Core instance with its own P4ROOT, its own set of users and
	changelists, and its own license file (copied from the master instance).

	Note that while the first run of mkdirs.sh on a given machine should be
	done as root, but subsequent instance additions can be done as the
	'perforce' user (or whatever operating system user accounts Perforce
	Helix services run as).

	$ sudo su - perforce
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ cp mkdirs.cfg mkdirs.abc.cfg
	$ chmod +w mkdirs.abc.cfg
	$ vi mkdirs.abc.cfg

	Adjust settings in mkdirs.abc.cfg as desired, e.g P4PORT, P4BROKERPORT, etc.

	$ ./mkdirs.sh abc

	A log will be generated, mkdirs.abc.<timestamp>.log

	Example 3: Setup of additional instance named 'alpha' to run a standalone p4p
	targeting commit.example.com:1666 and listening locally on port 1666.

	$ sudo su -
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ ./mkdirs.sh alpha -t p4proxy -tp commit.example.com:1666 -lp 1666

	Example 4: Setup of instance named '1' to run a standalone p4broker
	targeting commit.example.com:1666 and listening locally on port 1666.

	$ sudo su -
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ ./mkdirs.sh 1 -t p4broker -tp commit.example.com:1666 -lp 1666

	Example 5: Setup 2 instances A and B with limited sudoers on a fresh new machine:

	$ sudo su -
	$ cd /hxdepots/sdp/Server/Unix/setup
	$ cp mkdirs.cfg mkdirs.A.cfg

	Adjust settings in mkdirs.A.cfg as desired, e.g P4PORT, P4BROKERPORT, etc.

	$ cp mkdirs.A.cfg mkdirs.B.cfg

	Adjust settings in mkdirs.B.cfg as desired, e.g P4PORT, P4BROKERPORT, etc.
	Ensure port numbers do not conflict. Then generate Instance A:

	$ ./mkdirs.sh A -ls

	A log will be generated, mkdirs.A.<timestamp>.log

	Next generate instnace B, updating the limited sudoers to reference both
	instances.

	$ ./mkdirs.sh B -ls

SEE ALSO:
	See 'install_sdp.sh -man' for more info on installing on a new machine.

	See 'gen_sudoers.sh -man' for more info on generating/replacing sudoers.

	See template:
	* systemd service file templates: /p4/common/etc/systemd/system
	* firewalld templates: /p4/common/etc/firewalld
	* ufw firewall templates: /p4/common/etc/ufw
	* Init script templates: /p4/common/etc/init.d

Usage:

   p4pcm.pl [-d "proxy_cache_dir"] [-tlow <low_threshold>] [-thigh <high_threshold>] [-n/-s]
or
   p4pcm.pl -h

This utility removes files in the proxy cache if the amount of free disk
space available to the cache falls below the low threshold (default 12).
It removes cache files based on time last accessed starting with the least
recently accessed continuing until either all files are deleted or the free disk
space available to the cache specified by the high threshold (default 25)
is reached. Specify numeric threshold values in kilobyte units (kb), or as a number
less than 100 to specify percentage of the total disk space available to
the cache. A high_threshold near the available disk space typically results in a
full clear of the cache defeating the purpose of a proxy.

The '-d "proxy_cache_dir"' argument is required unless $P4PCACHE is defined.
The -d argument takes precedence. proxy_cache_dir should be a fully rooted path
starting with '/'. Relative or local paths are fatal to tool operation.

The log is $LOGS/p4pcm.log if $LOGS is defined, else p4pcm.log in the current
directory.

The removal nomination file list is $LOGS/p4pcm.nomlist if $LOGS is
defined, else p4pcm.nomlist in the current directory. The nomination list
file contains an access time ordered list of all cache files. If the nomination
list file exists at the start of this tool, the tool exits assuming a separate
run is currently in progress. The nomination list file is deleted when the tool
completes operation unless the '-s' argument is specified.

Use '-n' or '-s' to show what files would be removed. '-s' also causes the
nomination list file to remain undeleted. The nomination list file must be
manually removed prior to a subsequent successful use of this tool.