README.html #9

<h1>What is the Server Deployment Package (SDP)?</h1>

<h2>Supported Open Source Software</h2>

<p><em>When deployed by Perforce Consulting</em>, the core functionality of the SDP (including init scripts, offline checkpoint mechanism, and backup-related scripts) are officially supported by Perforce Support.  Also, using the SDP with its standard structure makes your environment easier to Support.  Note that supplemental scripts, such as Helix Management System (HMS) and various maintenance scripts included with the SDP are supported by Perforce Consulting only.</p>

<p>The SDP is open source software (see the <a href="https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/main/LICENSE">LICENSE</a>), and accepts contributions (pending code review) from the user community.</p>

<h2>Overview</h2>

<p>The Server Deployment Package (SDP) is the implementation of Perforce's recommendations for operating and managing a production Perforce Helix Core Version Control System.</p>

<h2>Simplified Management</h2>

<p>The SDP provides a set of management scripts for operating various Perforce Helix Core topology components, such as:</p>

<ul>
<li><strong>p4d</strong> - Helix Core Version Control System, including master/commit servers, replicas, and edge servers.</li>
<li><strong>p4broker</strong> - A topology component of Helix Core</li>
<li><strong>p4p</strong> - Perforce Helix Proxy</li>
<li><strong>p4dtg</strong> - Perforce Defect Tracking Gateway</li>
<li>and others ...</li>
</ul>

<p>For <strong>p4d</strong> in particular, the SDP provides an implementation of best practices suited for production environments, such as performing daily <em>offline checkpoints</em> <strong>with zero downtime</strong>, rotating logs and journals routinely.  The SDP provides optimized init scripts that do database sanity/safety check before starting the server. Init scripts support both SysV and systemd init mechanisms.</p>

<h2>High Availability (HA)</h2>

<p>The SDP supports a variety of methods for achieving high availability that benefit simple, single-server environments and sophisticated global enterprise environments, and all in between.  Standardized and documented recovery procedures, offline databases, and optional replication (made easier with the SDP) all make HA easier to achieve.</p>

<h2>Disaster Recovery (DR)</h2>

<p>Just as with HA, the SDP makes it easier to implement robust, reliable DR strategies.</p>

<h2>Fast and Safe Upgrades</h2>

<p>The SDP makes upgrades of Helix server products fast and easy, regardless of the scale of your environment.  By taking advantage of the offline database structure, the SDP enables upgrades with just a few minutes of downtime (enough to do a journal rotation).</p>

<h2>Production Focus</h2>

<p>The p4d executable, without any configuration, is optimized for evaluation and demonstration purposes.  While it has many features that can make it reliable and robust, many of those features must be configured.  The SDP is intended to be used for production environments or more realistic Proof of Concept installations.</p>

<p>See the <a href="https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer">Helix Installer</a> which makes quick work of installing the SDP for PoC evaluation with a production-like setup, complete with the Sample Depot training data set.</p>

<h2>Best Practice Configurables</h2>

<p>The SDP also maintains, on an ongoing basis, a set of best-practices configuration settings (e.g. 'configurables' and environment settings) suited to production environments, which often differ from p4d defaults.</p>

<h2>Optimal Performance, Data Safety, and Simplified Backup</h2>

<p>The SDP provides a standard structure for operating Perforce that is optimized for performance, scalability, and ease of backup. The SDP Guide includes documentation that promotes volume layout and storage architecture best practices.</p>

<p>To simplify backup, all digital assets are in a single directory.  Live databases that cannot be backed up live in another, and active log files in another.  For any scale environment, this makes it easy to back up the server's critical data.  For larger scale environments, these directories become storage volume mount points, with the layout promoting best performance and data safety.</p>

<h2>Sophistication Simplified</h2>

<p>In addition to the basics, the SDP provides structure, standards, guidance, and automation that greatly simplify deployment and management of sophisticated Helix global topologies.  For example, the SDP includes tools that greatly simplify complex tasks such as creating new replicas, backing up edge servers without down time, etc.</p>

<h2>Multi-Instance Management</h2>

<p>The SDP supports operation of multiple instances of Helix on a machine, allowing each instance to have differing configuration, e.g. different:</p>

<ul>
<li>versions of software components (p4d, p4broker, etc.)</li>
<li>case sensitivity setting (sensitive/insensitive)</li>
<li>Unicode setting (enabled/disabled)</li>
</ul>

<h2>Shell Environment Management</h2>

<p>The SDP provides standard mechanism for defining a controlled shell environment.  This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation.</p>

<h2>Perforce Automation Standards</h2>

<p>The SDP defines a standard for installing and configuring derived APIs.  For example, Python is built with P4Python in /p4/common/python, and Perl with P4Perl in /p4/common/perl.</p>

<h2>Stock vs. Custom SDP</h2>

<p>Many Perforce customers use the SDP as-is.  Others benefit by incorporating concepts and ideas from the SDP into their own environments, e.g. for development of custom policy enforcement triggers, systems integrations, promoting or enforcing local naming conventions, etc.</p>

<h2>Helix Management System</h2>

<p>The Helix Management System (HMS) is the latest addition to the SDP, and is included with the SDP.  HMS provides:</p>

<ul>
<li>Helix Topology Definition Standard</li>
<li>Centralized Helix Management</li>
<li>Tight Ship Management of SDP in a global topology</li>
<li>Automation of Failover Execution (Human Initiated)</li>
<li>Simplified Topology-Wide Update/Upgrade (future, not yet implemented)</li>
</ul>

<h1>Bugs, Enhancements, and Future Plans</h1>

<p>Perforce Jobs are used to file bugs and request enhancements to the SDP.  The list of jobs (other than closed ones) can be viewed <a href="https://swarm.workshop.perforce.com/projects/perforce-software-sdp/jobs" title="SDP Open Jobs">here</a>.</p>

<p>You can see all jobs (including closed ones if desired) from the command line interface.  Connect to The Workshop and run the <code>p4 jobs</code> command, something like this:</p>

<p><code>
p4 jobs -l -e "Project=perforce-software-sdp ^Status=closed"
</code></p>

<p>While there is currently no formal roadmap for the SDP, it is actively maintained and continues to evolve alongside P4D and other Helix Core components it manages.  Jobs are occasionally reviewed and addressed as time allows for consultants and contributors, sometimes driven by customer-funded consulting projects.</p>

<h1>Contributing</h1>

<p>Contributions to the SDP are most welcome!  Note that contributing requires a basic user knowledge of working with Perforce.</p>

<h2>Contributing by Shelving (Small Changes)</h2>

<p>For contributing small changes, contribute by shelving changelists to the 'dev' branch your workspace.</p>

<p>All registered Workshop users have <strong>open</strong> access (but not <strong>write</strong>) to the SDP dev branch.  This allows shelving (but not submitting) updates to the SDP dev branch path:</p>

<p><code>
//guest/perforce_software/sdp/dev/...
</code>
To contribute by shelving, create a workspace that maps that path, named something like '<em>your_user</em>.<em>your_hostname</em>.sdp'.  Make your changes in that workspace, provide a good changelist description, and include a <code>#review</code> tag in your changelist description.</p>

<p>When you are ready, shelve the changelist.  Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.</p>

<h2>Contributing by Branching (Bigger Changes)</h2>

<p>Optionally, you may also branch the SDP dev branch, or some subset of it, into your own guest area.  Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this:</p>

<p>```
Branch: your_name-sdp</p>

<p>Owner: your_name</p>

<p>Description:
    SDP Updates by your_name.</p>

<p>Options:    unlocked</p>

<p>View:
    //guest/perforce<em>software/sdp/dev/.. //guest/your</em>name/sdp/dev/...
```</p>

<p>If you branch a subset, maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date.</p>

<p>Edit, test and submit changes in your branch.  Keep your code up to date with other changes by merging down from the shared dev branch to your dev branch, e.g. by doing:</p>

<p><code>
p4 merge -b your_name-sdp
</code></p>

<p>... followed by the usual resolve and submit.</p>

<p>When you'e ready to copy your versions up to our dev branch, do like so:</p>

<p><code>
p4 change
p4 copy -c YourCL -r -b your_name-sdp
</code></p>

<p>Be sure that your changelist has a clear and accurate description, and that the description contains a <code>#review</code> tag on a line by itself.  That will trigger a code review process when you shelve the change, like so:</p>

<p><code>
p4 shelve -c YourCL
</code></p>

<p>Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.</p>

<h2>Guidelines for SDP Contributors</h2>

<ul>
<li>Communicate! Use Swarm or email! Let folks know what you're planning before you work too hard on something that might already be underway.</li>
<li>File a job (Project=<strong>perforce-software-sdp</strong>) first.  That's not required, but increases the transparency of what you're doing, and helps promote it.</li>
<li>We prefer changes that are generic, and have a wide appeal.</li>
<li>The core of the SDP, the scripts that do offline checkpoints for p4d, prioritize Perforce best practices over scripting magic.  Don't get too fancy! :)</li>
</ul>

<h1>Versioning the SDP Locally</h1>

<h2>Tight Ship Management</h2>

<p>In line with the "Version Everything" mindset, the SDP itself should be versioned in your environment.  The best way to version the SDP is to do a "Tight Ship" installation, using the Helix Management System (HMS) tight ship setup script, <a href="https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/dev/doc/hms_ts_setup.command_summary.txt">hms<em>ts</em>setup.sh</a>.</p>

<p>This script, which is included in the SDP, initializes a separate <code>/p4/hms</code> instance of the SDP to manage the SDP.  It will get you started on the path to enterprise-grade Helix Core server management.  It creates an initial workspace for managing the SDP files, creates a connection to The Workshop, and enables a 'fetch' and 'merge' workflow to use the power of Perforce merging to greatly simplify taking future updates to the SDP.</p>

<p>When Tight Ship management is used, we recommend deploying an optional Swarm instance for the hms instance, to provide the usual Swarm benefits (code review, email notification, transparency of changes, etc.).</p>

<h2>Upgrading Older Existing SDP Installations</h2>

<p>If you have an earlier version of the Server Deployment Package (SDP) installed and are <em>not</em> using Tight Ship style of management, you'll want to be aware of the <code>-test</code> flag to the SDP setup script, <code>mkdirs.sh</code>.  The following update instructions assume a simple, single-server topology. (For older SDP installations, this path may be <code>/depotdata/sdp</code>.)</p>

<p>Get the new software from The Workshop, which you'll find in the form of a SDP.Unix.*.tgz zipped tar file.  Put the tar file in the <code>/hxdepots</code> folder on your server.  <strong>BE SURE TO MOVE THE OLD</strong> <code>/hxdepots/sdp</code> folder aside first if it is there, e.g. to <code>/hxdepots/sdp.old.2014-10-08</code>.</p>

<p>Then unpack it like this:</p>

<p><code>
    tar -xzvpf SDP.Unix.2017.5.####.tgz
    cd /hxdepots/sdp/Unix/setup
</code></p>

<p>Modify the mkdirs.sh script to suit your environment (possibly borrowing values from your old <code>mkdirs.sh</code> script).</p>

<p>Then run it:</p>

<pre><code>mkdirs.sh 1 -test
</code></pre>

<p>That'll do a mock install, creating a <code><strong>/tmp/</strong>p4</code> structure, with folders like <code>/tmp/p4/common/bin</code> and <code>/tmp/p4/1/bin</code>.  The output will suggest some <code>diff -r</code> commands to run to to view the diffs and apply updates manually, being careful not to lose any customizations or local configuration changes you may have made.</p>

<p>If the SDP itself is versioned in Perforce in your environment, you can also put the new things from /tmp/p4 into your Perforce server where the SDP is.  Commonly, there is a branch for receiving pristine updates of the generic SDP, e.g. 'p4c', which allow you to use Perforce merging to merge update into your local SDP mainline.</p>

<p>Contact <a href="mailto:consulting@perforce.com">Perforce Consulting Services</a> to ask about engaging professional services to upgrade the SDP for you.</p>

<p>A few quick things to be aware of:</p>

<ul>
<li><p>The old <code>p4<em>1</em>init</code> script (more generically <code>p4<em>instance</em>init</code>) was renamed to <code>p4<strong>d</strong><em>1</em>init</code>.  This will require updates to init script or symlink, e.g. in <code>/etc/init.d</code>.</p></li>
<li><p>Once you upgrade p4d to 2014.1+, you can do away with the SDP trigger that updates depot spec map fields.  You can clear the <code>Map:</code> fields of depots, and use the <code>server.depot.root</code> configurable.  Going forward, you'd only use <code>Map:</code> fields for depots that really do require a unique path, e.g. archive depots.</p></li>
</ul>

<h1>SDP Structure</h1>

<p>The SDP structure enables two major views of a Perforce server, the system administrator/storage architect view, and the application administrator view.  THe system admin sees this structure:</p>

<pre><code>/hxdepots
/hxmetadata1
/hxmetadata2
/hxlogs
</code></pre>

<p>System administrators know they need to backup <code>/hxdepots</code>.  Backing up <code>/hxlogs</code> is optional, and /hxmetadata1 &amp; /hxmetadata2 are to be avoided by backup utilities, virus scanners, and anything else that might touch files in that area.  This view is designed for optimizing for performance, as each of those folders has different performance profiles and usage patterns demanded by p4d.  This view is not concerned with how many instances there are, or how they vary.</p>

<p>Application administrators (Perforce Admins), though a structure of symlinks, see Perforce from a view like this:</p>

<p>```
/p4/1/bin
/p4/1/depots
/p4/1/root
/p4/1/tmp
/p4/common/config
/p4/common/bin
/p4/ssl</p>

<p>```</p>

<p>This structure illustrates an instance named '1' (it need not be numeric), and a /p4/common area containing files common to all instances.  Each of those folders is physically stored under an appropriate storage folder or volume as defined in the system admin view above.</p>

<p>The <code>/p4/<em>instance</em>/bin</code> folder defines which Perforce products are active for that instance on the current machine (p4d, p4broker, p4p, etc.).  The <strong>root</strong> folder contains P4ROOT, while <strong>depots</strong> contains archive files.</p>

<h1>The Windows SDP</h1>

<p>There are two flavors of the SDP, Unix and Windows.  Both are available in the same folder structure in the Workshop.</p>

<h1>SDP Code Quality</h1>

<p>An automated regression test suite and pre-commit code review processes help ensure code quality.</p>

<p>See more details in sdp/test/README.md</p>