The Server Deployment Package (SDP) provides a set of management scripts for operating various Perforce software products, such as:
For p4d in particular, the SDP provides a sample implementation of best practices, such as performing offline checkpoints, rotating logs and journals routinely, and doing a database sanity/safety check before starting the server.
The SDP provides a structure for operating multiple instances of Perforce on a given machine, allowing for the possibility of each instance having different configuration aspects, e.g.
Perforce Jobs are used to file bugs and request enhancements to the SDP. The list of jobs (other than closed ones) can be viewed here.
You can see all jobs (including closed ones if desired) from the command line interface. Connect to The Workshop and run the <code>p4 jobs</code> command, something like this:
<code>p4 jobs -l -e "Project=perforce-software-sdp ^status=closed"</code>
There is no formal roadmap for the SDP, though it continues to evolve alongside P4D and other Helix components it manages.
Contributions to the SDP are most welcome! Note that contributing requires a basic user knowledge of working with Perforce.
All registered Workshop users have open access (but not write) to the SDP, specifically to this path:
<code>//guest/perforce_software/sdp/main/...</code>
The open access level confers the ability to edit and shelve changes for a pre-commit review process. We'll review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and also include the tag/text "[review]" in your changelist description to initiate a Swarm code review.
Optionally, you may also branch the SDP folder, or some subset of it, into your own guest area. Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this:
Branch: your_name-sdp
Owner: your_name
Description:
SDP Updates by your_name.
Options: unlocked
View:
//guest/perforce_software/sdp/main/Server/Unix/etc/... //guest/your_name/sdp/main/Server/Unix/etc/...If you branch a subset, it's a good idea to maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date.
Edit, test and submit changes in your branch. Then when you are done, merge your changes back into our mainline, shelving them for review.
In line with the "Version Everything" mindset, the SDP itself should be versioned in your environment.
Commonly the SDP is versioned in a depot such as <code>//Perforce</code> or <code>//p4admin</code>, with a structure like this: <PRE> //p4admin/p4c/sdp/... //p4admin/main/sdp/... //p4admin/host/hostname/p4/1/bin/... </PRE>
The p4c and main folders are branches. The p4c branch is designated for receiving generic SDP updates from The Workshop. The main branch is where local customizations and/or configuration changes are applied.
The host folder contains instance-specific and host-specific files, which effectively communicate part of your topology. These define which services are active on which hosts.
On the Perforce server machines, basic workspaces are used to manage the SDP.
Recent versions of the SDP contain a <code>.p4ignore</code> file that helps avoid versioning of files that do not belong in version control, things like password files, editor temp files, and generated/compiled Python cache files.
When installing Perforce Swarm or Perforce Git Fusion in the SDP structure, they can and should be deployed in a way so as to benefit from the SDP structure, e.g. instances of Swarm and/or Git Fusion to run against each p4d instance.
Details for doing this are not yet incorporated the mainline SDP, though they have been deployed at customer sites. However, the basic ideas are fairly straightforward. As a hint for Swarm, you'll want to apply the <code>SWARM_DATA_PATH</code> in the httpd configuration to reference Swarm for each SDP instance.
If you have an earlier version of the Server Deployment Package (SDP) installed, you'll want to be aware of the new <code>-test</code> flag to the SDP setup script, <code>mkdirs.sh</code>. The following update instructions assume a simple, single-server topology.
Get the new software from The Workshop, which you'll find in the form of a SDP.Unix.*.tgz zipped tar file. Put the tar file in the <code>/depotdata</code> folder on your server. BE SURE TO MOVE THE OLD <code>/depotdata/sdp</code> folder aside first if it is there, e.g. to <code>/depotdata/sdp.old.2014-10-08</code>.
Then unpack it like this:
tar -xzvpf SDP.Unix.2015.1.####.tgz
cd /depotdata/sdp/Unix/setupModify the mkdirs.sh script to suit your environment (possibly borrowing values from your old <code>mkdirs.sh</code> script).
Then run it:
mkdirs.sh 1 -testThat'll do a mock install, creating a <code>/tmp/p4</code> structure, with folders like <code>/tmp/p4/common/bin</code> and <code>/tmp/p4/1/bin</code>. The output will suggest some <code>diff -r</code> commands to run to to view the diffs and apply updates manually, being careful not to lose any customizations or local configuration changes you may have made.
If the SDP itself is versioned in Perforce in your environment, you can also put the new things from /tmp/p4 into your Perforce server where the SDP is. Commonly, there is a branch for receiving pristine updates of the generic SDP, e.g. 'p4c', which allow you to use Perforce merging to merge update into your local SDP mainline.
Contact Perforce Consulting Services to ask about engaging professional services to upgrade the SDP for you.
A few quick things to be aware of:
The old <code>p4_1_init</code> script (more generically <code>p4_instance_init</code>) was renamed to <code>p4d_1_init</code>. This will require updates to init script or symlink, e.g. in <code>/etc/init.d</code>.
Once you upgrade p4d to 2014.1+, you can do away with the SDP trigger that updates depot spec map fields. You can clear the <code>Map:</code> fields of depots, and use the <code>server.depot.root</code> configurable. Going forward, you'd only use <code>Map:</code> fields for depots that really do require a unique path, e.g. archive depots.
The SDP structure enables two major views of a Perforce server, the system administrator/storage architect view, and the application administrator view. THe system admin sees this structure:
/depotdata
/metadata
/logsSystem administrators know they need to backup <code>/depotdata</code>. Backing up <code>/logs</code> is optional, and /metadata is to be avoided by backup utilities, virus scanners, and anything else that might touch files in that area. This view is designed for optimizing for performance, as each of those folders has different performance profiles and usage patterns demanded by p4d. This view is not concerned with how many instances there are, or how they vary.
Application administrators (Perforce Admins), though a structure of symlinks, see Perforce from a view like this:
/p4/1/bin
/p4/1/config
/p4/1/depots
/p4/1/root
/p4/1/ssl
/p4/1/tmp
/p4/common/binThis structure illustrates an instanced named '1' (it need not be numeric), as a /p4/common area which contains files common to all instances. Each of those folders is physically stored under an appropriate storage folder or volume as defined in the system admin view above.
The <code>/p4/instance/bin</code> folder defines which Perforce products are active for that instance on the current machine (p4d, p4broker, p4p, etc.). The root folder contains P4ROOT, while depots contains archive files.
The SDP provides standard mechanism for defining a controlled shell environment. This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation, such as Perforce standards guidance and policy enforcement scripts (e.g. p4d triggers or p4broker filter scripts).
For advanced deployments, the SDP defines a standard for installing and configuring derived APIs, such as P4Python into /p4/common/python.
Many Perforce customers use the SDP as-is. Others benefit by incorporating concepts and ideas from the SDP into their own environments.
There are two flavors of the SDP, Unix and Windows. Both are available in the same folder structure in the Workshop.
There is automated tests using Vagrant for the Unix variant of the SDP. See test/README.md for details.
What is the Server Deployment Package (SDP)? === Overview --- The Server Deployment Package (SDP) provides a set of management scripts for operating various Perforce software products, such as: * p4d * p4broker * p4p * p4web * p4dtg * p4ftpd For **p4d** in particular, the SDP provides a sample implementation of best practices, such as performing _offline checkpoints_, rotating logs and journals routinely, and doing a database sanity/safety check before starting the server. The SDP provides a structure for operating multiple instances of Perforce on a given machine, allowing for the possibility of each instance having different configuration aspects, e.g. * versions of software components (p4d, p4broker, etc.) * case sensitivity setting (sensitive/insensitive) * Unicode setting (enabled/disabled) Bugs, Enhancements, and Future Plans === Perforce Jobs are used to file bugs and request enhancements to the SDP. The list of jobs (other than closed ones) can be viewed [here](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/jobs "SDP Open Jobs"). You can see all jobs (including closed ones if desired) from the command line interface. Connect to The Workshop and run the <code>p4 jobs</code> command, something like this: <code>p4 jobs -l -e "Project=perforce-software-sdp ^status=closed"</code> There is no formal roadmap for the SDP, though it continues to evolve alongside P4D and other Helix components it manages. Contributing === Contributions to the SDP are most welcome! Note that contributing requires a basic user knowledge of working with Perforce. Contributing by Shelving --- All registered Workshop users have **open** access (but not **write**) to the SDP, specifically to this path: <code>//guest/perforce_software/sdp/main/...</code> The **open** access level confers the ability to edit and **shelve** changes for a pre-commit review process. We'll review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and also include the tag/text "[review]" in your changelist description to initiate a Swarm code review. Contributing by Branching --- Optionally, you may also branch the SDP folder, or some subset of it, into your own guest area. Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this: Branch: your_name-sdp Owner: your_name Description: SDP Updates by your_name. Options: unlocked View: //guest/perforce_software/sdp/main/Server/Unix/etc/... //guest/your_name/sdp/main/Server/Unix/etc/... If you branch a subset, it's a good idea to maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date. Edit, test and submit changes in your branch. Then when you are done, merge your changes back into our mainline, shelving them for review. Guidelines for SDP Contributors --- * Communicate using Swarm! Let folks know what you're planning before you work to hard on something that might already be underway. * File a job (Project=**perforce-software-sdp**) first. That's not required, but increases the transparency of what you're doing, and helps promote it. * We prefer changes that are generic, and have a wide appeal. * The core of the SDP, the scripts that do offline checkpoints for p4d, prioritize Perforce best practices over scripting magic. Don't get too fancy! :) Versioning the SDP Locally === In line with the "Version Everything" mindset, the SDP itself should be versioned in your environment. Commonly the SDP is versioned in a depot such as <code>//Perforce</code> or <code>//p4admin</code>, with a structure like this: <PRE> //p4admin/p4c/sdp/... //p4admin/main/sdp/... //p4admin/host/_hostname_/p4/1/bin/... </PRE> The **p4c** and **main** folders are branches. The **p4c** branch is designated for receiving generic SDP updates from The Workshop. The **main** branch is where local customizations and/or configuration changes are applied. The **host** folder contains instance-specific and host-specific files, which effectively communicate part of your topology. These define which services are active on which hosts. On the Perforce server machines, basic workspaces are used to manage the SDP. Recent versions of the SDP contain a <code>.p4ignore</code> file that helps avoid versioning of files that do not belong in version control, things like password files, editor temp files, and generated/compiled Python cache files. Perforce Swarm and Perforce Git Fusion --- When installing Perforce Swarm or Perforce Git Fusion in the SDP structure, they can and should be deployed in a way so as to benefit from the SDP structure, e.g. instances of Swarm and/or Git Fusion to run against each p4d instance. Details for doing this are not yet incorporated the mainline SDP, though they have been deployed at customer sites. However, the basic ideas are fairly straightforward. As a hint for Swarm, you'll want to apply the <code>SWARM_DATA_PATH</code> in the httpd configuration to reference Swarm for each SDP instance. Upgrading Existing SDP Installations === If you have an earlier version of the Server Deployment Package (SDP) installed, you'll want to be aware of the new <code>-test</code> flag to the SDP setup script, <code>mkdirs.sh</code>. The following update instructions assume a simple, single-server topology. Get the new software from The Workshop, which you'll find in the form of a SDP.Unix.*.tgz zipped tar file. Put the tar file in the <code>/depotdata</code> folder on your server. **BE SURE TO MOVE THE OLD** <code>/depotdata/sdp</code> folder aside first if it is there, e.g. to <code>/depotdata/sdp.old.2014-10-08</code>. Then unpack it like this: tar -xzvpf SDP.Unix.2015.1.####.tgz cd /depotdata/sdp/Unix/setup Modify the mkdirs.sh script to suit your environment (possibly borrowing values from your old <code>mkdirs.sh</code> script). Then run it: mkdirs.sh 1 -test That'll do a mock install, creating a <code>**/tmp/**p4</code> structure, with folders like <code>/tmp/p4/common/bin</code> and <code>/tmp/p4/1/bin</code>. The output will suggest some <code>diff -r</code> commands to run to to view the diffs and apply updates manually, being careful not to lose any customizations or local configuration changes you may have made. If the SDP itself is versioned in Perforce in your environment, you can also put the new things from /tmp/p4 into your Perforce server where the SDP is. Commonly, there is a branch for receiving pristine updates of the generic SDP, e.g. 'p4c', which allow you to use Perforce merging to merge update into your local SDP mainline. Contact [Perforce Consulting Services](mailto:consulting@perforce.com) to ask about engaging professional services to upgrade the SDP for you. A few quick things to be aware of: * The old <code>p4_1_init</code> script (more generically <code>p4_instance_init</code>) was renamed to <code>p4**d**_1_init</code>. This will require updates to init script or symlink, e.g. in <code>/etc/init.d</code>. * Once you upgrade p4d to 2014.1+, you can do away with the SDP trigger that updates depot spec map fields. You can clear the <code>Map:</code> fields of depots, and use the <code>server.depot.root</code> configurable. Going forward, you'd only use <code>Map:</code> fields for depots that really do require a unique path, e.g. archive depots. SDP Structure === The SDP structure enables two major views of a Perforce server, the system administrator/storage architect view, and the application administrator view. THe system admin sees this structure: /depotdata /metadata /logs System administrators know they need to backup <code>/depotdata</code>. Backing up <code>/logs</code> is optional, and /metadata is to be avoided by backup utilities, virus scanners, and anything else that might touch files in that area. This view is designed for optimizing for performance, as each of those folders has different performance profiles and usage patterns demanded by p4d. This view is not concerned with how many instances there are, or how they vary. Application administrators (Perforce Admins), though a structure of symlinks, see Perforce from a view like this: /p4/1/bin /p4/1/config /p4/1/depots /p4/1/root /p4/1/ssl /p4/1/tmp /p4/common/bin This structure illustrates an instanced named '1' (it need not be numeric), as a /p4/common area which contains files common to all instances. Each of those folders is physically stored under an appropriate storage folder or volume as defined in the system admin view above. The <code>/p4/_instance_/bin</code> folder defines which Perforce products are active for that instance on the current machine (p4d, p4broker, p4p, etc.). The **root** folder contains P4ROOT, while **depots** contains archive files. The SDP provides standard mechanism for defining a controlled shell environment. This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation, such as Perforce standards guidance and policy enforcement scripts (e.g. p4d triggers or p4broker filter scripts). For advanced deployments, the SDP defines a standard for installing and configuring derived APIs, such as P4Python into /p4/common/python. Many Perforce customers use the SDP as-is. Others benefit by incorporating concepts and ideas from the SDP into their own environments. The Windows SDP === There are two flavors of the SDP, Unix and Windows. Both are available in the same folder structure in the Workshop. Testing the SDP === There is automated tests using Vagrant for the Unix variant of the SDP. See test/README.md for details.
| # | Change | User | Description | Committed | |
|---|---|---|---|---|---|
| #23 | 28861 | C. Thomas Tyler | Merged patch to README.md. | ||
| #22 | 28852 | C. Thomas Tyler |
Release notes corrections and tweaks to SDP home page related to the major version change.. |
||
| #21 | 28385 | C. Thomas Tyler |
Updated README for coming r21.2 release. Enhanced product description to emphazise key SDP benefits, better describing the value of the Offline DB. #review-28386 |
||
| #20 | 27482 | C. Thomas Tyler |
README.md updates: * Added Documentation References section, with less clutter and easier links to PDF/HTML versions of key docs, including the upgrade guide. * Enhanced "Sophistication Simplified" section. * Simplified Support status text. * General grammer corrections. * Removed a broken link. * Simpflified Contribing sections. * Enhanced formatting/TOC levels, moved several sections under new "Features and Benefits" section. #review-27483 |
||
| #19 | 27444 | C. Thomas Tyler | Fixed issue with broken doc links. | ||
| #18 | 27015 | C. Thomas Tyler |
Replaced notes in README.md re: upgrading SDP with ref to new SDP Upgrade Guide doc. The doc URL references not-yet-existing SDP r20.1 branch. |
||
| #17 | 26781 | Robert Cowham | Reference new docs. | ||
| #16 | 26652 | Robert Cowham |
This is Tom's change: Introduced new 'Unsupported' directory to clarify that some files in the SDP are not officially supported. These files are samples for illustration, to provide examples, or are deprecated but not yet ready for removal from the package. The Maintenance and many SDP triggers have been moved under here, along with other SDP scripts and triggers. Added comments to p4_vars indicating that it should not be edited directly. Added reference to an optional site_global_vars file that, if it exists, will be sourced to provide global user settings without needing to edit p4_vars. As an exception to the refactoring, the totalusers.py Maintenance script will be moved to indicate that it is supported. Removed settings to support long-sunset P4Web from supported structure. Structure under new .../Unsupported folder is: Samples/bin Sample scripts. Samples/triggers Sample trigger scripts. Samples/triggers/tests Sample trigger script tests. Samples/broker Sample broker filter scripts. Deprecated/triggers Deprecated triggers. To Do in a subsequent change: Make corresponding doc changes. |
||
| #15 | 26595 | C. Thomas Tyler | Minor updates to README.md, WIP. | ||
| #14 | 25591 | C. Thomas Tyler |
Clarified that Maintenance folder scripts are samples and are NOT officially supported. Updated references to HMS now that it has been split back out into a separate product. |
||
| #13 | 25155 | Robert Cowham | Refer to test/README.md | ||
| #12 | 25043 | C. Thomas Tyler |
Prepared patch SDP/MultiArch/2018.1/23583.p2 Changes: * In README, moved 'Supported Open Source Software' section to the top of the page, to make Support position more visible. * Copyright updated to 2019. * Cleanup of excess script recreate_db_sync_replica.sh * Re-removal of previously deleted script recreate_db_checkpoint.sh, and corresponding removal from test suite. * This patch tarball will also contain some Docker test suite updates already staged in main. By-passing review to trigger an automated test. |
||
| #11 | 24511 | C. Thomas Tyler |
Fixed an obvious doc typo in README files. Bypassing code review. |
||
| #10 | 23353 | C. Thomas Tyler | Fixed typo/grammatical error. | ||
| #9 | 23315 | C. Thomas Tyler |
Overhauled SDP README page to better convey its value, and much more strongly promote the SDP. This also documents the the nature of offical Perforce Support for the SDP, and the limitations of that Support to core SDP functionality. |
||
| #8 | 16784 | C. Thomas Tyler |
Routine Merge Down to dev from main using: p4 -s merge -n -b perforce_software-sdp-dev |
||
| #7 | 15797 | C. Thomas Tyler | Routine Merge Down to dev from main for SDP. | ||
| #6 | 13729 | C. Thomas Tyler | Tweaked formatting on README.md. | ||
| #5 | 12028 | C. Thomas Tyler | Refreshed SDP dev branch, merging down from main. | ||
| #4 | 11535 | Russell C. Jackson (Rusty) | Updated dev from main. | ||
| #3 | 11463 | Russell C. Jackson (Rusty) | Updated dev to prepare for Summit agreed changes. | ||
| #2 | 11038 | Robert Cowham | Catchup from Main | ||
| #1 | 10638 | C. Thomas Tyler | Populate perforce_software-sdp-dev. | ||
| //guest/perforce_software/sdp/main/README.md | |||||
| #1 | 10611 | C. Thomas Tyler | Added README file for the SDP in The Workshop. | ||