When deployed or certified by Perforce Consulting, the core functionality of the SDP is officially supported by Perforce Support, except for files in the Unsupported
directory tree (which are community supported).
The SDP is open source software (see the LICENSE), and accepts contributions (pending code review) from the user community.
The following documentation is included in the SDP:
sdp_upgrade.sh
.The following describes some of the many features and benefits of using the SDP to manage Helix Core.
The SDP provides a standard structure for operating Perforce that is optimized for performance, scalability, and ease of backup. The SDP Guide includes documentation that promotes volume layout and storage architecture best practices.
A key SDP feature is that it maintains an offline copy of live databases. This has many benefits:
To simplify backup, all digital assets are in a single directory. Live databases that should not be backed up directly live in another, and active log files in another. For any scale environment, this makes it easy to back up the server's critical data. These directories are a natural fit for storage volume mount points, with the layout promoting best performance and data safety.
The SDP also maintains, on an ongoing basis, a set of best-practices configuration settings (e.g. 'configurables' and environment settings) suited to production environments, which often differ from p4d defaults.
The SDP supports a variety of methods for achieving high availability that benefit simple, single-server environments and sophisticated global enterprise environments, and all in between. Standardized and documented recovery procedures, offline databases, and optional replication (made easier with the SDP) all make HA easier to achieve.
There is more detailed discussions and references in the SDP Guides noted above.
Just as with HA, the SDP makes it easier to implement robust, reliable DR strategies.
There is more detailed discussions and references in the SDP Guides noted above.
The SDP makes upgrades of Helix server products fast and easy, regardless of the scale of your environment. By taking advantage of the offline database structure, the SDP enables upgrades with just a few minutes of downtime (enough to do a journal rotation).
The p4d executable, without any configuration, is optimized for evaluation and demonstration purposes. While it has many features that can make it reliable and robust, many of those features must be configured. The SDP is intended to be used for production environments or more realistic Proof of Concept installations.
In addition to the basics, the SDP provides structure, standards, guidance, and automation that greatly simplify deployment and management of sophisticated Helix global topologies. For example, the SDP includes tools that greatly simplify complex tasks such as:
The SDP supports operation of multiple instances of Helix on a machine, allowing each instance to have differing configuration, e.g. different:
The SDP provides standard mechanism for defining a controlled shell environment. This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation.
The SDP defines a standard for installing and configuring derived APIs. For example, Python is built with P4Python in /p4/common/python, and Perl with P4Perl in /p4/common/perl.
Many Perforce customers use the SDP as-is. Others benefit by incorporating concepts and ideas from the SDP into their own environments, e.g. for development of custom policy enforcement triggers, systems integrations, promoting or enforcing local naming conventions, etc.
The Helix Management System (HMS) is a separate, community-supported product that builds on the SDP. This is likely to be of interest at installations with sophisticated global topologies and/or a fleet of SDP instances to manage.
Perforce Jobs are used to file bugs and request enhancements to the SDP. The list of jobs (other than closed ones) can be viewed here.
You can see all jobs (including closed ones if desired) from the command line interface. Connect to The Workshop and run the <code>p4 jobs</code> command, something like this:
p4 jobs -l -e "Project=perforce-software-sdp ^Status=closed"
While there is currently no formal roadmap for the SDP, it is actively maintained and continues to evolve alongside P4D and other Helix Core components it manages. Jobs are occasionally reviewed and addressed as time allows for consultants and contributors, sometimes driven by customer-funded consulting projects.
Contributions to the SDP are most welcome! Note that contributing requires a basic user knowledge of working with Perforce.
The SDP currently lives in a Classic (i.e. non-Stream) depot for historic reasons, in this path:
//guest/perforce_software/sdp/...
However, an empty //sdp stream depot exists and is used to simplify workspace creation and management.
Here is a sample procedure to set up a workspace. These instructions work on Linux/UNIX, MacOS, or Windows. This sample uses P4CONFIG files and creates a pub
folder to contain multiple workspaces for the Public Depot (P4PORT=public.perforce.com:1666
), and an sdp
folder under there for the SDP development workspace.
First, set the environment variable P4CONFIG to a value of .p4config
, and be sure the p4
command line client is in your PATH.
In the text below, replace YourUserID
with your P4USER account on the Public Depot, and replace YourMachine
with a tag identifying your machine, e.g. MacbookPro4
. From your home directory (or any directory where you store workspaces):
mkdir pub
cd pub
echo P4PORT=public.perforce.com:1666>>.p4config
echo P4USER=YourUserID>>.p4config
p4 set -q P4PORT
The P4PORT
value displayed should be public.perforce.com:1666.
mkdir sdp
cd sdp
echo P4CLIENT=YourUserID.YourMachine.sdp>>.p4config
p4 set -q P4CLIENT
The P4CLIENT
value displayed should be the one you set with the echo
command. Next, verify you can connect to the Public Depot server. This requires outbound network access to the Perforce Public Depot Server.
p4 info
The p4 info
should show a ServerID of guru_master
, indicating you're connected to the public depot. If so, proceed with creating your workspace:
p4 login
p4 client -S //sdp/dev -o | p4 client -i
p4 -s sync -q
That shoudd leave you with a workspace that is ready to use.
For contributing small changes, contribute by shelving changelists to the 'dev' branch from your workspace.
All registered Workshop users have open access (but not write) to the SDP dev branch. This allows shelving (but not submitting) updates to the SDP dev branch path:
//guest/perforce_software/sdp/dev/...
To contribute by shelving, create a workspace named something like 'your_user.your_hostname.sdp', and associate it with the //sdp/dev
stream. Make your changes in that workspace, provide a good changelist description, and include a <code>#review</code> tag in your changelist description before shelving it. When you are ready, shelve the changelist. Upon shelving, Perforce Consultants will be notified and review the change, and either incorporate (pending successful regression testing) it or provide feedback. (Note: Using the //sdp/dev
stream is optional as a convenience; you can also use a non-Stream workspace and map the path above).
Optionally, you may also branch the SDP dev branch, or some subset of it, into your own guest area. Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this:
Branch: your_name-sdp
Owner: your_name
Description:
SDP Updates by your_name.
Options: unlocked
View:
//guest/perforce_software/sdp/dev/.. //guest/your_name/sdp/dev/...
If you branch a subset, maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date.
Edit, test and submit changes in your branch. Keep your code up to date with other changes by merging down from the shared dev branch to your dev branch, e.g. by doing:
p4 merge -b your_name-sdp
... followed by the usual resolve and submit.
When you'e ready to copy your versions up to our dev branch, do like so:
p4 change
p4 copy -c YourCL -r -b your_name-sdp
Be sure that your changelist has a clear and accurate description, and that the description contains a <code>#review</code> tag on a line by itself. Then shelve the change:
p4 shelve -c YourCL
Upon shelving, Perforce Consultants will be notified and review the change, and either incorporate it (pending successful regression testing) or provide feedback.
You may want to version the SDP locally. See the SDP Guide for basic versioning instructions.
Optionally, sophisticated enterprise installations may want to use the "Tight Ship" style of management, to be in line with the "Version Everything" mindset. For more info, see the Helix Management System (HMS).
An automated regression test suite and pre-commit code review processes help ensure code quality. This is supplemented by manual testing in the Battle School training lab environment. Contact Perforce Consulting for more information.
See more details about the test suite in the Test Suite README file.
# What is the Server Deployment Package (SDP)? ## Supported Open Source Software *When deployed or certified by Perforce Consulting*, the core functionality of the SDP is officially supported by Perforce Support, except for files in the `Unsupported` directory tree (which are community supported). The SDP is open source software (see the [LICENSE](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/main/LICENSE)), and accepts contributions (pending code review) from the user community. ## Documentation References The following documentation is included in the SDP: * Release Notes ([PDF](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/ReleaseNotes.pdf) | [HTML](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/ReleaseNotes.html)) - Learn What's New in SDP 2024.1, What's coming in 2024.2 later this year, see summarized Change History, and more. * SDP_Guide.Unix ([PDF](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.pdf) | [HTML](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.html)) - Detailed documentation for SDP on UNIX/Linux. * SDP_Legacy_Upgrades.Unix ([PDF](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Legacy_Upgrades.Unix.pdf) | [HTML](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Legacy_Upgrades.Unix.html)) - Detailed documentation for upgrading SDP from pre-2020.1 versions. Once on SDP 2020.1, upgrades to newer versions are simplified with `sdp_upgrade.sh`. * SDP_Guide.Windows ([PDF](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Windows.pdf) | [HTML](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Windows.html)) - Detailed documentation for SDP on Windows. * SDP_Failover_Guide [PDF](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Failover_Guide.pdf) | [HTML](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Failover_Guide.html)) - Targeted document with information you need to know when *executing* a planned failover (on UNIX/Linux or Windows). The SDP guides above contain more information for *planning* for failover. ## Features and Benefits The following describes some of the many features and benefits of using the SDP to manage Helix Core. ### Optimal Performance, Data Safety, and Simplified Backup The SDP provides a standard structure for operating Perforce that is optimized for performance, scalability, and ease of backup. The SDP Guide includes documentation that promotes volume layout and storage architecture best practices. A key SDP feature is that it maintains an offline copy of live databases. This has many benefits: * **Zero downtime** for daily metadata checkpoint operations. * Greater data safety and redundancy with an extra copy of metadata. * Faster alternative path to recovery, bypassing checkpoint replays. * Ability to swap live with regenerated databases with minimal downtime. * Faster edge seed checkpoint creation. * Edge reseed or recovery with minimized downtime. To simplify backup, all digital assets are in a single directory. Live databases that should not be backed up directly live in another, and active log files in another. For any scale environment, this makes it easy to back up the server's critical data. These directories are a natural fit for storage volume mount points, with the layout promoting best performance and data safety. ### Best Practice Configurables The SDP also maintains, on an ongoing basis, a set of best-practices configuration settings (e.g. 'configurables' and environment settings) suited to production environments, which often differ from p4d defaults. ### High Availability (HA) The SDP supports a variety of methods for achieving high availability that benefit simple, single-server environments and sophisticated global enterprise environments, and all in between. Standardized and documented recovery procedures, offline databases, and optional replication (made easier with the SDP) all make HA easier to achieve. There is more detailed discussions and references in the SDP Guides noted above. ### Disaster Recovery (DR) Just as with HA, the SDP makes it easier to implement robust, reliable DR strategies. There is more detailed discussions and references in the SDP Guides noted above. ### Fast and Safe Upgrades The SDP makes upgrades of Helix server products fast and easy, regardless of the scale of your environment. By taking advantage of the offline database structure, the SDP enables upgrades with just a few minutes of downtime (enough to do a journal rotation). ### Production Focus The p4d executable, without any configuration, is optimized for evaluation and demonstration purposes. While it has many features that can make it reliable and robust, many of those features must be configured. The SDP is intended to be used for production environments or more realistic Proof of Concept installations. ### Sophistication Simplified - Replicas, Edges, Proxies, Brokers In addition to the basics, the SDP provides **structure**, **standards**, **guidance**, and **automation** that greatly simplify deployment and management of sophisticated Helix global topologies. For example, the SDP includes tools that greatly simplify complex tasks such as: * Adding new replicas and edge servers to a topology * Checkpointing master and edge servers without down time for users * Providing a consistent way to upgrade core Helix components in a global topology. ### Multi-Instance Management The SDP supports operation of multiple instances of Helix on a machine, allowing each instance to have differing configuration, e.g. different: * Versions of software components (p4d, p4broker, etc.) * Case Sensitivity setting (sensitive/insensitive) * Unicode setting (enabled/disabled) ### Shell Environment Management The SDP provides standard mechanism for defining a controlled shell environment. This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation. ### Perforce Automation Standards The SDP defines a standard for installing and configuring derived APIs. For example, Python is built with P4Python in /p4/common/python, and Perl with P4Perl in /p4/common/perl. ### Stock vs. Custom SDP Many Perforce customers use the SDP as-is. Others benefit by incorporating concepts and ideas from the SDP into their own environments, e.g. for development of custom policy enforcement triggers, systems integrations, promoting or enforcing local naming conventions, etc. ### Helix Management System The [Helix Management System (HMS)](https://swarm.workshop.perforce.com/projects/perforce_software-hms) is a separate, community-supported product that builds on the SDP. This is likely to be of interest at installations with sophisticated global topologies and/or a fleet of SDP instances to manage. ## Bugs, Enhancements, and Future Plans Perforce Jobs are used to file bugs and request enhancements to the SDP. The list of jobs (other than closed ones) can be viewed [here](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/jobs "SDP Open Jobs"). You can see all jobs (including closed ones if desired) from the command line interface. Connect to The Workshop and run the <code>p4 jobs</code> command, something like this: ``` p4 jobs -l -e "Project=perforce-software-sdp ^Status=closed" ``` While there is currently no formal roadmap for the SDP, it is actively maintained and continues to evolve alongside P4D and other Helix Core components it manages. Jobs are occasionally reviewed and addressed as time allows for consultants and contributors, sometimes driven by customer-funded consulting projects. ## Contributing Contributions to the SDP are most welcome! Note that contributing requires a basic user knowledge of working with Perforce. ### Setting up a workspace. The SDP currently lives in a Classic (i.e. non-Stream) depot for historic reasons, in this path: ``` //guest/perforce_software/sdp/... ``` However, an empty //sdp stream depot exists and is used to simplify workspace creation and management. Here is a sample procedure to set up a workspace. These instructions work on Linux/UNIX, MacOS, or Windows. This sample uses P4CONFIG files and creates a `pub` folder to contain multiple workspaces for the Public Depot (`P4PORT=public.perforce.com:1666`), and an `sdp` folder under there for the SDP development workspace. First, set the environment variable P4CONFIG to a value of `.p4config`, and be sure the `p4` command line client is in your PATH. In the text below, replace `YourUserID` with your P4USER account on the Public Depot, and replace `YourMachine` with a tag identifying your machine, e.g. `MacbookPro4`. From your home directory (or any directory where you store workspaces): ``` mkdir pub cd pub echo P4PORT=public.perforce.com:1666>>.p4config echo P4USER=YourUserID>>.p4config p4 set -q P4PORT ``` The `P4PORT` value displayed should be public.perforce.com:1666. ``` mkdir sdp cd sdp echo P4CLIENT=YourUserID.YourMachine.sdp>>.p4config p4 set -q P4CLIENT ``` The `P4CLIENT` value displayed should be the one you set with the `echo` command. Next, verify you can connect to the Public Depot server. This requires outbound network access to the Perforce Public Depot Server. ``` p4 info ``` The `p4 info` should show a ServerID of `guru_master`, indicating you're connected to the public depot. If so, proceed with creating your workspace: ``` p4 login p4 client -S //sdp/dev -o | p4 client -i p4 -s sync -q ``` That shoudd leave you with a workspace that is ready to use. ### Contributing by Shelving (Small Changes) For contributing small changes, contribute by shelving changelists to the 'dev' branch from your workspace. All registered Workshop users have **open** access (but not **write**) to the SDP dev branch. This allows shelving (but not submitting) updates to the SDP dev branch path: ``` //guest/perforce_software/sdp/dev/... ``` To contribute by shelving, create a workspace named something like '*your_user*.*your_hostname*.sdp', and associate it with the `//sdp/dev` stream. Make your changes in that workspace, provide a good changelist description, and include a <code>#review</code> tag in your changelist description before shelving it. When you are ready, shelve the changelist. Upon shelving, Perforce Consultants will be notified and review the change, and either incorporate (pending successful regression testing) it or provide feedback. (Note: Using the `//sdp/dev` stream is optional as a convenience; you can also use a non-Stream workspace and map the path above). ### Contributing by Branching (Bigger Changes) Optionally, you may also branch the SDP dev branch, or some subset of it, into your own guest area. Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this: ``` Branch: your_name-sdp Owner: your_name Description: SDP Updates by your_name. Options: unlocked View: //guest/perforce_software/sdp/dev/.. //guest/your_name/sdp/dev/... ``` If you branch a subset, maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date. Edit, test and submit changes in your branch. Keep your code up to date with other changes by merging down from the shared dev branch to your dev branch, e.g. by doing: ``` p4 merge -b your_name-sdp ``` ... followed by the usual resolve and submit. When you'e ready to copy your versions up to our dev branch, do like so: ``` p4 change p4 copy -c YourCL -r -b your_name-sdp ``` Be sure that your changelist has a clear and accurate description, and that the description contains a <code>#review</code> tag on a line by itself. Then shelve the change: ``` p4 shelve -c YourCL ``` Upon shelving, Perforce Consultants will be notified and review the change, and either incorporate it (pending successful regression testing) or provide feedback. ### Guidelines for SDP Contributors * Communicate! Use Swarm or email! Let folks know what you're planning before you work too hard on something that might already be underway. * File a job by sending a request to support-helix-core@perforce.com, and let us know you intend to work on it. While not required, this increases the transparency of what you're doing, and helps promote it. * We prefer changes that are generic, and have a wide appeal. * The core of the SDP, the scripts that do offline checkpoints for p4d, prioritize Perforce best practices over scripting magic. Don't get too fancy! :) ## Versioning the SDP Locally You may want to version the SDP locally. See the SDP Guide for basic versioning instructions. Optionally, sophisticated enterprise installations may want to use the "Tight Ship" style of management, to be in line with the "Version Everything" mindset. For more info, see the [Helix Management System (HMS)](https://swarm.workshop.perforce.com/projects/perforce_software-hms). ## SDP Code Quality An automated regression test suite and pre-commit code review processes help ensure code quality. This is supplemented by manual testing in the Battle School training lab environment. Contact [Perforce Consulting](mailto:consulting@perforce.com) for more information. See more details about the test suite in the [Test Suite README file](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/main/test/README.md).
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#24 | 30389 | C. Thomas Tyler | Fixed a typo. | ||
#23 | 30388 | C. Thomas Tyler |
Released SDP 2024.1.30385 (2024/06/11). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#22 | 30043 | C. Thomas Tyler |
Released SDP 2023.2.30041 (2023/12/22). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#21 | 29962 | C. Thomas Tyler | Tweaks to README.md. | ||
#20 | 29916 | Robert Cowham | Removed mention of Helix Installer while we resolve issues around inexperienced users managing to zap their existing repository. | ||
#19 | 28860 | C. Thomas Tyler | Re-generated README.md. | ||
#18 | 28858 | C. Thomas Tyler |
Released SDP 2022.1.28855 (2022/05/27). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#17 | 28412 | C. Thomas Tyler |
Released SDP 2021.2.28410 (2021/11/24). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#16 | 27527 | C. Thomas Tyler |
Released SDP 2020.1.27524 (2021/02/26). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#15 | 27463 | C. Thomas Tyler |
Released SDP 2020.1.27457 (2021/02/17). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#14 | 27331 | C. Thomas Tyler |
Released SDP 2020.1.27325 (2021/01/29). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#13 | 25596 | C. Thomas Tyler |
Released SDP 2019.2.25594 (2019/05/02). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#12 | 25245 | C. Thomas Tyler |
Released SDP 2019.1.25238 (2019/03/02). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#11 | 25051 | C. Thomas Tyler |
Released SDP 2018.1.23583.p2 (2019/01/23). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev', with selective removal of work-in-progress files. |
||
#10 | 23357 | C. Thomas Tyler |
Released SDP 2017.4.23354 (2017/12/08). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#9 | 23331 | C. Thomas Tyler |
Released SDP 2017.4.23329 (2017/12/05). Copy Up using 'p4 copy -r -b perforce_software-sdp-dev'. |
||
#8 | 16682 | C. Thomas Tyler | Typo | ||
#7 | 15795 | C. Thomas Tyler | Just a few cosmetic tweaks. | ||
#6 | 13908 | C. Thomas Tyler | Pushing SDP 2015.1.13906. | ||
#5 | 11829 | Robert Cowham | Link to test/README.md for testing. | ||
#4 | 11525 | Russell C. Jackson (Rusty) | Updated Version and Release notes. | ||
#3 | 11127 | Robert Cowham | Update mention of Windows SDP | ||
#2 | 11026 | mark_foundry |
Fixing typos in the README.md file. I assume that the README.html is generated from the markdown file, so I've not fixed that file. |
||
#1 | 10611 | C. Thomas Tyler | Added README file for the SDP in The Workshop. |