README.md #19

What is the Server Deployment Package (SDP)?

Supported Open Source Software

When deployed or certified by Perforce Consulting, the core functionality of the SDP is officially supported by Perforce Support, except for files in the Unsupported directory tree.

The Unsupported folder included with the SDP is community supported, but officially supported by Perforce Support. (Unsupported files were in various folders prior to the 2020.1 release.)

The SDP is open source software (see the LICENSE), and accepts contributions (pending code review) from the user community.

Please see the doc directory for:

High Availability (HA)

The SDP supports a variety of methods for achieving high availability that benefit simple, single-server environments and sophisticated global enterprise environments, and all in between. Standardized and documented recovery procedures, offline databases, and optional replication (made easier with the SDP) all make HA easier to achieve.

There is more detailed discussions and references in the SDP Guides noted above.

Disaster Recovery (DR)

Just as with HA, the SDP makes it easier to implement robust, reliable DR strategies.

There is more detailed discussions and references in the SDP Guides noted above.

Fast and Safe Upgrades

The SDP makes upgrades of Helix server products fast and easy, regardless of the scale of your environment. By taking advantage of the offline database structure, the SDP enables upgrades with just a few minutes of downtime (enough to do a journal rotation).

Production Focus

The p4d executable, without any configuration, is optimized for evaluation and demonstration purposes. While it has many features that can make it reliable and robust, many of those features must be configured. The SDP is intended to be used for production environments or more realistic Proof of Concept installations.

See the Helix Installer which makes quick work of installing the SDP for PoC evaluation with a production-like setup, complete with the Sample Depot training data set.

Best Practice Configurables

The SDP also maintains, on an ongoing basis, a set of best-practices configuration settings (e.g. 'configurables' and environment settings) suited to production environments, which often differ from p4d defaults.

Optimal Performance, Data Safety, and Simplified Backup

The SDP provides a standard structure for operating Perforce that is optimized for performance, scalability, and ease of backup. The SDP Guide includes documentation that promotes volume layout and storage architecture best practices.

To simplify backup, all digital assets are in a single directory. Live databases that cannot be backed up live in another, and active log files in another. For any scale environment, this makes it easy to back up the server's critical data. For larger scale environments, these directories become storage volume mount points, with the layout promoting best performance and data safety.

Sophistication Simplified

In addition to the basics, the SDP provides structure, standards, guidance, and automation that greatly simplify deployment and management of sophisticated Helix global topologies. For example, the SDP includes tools that greatly simplify complex tasks such as creating new replicas, backing up edge servers without down time, etc.

Multi-Instance Management

The SDP supports operation of multiple instances of Helix on a machine, allowing each instance to have differing configuration, e.g. different:

  • versions of software components (p4d, p4broker, etc.)
  • case sensitivity setting (sensitive/insensitive)
  • Unicode setting (enabled/disabled)

Shell Environment Management

The SDP provides standard mechanism for defining a controlled shell environment. This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation.

Perforce Automation Standards

The SDP defines a standard for installing and configuring derived APIs. For example, Python is built with P4Python in /p4/common/python, and Perl with P4Perl in /p4/common/perl.

Stock vs. Custom SDP

Many Perforce customers use the SDP as-is. Others benefit by incorporating concepts and ideas from the SDP into their own environments, e.g. for development of custom policy enforcement triggers, systems integrations, promoting or enforcing local naming conventions, etc.

Helix Management System

The Helix Management System (HMS) is a separate, community-supported product that builds on the SDP. This is likely to be of interest at installations with sophisticated global topologies and/or a fleet of SDP instances to manage.

Bugs, Enhancements, and Future Plans

Perforce Jobs are used to file bugs and request enhancements to the SDP. The list of jobs (other than closed ones) can be viewed here.

You can see all jobs (including closed ones if desired) from the command line interface. Connect to The Workshop and run the <code>p4 jobs</code> command, something like this:

p4 jobs -l -e "Project=perforce-software-sdp ^Status=closed"

While there is currently no formal roadmap for the SDP, it is actively maintained and continues to evolve alongside P4D and other Helix Core components it manages. Jobs are occasionally reviewed and addressed as time allows for consultants and contributors, sometimes driven by customer-funded consulting projects.

Contributing

Contributions to the SDP are most welcome! Note that contributing requires a basic user knowledge of working with Perforce.

Contributing by Shelving (Small Changes)

For contributing small changes, contribute by shelving changelists to the 'dev' branch your workspace.

All registered Workshop users have open access (but not write) to the SDP dev branch. This allows shelving (but not submitting) updates to the SDP dev branch path:

//guest/perforce_software/sdp/dev/...

To contribute by shelving, create a workspace that maps that path, named something like 'your_user.your_hostname.sdp'. Make your changes in that workspace, provide a good changelist description, and include a <code>#review</code> tag in your changelist description.

When you are ready, shelve the changelist. Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.

Contributing by Branching (Bigger Changes)

Optionally, you may also branch the SDP dev branch, or some subset of it, into your own guest area. Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this:

Branch: your_name-sdp

Owner: your_name

Description:
    SDP Updates by your_name.

Options:    unlocked

View:
    //guest/perforce_software/sdp/dev/.. //guest/your_name/sdp/dev/...

If you branch a subset, maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date.

Edit, test and submit changes in your branch. Keep your code up to date with other changes by merging down from the shared dev branch to your dev branch, e.g. by doing:

p4 merge -b your_name-sdp

... followed by the usual resolve and submit.

When you'e ready to copy your versions up to our dev branch, do like so:

p4 change
p4 copy -c YourCL -r -b your_name-sdp

Be sure that your changelist has a clear and accurate description, and that the description contains a <code>#review</code> tag on a line by itself. That will trigger a code review process when you shelve the change, like so:

p4 shelve -c YourCL

Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.

Guidelines for SDP Contributors

  • Communicate! Use Swarm or email! Let folks know what you're planning before you work too hard on something that might already be underway.
  • File a job (Project=perforce-software-sdp) first. That's not required, but increases the transparency of what you're doing, and helps promote it.
  • We prefer changes that are generic, and have a wide appeal.
  • The core of the SDP, the scripts that do offline checkpoints for p4d, prioritize Perforce best practices over scripting magic. Don't get too fancy! :)

Versioning the SDP Locally

You may want to version the SDP locally. See the SDP Guide for basic versioning instructions.

Optionally, sophisticated enterprise installations may want to use the "Tight Ship" style of management, to be in line with the "Version Everything" mindset. For more info, see the Helix Management System (HMS).

Upgrading Older Existing SDP Installations

See the SDP_Upgrade_Guide.Unix

The Windows SDP

There are two flavors of the SDP, Unix and Windows. Both are available from the SDP downloads folder.

SDP Code Quality

An automated regression test suite and pre-commit code review processes help ensure code quality. This is supplemented by manual testing in the Battle School training lab environment. Contact Perforce Consulting for more information.

See more details about the test suite in the Test Suite README file.

What is the Server Deployment Package (SDP)?
===

Supported Open Source Software
---

*When deployed or certified by Perforce Consulting*, the core functionality of the SDP is officially supported by Perforce Support, except for files in the `Unsupported` directory tree.

The `Unsupported` folder included with the SDP is community supported, but officially supported by Perforce Support. (Unsupported files were in various folders prior to the 2020.1 release.)

The SDP is open source software (see the [LICENSE](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/main/LICENSE)), and accepts contributions (pending code review) from the user community.

Please see the `doc` directory for:

* SDP_GUIDE.Unix [PDF version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.pdf), [HTML version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.html), or [ASCIIDoctor version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Unix.adoc)
* SDP_GUIDE.Windows [PDF version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Windows.pdf), [HTML version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Windows.html), or [ASCIIDoctor version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Guide.Windows.adoc)
* SDP_Failover_Guide [PDF version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Failover_Guide.pdf), [HTML version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Failover_Guide.html), or [ASCIIDoctor version](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/view/main/doc/SDP_Failover_Guide.adoc)

High Availability (HA)
---
The SDP supports a variety of methods for achieving high availability that benefit simple, single-server environments and sophisticated global enterprise environments, and all in between.  Standardized and documented recovery procedures, offline databases, and optional replication (made easier with the SDP) all make HA easier to achieve.

There is more detailed discussions and references in the SDP Guides noted above.

Disaster Recovery (DR)
---
Just as with HA, the SDP makes it easier to implement robust, reliable DR strategies.

There is more detailed discussions and references in the SDP Guides noted above.

Fast and Safe Upgrades
---
The SDP makes upgrades of Helix server products fast and easy, regardless of the scale of your environment.  By taking advantage of the offline database structure, the SDP enables upgrades with just a few minutes of downtime (enough to do a journal rotation).

Production Focus
---
The p4d executable, without any configuration, is optimized for evaluation and demonstration purposes.  While it has many features that can make it reliable and robust, many of those features must be configured.  The SDP is intended to be used for production environments or more realistic Proof of Concept installations.

See the [Helix Installer](https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer) which makes quick work of installing the SDP for PoC evaluation with a production-like setup, complete with the Sample Depot training data set.

Best Practice Configurables
---

The SDP also maintains, on an ongoing basis, a set of best-practices configuration settings (e.g. 'configurables' and environment settings) suited to production environments, which often differ from p4d defaults.

Optimal Performance, Data Safety, and Simplified Backup
---
The SDP provides a standard structure for operating Perforce that is optimized for performance, scalability, and ease of backup. The SDP Guide includes documentation that promotes volume layout and storage architecture best practices.

To simplify backup, all digital assets are in a single directory.  Live databases that cannot be backed up live in another, and active log files in another.  For any scale environment, this makes it easy to back up the server's critical data.  For larger scale environments, these directories become storage volume mount points, with the layout promoting best performance and data safety.

Sophistication Simplified
---
In addition to the basics, the SDP provides structure, standards, guidance, and automation that greatly simplify deployment and management of sophisticated Helix global topologies.  For example, the SDP includes tools that greatly simplify complex tasks such as creating new replicas, backing up edge servers without down time, etc.

Multi-Instance Management
---
The SDP supports operation of multiple instances of Helix on a machine, allowing each instance to have differing configuration, e.g. different:

* versions of software components (p4d, p4broker, etc.)
* case sensitivity setting (sensitive/insensitive)
* Unicode setting (enabled/disabled)

Shell Environment Management
---

The SDP provides standard mechanism for defining a controlled shell environment.  This benefits p4d and other Perforce software products directly, and also provides good examples of controlling the environment for supporting automation.

Perforce Automation Standards
---
The SDP defines a standard for installing and configuring derived APIs.  For example, Python is built with P4Python in /p4/common/python, and Perl with P4Perl in /p4/common/perl.

Stock vs. Custom SDP
---
Many Perforce customers use the SDP as-is.  Others benefit by incorporating concepts and ideas from the SDP into their own environments, e.g. for development of custom policy enforcement triggers, systems integrations, promoting or enforcing local naming conventions, etc.

Helix Management System
---
The [Helix Management System (HMS)](https://swarm.workshop.perforce.com/projects/perforce_software-hms) is a separate, community-supported product that builds on the SDP.  This is likely to be of interest at installations with sophisticated global topologies and/or a fleet of SDP instances to manage.

Bugs, Enhancements, and Future Plans
===
Perforce Jobs are used to file bugs and request enhancements to the SDP.  The list of jobs (other than closed ones) can be viewed [here](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/jobs "SDP Open Jobs").

You can see all jobs (including closed ones if desired) from the command line interface.  Connect to The Workshop and run the <code>p4 jobs</code> command, something like this:

```
p4 jobs -l -e "Project=perforce-software-sdp ^Status=closed"
```

While there is currently no formal roadmap for the SDP, it is actively maintained and continues to evolve alongside P4D and other Helix Core components it manages.  Jobs are occasionally reviewed and addressed as time allows for consultants and contributors, sometimes driven by customer-funded consulting projects.

Contributing
===
Contributions to the SDP are most welcome!  Note that contributing requires a basic user knowledge of working with Perforce.

Contributing by Shelving (Small Changes)
---

For contributing small changes, contribute by shelving changelists to the 'dev' branch your workspace.

All registered Workshop users have **open** access (but not **write**) to the SDP dev branch.  This allows shelving (but not submitting) updates to the SDP dev branch path:

```
//guest/perforce_software/sdp/dev/...
```
To contribute by shelving, create a workspace that maps that path, named something like '*your_user*.*your_hostname*.sdp'.  Make your changes in that workspace, provide a good changelist description, and include a <code>#review</code> tag in your changelist description.

When you are ready, shelve the changelist.  Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.

Contributing by Branching (Bigger Changes)
---
Optionally, you may also branch the SDP dev branch, or some subset of it, into your own guest area.  Creating a branch spec is recommended for this purpose, e.g. with a branch spec like this:

```
Branch: your_name-sdp

Owner: your_name

Description:
    SDP Updates by your_name.

Options:	unlocked
	
View:
    //guest/perforce_software/sdp/dev/.. //guest/your_name/sdp/dev/...
```

If you branch a subset, maintain the full path structure (as in the example above), so you can easily expand the scope of what you branched initially at some later date.

Edit, test and submit changes in your branch.  Keep your code up to date with other changes by merging down from the shared dev branch to your dev branch, e.g. by doing:

```
p4 merge -b your_name-sdp
```

... followed by the usual resolve and submit.

When you'e ready to copy your versions up to our dev branch, do like so:

```
p4 change
p4 copy -c YourCL -r -b your_name-sdp
```

Be sure that your changelist has a clear and accurate description, and that the description contains a <code>#review</code> tag on a line by itself.  That will trigger a code review process when you shelve the change, like so:

```
p4 shelve -c YourCL
```

Perforce Consultants will review the change, and either incorporate it or provide feedback. Be sure to provide a detailed change description, and don't forget include the <code>#review</code> tag in your changelist description to initiate a Swarm pre-commit code review.

Guidelines for SDP Contributors
---

* Communicate! Use Swarm or email! Let folks know what you're planning before you work too hard on something that might already be underway.
* File a job (Project=**perforce-software-sdp**) first.  That's not required, but increases the transparency of what you're doing, and helps promote it.
* We prefer changes that are generic, and have a wide appeal.
* The core of the SDP, the scripts that do offline checkpoints for p4d, prioritize Perforce best practices over scripting magic.  Don't get too fancy! :)

Versioning the SDP Locally
===

You may want to version the SDP locally.  See the SDP Guide for basic versioning instructions.

Optionally, sophisticated enterprise installations may want to use the "Tight Ship" style of management, to be in line with the "Version Everything" mindset. For more info, see the [Helix Management System (HMS)](https://swarm.workshop.perforce.com/projects/perforce_software-hms).

Upgrading Older Existing SDP Installations
===

See the [SDP_Upgrade_Guide.Unix](https://swarm.workshop.perforce.com/files/guest/perforce_software/sdp/r20.1/doc/SDP_Upgrade_Guide.Unix.html)

The Windows SDP
===
There are two flavors of the SDP, Unix and Windows.  Both are available from the [SDP downloads folder](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/downloads).

SDP Code Quality
===
An automated regression test suite and pre-commit code review processes help ensure code quality. This is supplemented by manual testing in the Battle School training lab environment.  Contact [Perforce Consulting](mailto:consulting@perforce.com) for more information.

See more details about the test suite in the [Test Suite README file](https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/main/test/README.md).
# Change User Description Committed
#25 29932 C. Thomas Tyler Merge down @29916 from main:

Removed mention of Helix Installer while we resolve issues around inexperienced users managing to zap their existing repository.

#review-29933
#24 29915 Robert Cowham Remove reference to Helix Installer for now.
#23 28861 C. Thomas Tyler Merged patch to README.md.
#22 28852 C. Thomas Tyler Release notes corrections and tweaks to SDP home page
related to the major version change..
#21 28385 C. Thomas Tyler Updated README for coming r21.2 release.

Enhanced product description to emphazise key SDP benefits, better
describing the value of the Offline DB.

#review-28386
#20 27482 C. Thomas Tyler README.md updates:
* Added Documentation References section, with less clutter and easier links to PDF/HTML versions of key docs, including the upgrade guide.
* Enhanced "Sophistication Simplified" section.
* Simplified Support status text.
* General grammer corrections.
* Removed a broken link.
* Simpflified Contribing sections.
* Enhanced formatting/TOC levels, moved several sections under new "Features and Benefits" section.

#review-27483
#19 27444 C. Thomas Tyler Fixed issue with broken doc links.
#18 27015 C. Thomas Tyler Replaced notes in README.md re: upgrading SDP with ref to new SDP Upgrade Guide doc.

The doc URL references not-yet-existing SDP r20.1 branch.
#17 26781 Robert Cowham Reference new docs.
#16 26652 Robert Cowham This is Tom's change:

Introduced new 'Unsupported' directory to clarify that some files
in the SDP are not officially supported. These files are samples for
illustration, to provide examples, or are deprecated but not yet
ready for removal from the package.

The Maintenance and many SDP triggers have been moved under here,
along with other SDP scripts and triggers.

Added comments to p4_vars indicating that it should not be edited
directly. Added reference to an optional site_global_vars file that,
if it exists, will be sourced to provide global user settings
without needing to edit p4_vars.

As an exception to the refactoring, the totalusers.py Maintenance
script will be moved to indicate that it is supported.

Removed settings to support long-sunset P4Web from supported structure.

Structure under new .../Unsupported folder is:
   Samples/bin             Sample scripts.
   Samples/triggers        Sample trigger scripts.
   Samples/triggers/tests  Sample trigger script tests.
   Samples/broker          Sample broker filter scripts.
   Deprecated/triggers     Deprecated triggers.

To Do in a subsequent change: Make corresponding doc changes.
#15 26595 C. Thomas Tyler Minor updates to README.md, WIP.
#14 25591 C. Thomas Tyler Clarified that Maintenance folder scripts are samples and are
NOT officially supported.

Updated references to HMS now that it has been split back out
into a separate product.
#13 25155 Robert Cowham Refer to test/README.md
#12 25043 C. Thomas Tyler Prepared patch SDP/MultiArch/2018.1/23583.p2

Changes:
* In README, moved 'Supported Open Source Software' section
to the top of the page, to make Support position more visible.
* Copyright updated to 2019.
* Cleanup of excess script recreate_db_sync_replica.sh
* Re-removal of previously deleted script recreate_db_checkpoint.sh,
and corresponding removal from test suite.
* This patch tarball will also contain some Docker test suite updates
already staged in main.

By-passing review to trigger an automated test.
#11 24511 C. Thomas Tyler Fixed an obvious doc typo in README files.
 Bypassing code
review.
#10 23353 C. Thomas Tyler Fixed typo/grammatical error.
#9 23315 C. Thomas Tyler Overhauled SDP README page to better convey its value, and
much more strongly promote the SDP.

This also documents the the nature of offical Perforce Support
for the SDP, and the limitations of that Support to core SDP
functionality.
#8 16784 C. Thomas Tyler Routine Merge Down to dev from main using:
p4 -s merge -n -b perforce_software-sdp-dev
#7 15797 C. Thomas Tyler Routine    Merge Down to dev from main for SDP.
#6 13729 C. Thomas Tyler Tweaked formatting on README.md.
#5 12028 C. Thomas Tyler Refreshed SDP dev branch, merging down from main.
#4 11535 Russell C. Jackson (Rusty) Updated dev from main.
#3 11463 Russell C. Jackson (Rusty) Updated dev to prepare for Summit agreed changes.
#2 11038 Robert Cowham Catchup from Main
#1 10638 C. Thomas Tyler Populate perforce_software-sdp-dev.
//guest/perforce_software/sdp/main/README.md
#1 10611 C. Thomas Tyler Added README file for the SDP in The Workshop.