# SDP Release Process Overview

## Introduction

This document describes the process used to release the Server Deployment Package (SDP). This is intended for those developing the SDP.

This is the first release process document written for the SDP's Streams-based depot structure (`//p4-sdp/...`), following the Classic-to-Streams migration. The stream topology and terminology used throughout are:

* `//p4-sdp/main` -- a `mainline` stream, long-lived and stable, reflecting whatever was most recently released.
* `//p4-sdp/dev` -- a `development` stream, parented by `main`. Long-lived, with work in progress at any given time, but always kept stable enough that a release could be cut from it on short notice.
* `//p4-sdp/dev_*` streams -- short-lived, per-task `development` streams parented by `dev` (e.g. `dev_rebrand`, or a future `dev_SDP-<issue>`), used for larger or higher-risk work. Small, low-risk, or hyper-urgent changes may instead go straight to `dev`.
* `//p4-sdp/r<YYYY>.<N>.<P>` release streams (e.g. `r26.1.0`) -- `release`-type streams parented by `main`, cut fresh for each GA release or patch. **The `<P>` patch digit is always present, never omitted** -- `0` for the GA release, incrementing with each patch after it (e.g. `r26.1.1` for the first patch). Long-lived and stable except briefly during the release process itself. **Patches do not modify an existing release stream** -- a patch is a brand new release stream cut from `main`, keeping the flow uniformly `dev -> main -> release` for every release, GA or patch.

`install_sdp.sh` downloads the SDP tarball from a single fixed URL under `main` (see Steps 19-20), never from a specific release stream's own URL -- so, unlike the release stream name itself, that URL never needs editing per release, and mirrors the one fixed download location the Classic depot always had. This depends on `downloads/` being `isolate`d on `main`, so a pre-release build never leaks into `main` via ordinary Copy Up, and `main`'s tarball only ever changes via the deliberate publish step.

## Preparation

Use a dedicated set of release workspace clients, separate from any you use for day-to-day SDP development, and free of cruft files (editor temp files, `.DS_Store`, etc.) -- the packaging mechanism excludes known-cruft file patterns, but a clean workspace avoids relying on that as a safety net.

Because different steps of this process operate on different streams, and a Perforce client can only be switched to one stream at a time, use **one client per stream**, each rooted in its own directory. A convenient convention, matching the directory names used throughout this document, is to root them as sibling directories, e.g.:

```
~/re/p4-sdp/dev    (client switched to //p4-sdp/dev)
~/re/p4-sdp/main   (client switched to //p4-sdp/main)
```

A third client, for whichever release stream is currently being cut (e.g. `~/re/p4-sdp/r26.1.0`), is created fresh as part of the process (see the "Cut the Release Stream" step below) and can be discarded once the release ships.

Note: `tools/` now lives inside each stream (it does not sit outside `dev`/`main` the way it did in the Classic depot layout), so each of these clients has its own copy of the release tooling in its own `tools/` subdirectory.

**macOS note**: several scripts used throughout this process (`package_downloads.sh`, `gen_file_format_check.sh`, `gen_script_man_pages.sh`, and possibly others) require bash 4.x+ features. macOS's own `/bin/bash` is permanently stuck at 3.2 for licensing reasons Apple controls, not us -- if you're running this process from a Mac (as more than one of us does), install a modern bash (e.g. via `homebrew`) and invoke these scripts explicitly with it (`/opt/homebrew/bin/bash somescript.sh`) rather than relying on `./somescript.sh` or a bare `bash somescript.sh`, either of which may pick up the ancient system one depending on your `PATH`.

Throughout this document, `$WSRoot` refers to the root directory of whichever client a given step says to use. From inside that client (anywhere under its root), you can always determine it with:

```
WSRoot=$(p4 -ztag -F %clientRoot% info)
```

## Procedure

### Step 1: Plan!

**Don't ship on a Friday, or just before a major holiday.** Per Technical Support's own request: if something goes wrong post-release, you want the team actually available to respond, not scattered for a weekend or holiday. Pick (or confirm) a release date with this in mind before locking in anything else below.

Use [this JIRA Project](https://perforce.atlassian.net/jira/software/c/projects/SDP/issues) to plan your work.  This JIRA instance is only accessible with Perforce corporate SSO.

The SDP product owner determines what the release will be.  For example **2025.1** would be the **SDP 2025.1, GA Release**, while **2025.1.1** would be **SDP 2025.1 Patch 1**.

The SDP shares the same major version as P4 Server (e.g. 2025.1).  Versions in JIRA use the major version number, and append an integer as the patch identifier. The patch number is omitted for the GA release, and increments starting from 1 for each patch after a major release.  While SDP major releases are bound to the P4D release schedule, SDP patches release on an independent schedule.

In JIRA, create Releases to be used for `Fix version` identifiers, such as 2024.2, 2024.2.1, 2025.1, etc.  The name of the release should following YYYY.M for a major release, or YYYY.M.P for a patch release.

For example, in JIRA, there will be a release named 2024.2.3, with a Description of "SDP 2024.2 Patch 3", and a 2025.1 release with a description of "SDP 2025.1."

Create JIRA search filters for each `Fix version`, e.g. `2025.1` for the SDP 2025.1 GA release, or `2024.2.3` for the SDP 2024.2 Patch 3 release. There is also a standing `SDP To Do` filter with all open jobs used during planning.

Generally, while some planning occurs early in the release cycle, other elements of planning occur iteratively and continuously throughout the SDP release cycle.

**Output**: The output of this step is a set of JIRA issues associated with `Fix versions`.

### Step 2: Develop!

Develop and test changes, either directly in `dev` (for small, low-risk, or hyper-urgent changes) or in a short-lived `dev_*` stream parented by `dev` (for larger or higher-risk changes), merged into `dev` once ready. This will occur over time.

When an issue is marked as complete, do the following in JIRA to get it included in the Release Notes:

* Set the `Fix versions` JIRA issue field value to the next release, e.g. `2024.2`.
* Update the `P4Blog` field of the JIRA issue with an indication of what change addressed the issue.  This can be a review URL from The Workshop, e.g. https://workshop.perforce.com/reviews/29811, or could also be the URL for the changelist in `//p4-sdp/dev` (or the `dev_*` stream) where the fix was made.

**Output**: The output of this step is an improved software product and information needed to create release notes.

### Step 3: Target New P4D in SDP Regression Test Suite (Major Releases Only)

This step applies to major releases only; skip this for patch releases. Done in the `dev` client, as part of ongoing development (not release-time-specific).

Update the SDP regression test suite to ensure that it references the new P4 Server release. For example, `Server/test/test_Upgrade.py`'s `--to-version` default argument needs updating to the new release (e.g. from `"24.2"` to `"25.1"`).

This code change might possibly break the regression. It is important to re-verify that the regression test suite is still successful after this change, adapting as needed.

**Output**: This step ensures the new P4 Server version doesn't break SDP regression tests suites, and to raise awareness of any issues that may cause customer issues when adopting the new release.

### Step 4: Verify SDP Regression Tests

Verify that all SDP regression test suites are successful, targeting `//p4-sdp/dev` (the release candidate) -- not `main`, which won't have this release's content until Copy Up happens later in this process. After Copy Up and cutting the release stream (later steps), do a final, lighter-weight spot check of the same suites against the new release stream itself, since real content moves at that point.

#### Regression Test Suite 1: Docker/Jenkins

The first test suite is the Docker-based test suite with a Jenkins server running and polling for changes.

The SDP regression test suite Jenkins job is only accessible from the Perforce corporate VPN.

This uses files in `//p4-sdp/dev/test/...` (except the `bsw` subdirectory) and `//p4-sdp/dev/Server/test`.

#### Regression Test Suite 2: Dedicated Battle School Workshop (BSW)

The second test suite is the Battle School Workshop test suite, which operates in a dedicated Battle School Workshop Lab Environment (Gen 7).

This uses files in `//p4-sdp/dev/test/bsw/...`

Tests are run via the lab's `qa` alias (or `qa_build` for a fully automated, unattended run), which syncs the lab's insitu SDP checkout and then runs `run_all_tests.sh` against it. The tests themselves are defined in `cli_tests.cfg` and `scripted_tests.cfg` in `test/bsw`. See `SDP-QA-LabEnvironment.adoc` (in the Gen7 lab project) for the full mechanism.

#### Regression Test Suite 3: The install_sdp.sh test servers

A fleet of EC2 instances of various operating systems ensures that the `install_sdp.sh` script and its dependencies run on a variety of supported Linux distros. This fleet, in AWS region us-east-2, is home to the dedicated test suite for the `install_sdp.sh` script. For optimal realism, these do not run in containers.

**Two fleets currently exist** -- an old one whose DVCS remotes still point at stale Classic-depot guest paths, and the correct, current one whose remotes point at the real `//p4-sdp/...` streams. Always use the current fleet (identifiable by its `ssh-test-install_sdp-*` wrapper scripts, as opposed to the old fleet's `ssh-install_sdp-test*` wrappers). See the `install-sdp-test-suite-run` skill for the full fleet inventory and how to tell them apart reliably.

The following operating systems are currently represented in the current fleet:

* Rocky 8
* Rocky 9
* Rocky 10
* SuSE 15
* Ubuntu 22.04
* Ubuntu 24.04
* Ubuntu 26.04 (aarch64) -- known NOT to work due to a POSIX `chown` compliance regression that the Ubuntu community has not yet addressed; a failure here is not by itself a sign of a new regression.

**This fleet's automated suite only covers fresh `install_sdp.sh` installs -- it does not exercise `sdp_upgrade.sh` at all.** This gap let two real bugs (SDP-1394, SDP-1395) ship in the 2026.1 GA release, caught only the day after via manual testing. If this release touches `install_sdp.sh`, `sdp_upgrade.sh`, `mkdirs.sh`, `configure_sample_depot_for_sdp.sh`, or anything else in the package-structure install/upgrade path, also manually run the install-then-upgrade procedure documented in the `install-sdp-test-suite-run` skill's "Manual upgrade testing" section -- here, before shipping, not just as part of Step 22's post-release smoke tests.

**Output**: The output of this step is confidence we're not regressing.

### Step 5: Determine Release Type

Determine if the SDP release is to be Major, Patch, or Doc Only.

Since the 2021.2 release of P4D, SDP major releases have been tied to the P4D release schedule, **with SDP releases occurring approximately 10-15 business days after P4D releases**.

Other releases will be Patch releases if there are any functionality changes, or Doc Only releases, where changes since the last release affect only documentation. The SDP documentation is key to its value, worth doing patch updates including only documentation changes.

Regardless of Major or Patch, the release mechanics in this document are identical: both are cut as a fresh release stream from `main` (e.g. `r26.1.0` for GA, `r26.1.1` for the first patch after it). Only a small number of steps below are marked "Major Releases Only."

**Output**: The output of this step is an intended SDP major version number and type, e.g. "2024.2 Patch 3", or "2025.1 GA".

### Step 6: Update Default P4D Version (Major Releases Only)

This step applies to major releases only; skip this for patch releases. Done in the `dev` client, as an ordinary development change.

Modify the script `$WSRoot/p4_binaries/get_p4_binaries.sh` as follows:

* Change the `DefaultP4Version` value to match the current P4 Server GA release version with a value like `r25.1`.  The binaries for this release should be accessible under https://ftp.perforce.com/perforce/r25.1 (replacing `r25.1` with the new value).

**Output**: The default P4 Server version acquired from our FTP server with the `get_p4_binaries.sh` script reflects the new major version for the P4 Server.

### Step 7: Update P4*.json release list files (Major Releases Only)

This step applies to major releases only; skip this for patch releases. Done in the `dev` client, as an ordinary development change.

Edit the *P4*.json release list files used by `get_p4_binaries.sh`.

```
cd "$WSRoot/p4_binaries"
p4 edit P4*.json
for jf in P4 P4D P4Broker P4Proxy; do
   curl -O "https://updates.perforce.com/static/${jf}/${jf}.json"
   p4 diff "${jf}.json"
done
```

Verify the contents of the downloaded *.json files contain valid JSON content. Then submit:

```
p4 submit -d "Updated JSON files" "P4*.json"
```

**REMINDER**:  The `DefaultP4Version` changes *only* for GA releases, not patches. Also, the P4*.json files need only be updated for major releases.

JIRA issue bookkeeping (updating `Fix versions`/`P4Blog`, ensuring issues are closed, drafting release-notes content) is ongoing work done throughout the release cycle per Steps 1 and 2 -- see the "Update Release Notes" step below for the actual release-notes authoring, which happens once, later in this process, after Copy Up.

**Output**: The output of this step is that the `get_p4_binaries.sh` script has the latest information on which platforms/architectures are supported by the new P4D version, which is contained in those json files. For example, those files know whether aarch64 (for Graviton processors) is available for a given p4d version.

### Step 8: Load Release Tools

In whichever release workspace client you're currently working from (`dev`, `main`, or the release stream once cut), load that stream's own copy of the release tooling:

```
cd "$WSRoot/tools"
p4 -s sync
source ./env.sh
```

Since `tools/` now lives inside each stream, this needs to be done once per client you switch into during the process, not just once globally.

**Output**: The shell environment now has needed tools in the PATH.

**"Merge Down, Copy Up" is a mantra, not just a step pair -- always Merge Down before a Copy Up, in any direction, at any point in this process (or in general P4 stream workflows), with the rarest of exceptions for deliberate cherry-picking (itself usually an anti-pattern).** A Copy Up done in isolation only pushes the source's content forward -- it cannot detect or preserve a fix that was made *only* in the destination (or, worse, in something further downstream that itself gets periodically refreshed from the destination). Confirmed the hard way twice on 2026-09-04: a `doc/ReleaseNotes.adoc` content gap and a `README.md` link-rendering regression both traced back to a fix landing in `dev` (or a release stream) that was never actually Copied Up into `main`, so a later Copy Up in the other direction silently overwrote it with `main`'s stale content. Before trusting any one-way copy, either do the Merge Down first or explicitly diff the file both directions to confirm nothing downstream-only would be lost.

**Always preview before merging or copying.** Run the `-n` form first -- `p4 merge -n` / `p4 copy -S <stream> -n` -- and look at what it lists before doing the real thing. Then check your own workspace state against that preview: either you have nothing already opened, or nothing already opened overlaps with what the preview would touch, or -- if you're deliberately setting up a merge against files you already have opened -- that's fine too, just make sure it's *deliberate*. An opened file caught up in a merge isn't something to avoid at all costs; it adds some resolve complexity, but that's manageable once you're expecting it. The point of previewing isn't to force a pristine workspace, it's to not be surprised.

### Step 9: Merge Down

Ensure `dev` is updated from `main`, from the **`dev` client**.

This ensures any hot fixes that may have been made directly in `main` are considered for incorporation in the formal release. Streams tracks `dev`'s parent (`main`) automatically, so no branch spec is needed:

```
cd "$WSRoot"
p4 merge -n
```

If there is anything to merge, take care of it in your regular SDP development workspace, not the release workspace. If needed, file new JIRA issues and reference them in the changelist description. Upon completion of any needed merge work, re-run `p4 merge -n` to confirm it's now clean, then do the real merge (`p4 merge`), resolve, and submit.

Note: Any hot fixes should generally be merged down immediately after the hot fix is complete, so that this check is just a formality.  No actual merge work is expected during the release process.  If merge work is required, regression test suites will need to be re-verified.

**Output**: This is the start of the "Merge Down, Copy Up" process, essential to any release process. It ensures any bugs fixed by emergency hot fixes (i.e. those made directly in `main`) don't rear their ugly heads to haunt future releases, including the current one in progress.

### Step 10: Copy Up

Push `dev`'s content up into `main`, from the **`main` client**.

```
cd "$WSRoot"
p4 copy -S //p4-sdp/dev -n
```

Review the preview per the "Always preview before merging or copying" note above, then do the real thing:

```
p4 copy -S //p4-sdp/dev
```

This is an all-or-nothing operation for the whole tree: we don't want a partial release. Proceed *immediately* to the next step once submitted -- everything from here through packaging is done in `main`, using `dev`'s just-copied-up content as the release candidate.

**WARNING FOR PARTIAL RELEASES**: If this is to be a partial release, revert any files that are not ready to be released before submitting the Copy Up changelist, and defer the corresponding JIRA issues' `Fix versions`. Given `dev` is meant to always be release-ready, this should be a rare, exceptional case, not the norm.

**Output**: This is the end of the "Merge Down, Copy Up" process. At this point, `main` has all of the content for the new release, and every remaining step through packaging operates on `main`.

### Step 11: Update Doc Versions (Major Releases Only)

This step applies to major releases only; skip this for patch releases. Done in the **`main` client**.

Update AsciiDoc `revnumber` and `revdate`.

In all *.adoc files, verify that the `:revnumber:` and `:revdate:` tags in the file refer to the correct SDP release version and date the file was last modified -- updating it to the current date.  The *.adoc files appear in the `doc` folder as well as the `Unsupported/doc` folder; ensure both are updated.

The fields look something like this example in each *.adoc file:

```
:revnumber: v2025.2
:revdate: 2026-07-21
```

This can be done like so:

```
cd "$WSRoot"
ThisWorkspace=$(p4 set -q P4CLIENT|cut -d= -f2)
ThisStream=$(p4 -ztag -F %Stream% client -o "$ThisWorkspace")
RevNum=2026.1
RevDate=$(date +'%Y-%m-%d')

p4 --field Description="Updated revnumber and date for release." --field Jobs= --field Files= change -o | p4 change -i
CL=$(p4 -ztag -F %change% changes -s pending -c $ThisWorkspace -m1)

{ for df in $(p4 -ztag -F %depotFile% grep -e :revnumber: "$ThisStream/doc/*.adoc" "$ThisStream/Unsupported/doc/*.adoc"); do f="$(p4 -ztag -F %path% where $df)"; echo -e "\nFile=$f"; p4 edit -c $CL "$f"; sed -e "s|^:revnumber:.*|:revnumber: v$RevNum|g" -e "s|^:revdate:.*|:revdate: $RevDate|g" "$f" > "$f.tmp" && mv "$f.tmp" "$f"; p4 diff "$f"; p4 revert -a "$f"; done; p4 describe -s $CL; } 2>&1 | tee /tmp/docup.log
```

Note: this deliberately avoids `sed -i`. BSD/macOS `sed -i` requires its backup-suffix argument to be given, even if empty, and consumes whatever immediately follows `-i` as that suffix (e.g. `sed -i -e '...'` silently treats `-e` as the suffix and leaves a stray `<file>-e` backup) -- behavior that differs from GNU sed and is easy to trigger by accident. Hit for real during the SDP 2026.1 GA release: it left 15 stray `*.adoc-E` backup files that Step 14's `p4 rec` then tried to add to the depot. The redirect-to-temp-then-`mv` pattern above works identically on both.

Review documents for any outdated references or content.

At this point all *.adoc files will be opened in the default changelist. Generate a pending changelist:

```
p4 --field Description="Updated rev{number,date} fields in adoc files for release." change -o | p4 change -i
```

Review the generated pending changelist, and submit. Ensure before submitting that the changelist contains only the *.adoc files.

```
p4 -s submit -c $CL
```

**Output**: Help prevent documentation from being outdated.

### Step 12: Update TAG Refs in SDP Upgrade Script

This step applies to any release (major or patch) in which changes to the `p4_vars.template` or `instance_vars.template` files were made. Skip this for patch releases in which no changes to templates were made. Done in the **`main` client**.

To verify whether changes are were made to the SDP Environment File since the last release or patch, do:

```
cd "$WSRoot"
LastReleaseCL=$(p4 print -q //p4-sdp/main/Version|cut -d '/' -f 4|cut -d ' ' -f 1)

p4 changes //p4-sdp/main/Server/Unix/p4/common/config/p4_vars.template@$LastReleaseCL,@now

p4 changes //p4-sdp/main/Server/Unix/p4/common/config/instance_vars.template@$LastReleaseCL,@now
```

**If no changes were made, skip to the next step**.  Otherwise, modify the script `$WSRoot/Server/Unix/p4/common/sdp_upgrade/sdp_upgrade.sh` as follows:

* Use the `gen_file_format_check.sh` script to check. If this reports that versions match, **you are done, move on to the next step.**. Otherwise, continue ...
* Open the script for edit and modify it.
* Change the `Version=` value, incrementing the 3rd digit by one. So if the current value is `1.6.4`, increment to `1.6.5`.
* Search the file for `TAG:` comments.
* Update `SDPEnvFileNewFormatVersion` if the format of the SDP Environment File `p4_vars` changed since the last release or patch.
* Update `SDPInstanceCfgFileNewFormatVersion` if the template for the Instance Config File changed since the last release or patch.
* Update the `SDPTargetMajorVersion` to the new version about to be released (Major Releases Only).
* Submit a new version of `sdp_upgrade.sh` when done if changes were needed to the `*FormatVersion*` or `SDPTargetMajorVersion` strings. If it turned out no changes were needed, revert the file (undoing the version number change).

TIP: These template version tags are important as they affect what format versions are displayed in the log when `sdp_upgrade.sh` runs.  However, the upgrade script always regenerates files using the correct/latest available versions of the templates in the release.

**Output**: The output of this step is that SDP Upgrades using the new version display correct output about updates to generated files such as `p4_vars` and the Instance Vars file(s).

### Step 13: Generate Script Docs

This step applies to any release (major or patch). Done in the **`main` client**.

Ensure generated documentation is current.

```
cd "$WSRoot/doc/gen"
./gen_script_man_pages.sh -submit
```

Observe the output.  It may or may not create a new changelist in `doc/gen/...`, depending on whether the various scripts have changed since docs were last generated for them.

This script runs scripts with the `-man` option and stores that in a file, which is then included in `SDP_Guide.Unix.adoc`.

Note: files under `doc/gen/*.man.txt` (and `doc/gen/sdp_install.cfg`) are `isolate`d on `//p4-sdp/main` -- this changelist stays local to whichever stream you submit it in and never flows via Merge Down/Copy Up. That's deliberate: each stream regenerates its own copy independently, avoiding merge conflicts on generated content. It also means this step needs repeating in the release stream once it's cut (see below), since script versions (derived from `$Id:$`/`$Change:$` RCS keywords) differ per stream even when the underlying script content is identical.

**Output**: The output of this step is that documentation generated from shell scripts is current with respect to their corresponding source files.

### Step 14: Generate HTML and PDF Docs

NOTE: For the following to work, you will need some utilities, including `asciidoctor`, `asciidoctor-pdf`, and `markdown`.

NOTE: This is the intended time to regenerate PDFs. During ordinary development, PDFs are deliberately *not* kept in sync with every doc change -- they're large, heavy files with little incremental value day to day, and regenerating them routinely just wastes disk space and bloats changelists. See the SDP Developer Guide's "Documentation Builds" section. This release step is where that gets caught up all at once.

Generate HTML and PDF files from the AsciiDoc (`*.adoc`) and Markdown files (`*.md`). Done in the **`main` client**.

```
cd "$WSRoot/doc"
make clean all
p4 rec
cd "$WSRoot/Unsupported/doc"
make clean all
p4 rec
p4 --field Description="Regenerated docs for release." change -o | p4 change -i
```

Review the generated pending changelist. Ensure before submitting that the changelist contains only expected files, generally *.html and *.pdf files.

If all is well the then use the command below with the appropriate changelist number
```
p4 -s submit -c #####
```

Note: like `doc/gen/*.man.txt`, all of `doc/*.html`, `doc/*.pdf`, `Unsupported/doc/*.html`, and `Unsupported/doc/*.pdf` are `isolate`d on `//p4-sdp/main` -- same reasoning as Step 13, and same requirement to repeat this step in the release stream once it's cut.

**Output**: All documentation is the consumable formats (HTML, PDF) are current with respect to the source formats (Markdown, AsciiDoc).

### Step 15: Update SDP Version File

Done in the **`main` client**, after Copy Up (Step 10) -- by this point `main` already has all of `dev`'s latest content, so this step operates entirely on `main`.

```
cd "$WSRoot/tools"
./preview_changes_and_update_version.sh 2>&1 | tee /tmp/preview.log
```
This script will prompt for the major release version, e.g. `2024.2`.

It will also checkout and display the change it proposes to make to the Version file, something like this:

```
Updating /Users/ttyler/re/p4-sdp/main/Version file.
//p4-sdp/main/Version#152 - opened for edit
==== //p4-sdp/main/Version#152 - /Users/ttyler/re/p4-sdp/main/Version ====
1c1
< Rev. SDP/MultiArch/2025.1/37000 (2025/05/15).
---
> Rev. SDP/MultiArch/2025.2/38000 (2005/12/01).
```

Review the `/tmp/preview.log` to make sure no significant changes were excluded from the release notes.

If the diff of the Version file looks reasonable, proceed.

TIP: The date of the release is the date of the last submit to `main` at this point (i.e. right after Copy Up). Historically and typically this is the same as the date this release process is exercised, but there can be some lag between the last submit and the shipping of the release.

Note: `Version` is `isolate`d on `//p4-sdp/main`, so this file's history is intentionally independent per stream -- it is never merged or copied between streams, only ever directly edited and submitted in whichever stream needs a new marker (here, `main`; later, the release stream will get its own once cut).

**Output**: The output of this step is that SDP Version file is updated. At this point we know the changelist number and date to be associated with the new SDP release.

### Step 16: Update Release Notes

Done in the **`main` client**.

Open the `ReleaseNotes.*` files for edit. In `ReleaseNotes.adoc`, review the titles of the "What's New" sections and update as needed, removing obsolete info and adding a very short summary of the theme of what's included in this new release.

**(Major Releases Only)** Also review and update the "SDP and P4D Version Compatibility" section. SDP intentionally follows the exact same support lifecycle as P4D itself, so this list should always mirror which P4D versions are currently within their maintenance window:

* Add the new release's own P4D version to the top of the supported-versions list.
* Drop any P4D version whose End of Maintenance (EOM) date has already passed, per the "P4 Server (P4D)" table at https://portal.perforce.com/s/article/Helix-Core-Maintenance-Lifecycle-Helix-Core-Server-P4D -- use that table's EOM column as the source of truth, not an assumed fixed-length window.
* Update the "As of `<date>`" wording and the specific version-compatibility example sentence (e.g. "SDP 2025.2 is guaranteed to work with P4D 2025.2") to reference the new release and today's date.

Not needed for patch releases -- the set of currently-supported P4D versions doesn't meaningfully change within a patch cycle.

Then update the **Change History** section, by adding a new section titled something like **SDP 2024.2 Patch 3** to include the list of JIRA issues fixed since the last SDP release or patch.

When editing, follow examples from earlier releases in the file.  In particular, a section of the file will looks something like this snippet, which includes the content of the `Version` file generated in the previous step:

```

== Change History

=== SDP 2024.2 Patch 3:

Released: Rev. SDP/MultiArch/2024.2/31367 (2025/03/23).

Jobs Fixed since change 31193 up to 31367 (B=Bug, D=Doc, F=Feature):

* https://perforce.atlassian.net/browse/SDP-1218[SDP-1218] (B): mkdirs.sh overwrites existing crontab when adding multiple instances.
* https://perforce.atlassian.net/browse/SDP-1217[SDP-1217] (B): Adjust perforce_bashrc template to avoid setting HISTFILE if it is readonly.

... and so on ...
```

To get the list of JIRA issues, run a JIRA report of SDP issues with the target `Fix versions`, e.g. `2024.2.3`, and ensure all issues listed in the report are considered for inclusion in the release notes.

It is also a good idea to run a report of all open JIRA issues, to include those with an unassigned or different value for `Fix versions`, and consider if they could or should be included in the current release.

We don't need a JIRA issue for every change.  Create JIRA issues to cover the essentials of what changed.  Folks who want the gory details can look at changelists, but folks looking only at the release notes should get a useful summary of what's new in this release or patch, and if appropriate, what to be aware of when upgrading to this release.

Some JIRA issues may be excluded from the public facing release notes if they're not helpful or relevant in helping a customer understand what changed.

Much of JIRA management work should be done during the development, mixed with work from Steps 1 and 2 above. At release time, this check should be a careful double-check to ensure release notes are complete and correct.

Then regenerate the ReleaseNotes.{html,pdf} files, and review them visually to ensure they look good and contain necessary info.

```
make ReleaseNotes.{html,pdf}
```

At this point, there should be a changelist with these files checked out:

```
$ p4 opened
//p4-sdp/main/Version#152 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.adoc#104 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.html#86 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.pdf#64 - edit default change (binary)
```

As a final double-check, bing up the JIRA release and make sure all associated issues are marked as done and completed. If any are not done or not indicated as done, we need to make corrective actions. Actions may include:

* Updating JIRA issues status to match reality, e.g. closing an issue if the code has already been delivered and tested.
* Updating reality to match JIRA reported issue status.
* Deferring: change the `Fix versions` of any features that are to be deferred until a future release.

Then submit:

```
p4 submit -d "Updated Version and Release Notes."
```

**Output**: The Version file and Release Notes are updated for the release, entirely within `main`.

### Step 17: Cut the Release Stream

Create the new release stream from `main`, e.g. `//p4-sdp/r26.1.0` for a GA release, or `//p4-sdp/r26.1.1` for the first patch after it. This is the first step in this process with no Classic-era equivalent -- Classic had no per-release streams, `main` itself served as the release archive.

Template the new stream's spec from `main` (so it correctly inherits `main`'s Paths, including the `isolate` rules for generated content and for `ai_dev_support`/`Version`):

```
p4 stream -o //p4-sdp/main | sed \
   -e "s@^Stream:.*@Stream:\t//p4-sdp/r26.1.0@" \
   -e "s@^Type:.*@Type:\trelease@" \
   -e "s@^Parent:.*@Parent:\t//p4-sdp/main@" \
   -e "s@^Name:.*@Name:\tr26.1.0@" \
   | grep -v '^Update:\|^Access:' \
   | p4 stream -i
```

(Replace `r26.1.0` with the actual release name being cut.)

Populate the new stream's initial content wholesale from `main`:

```
p4 populate -r -S //p4-sdp/r26.1.0 -d "Initial population of r26.1.0 from main."
```

**Don't drop the `-r`**: `-S stream` on its own populates *from* the new (empty) stream *into* its parent -- exactly backwards from what's needed here, and it'll fail outright once `main` has any content ("Can't populate target path when files already exist"). `-r` reverses the direction to parent -> child, which is what actually cuts the release stream's initial snapshot.

Then create a new release workspace client switched to it (e.g. rooted at `~/re/p4-sdp/r26.1.0`) and sync it.

**Output**: A new, long-lived, stable release stream exists, holding a snapshot of everything in `main` as of this release.

### Step 18: Final Regen in the Release Stream

Done in the **new release stream's client**.

Because script versions are derived from `$Id:$`/`$Change:$` RCS keywords (see the SDP Bash Coding Standard's Version Identification section), the same script's version string differs between `main` and the just-cut release stream, even though the content is identical -- submitting/populating a file into a new stream changes its `$Change:$` value. Since `doc/gen/*.man.txt`, `doc/*.html`, `doc/*.pdf`, and their `Unsupported/doc` equivalents are all `isolate`d (see Steps 13-14), they were never copied into the release stream by the populate above with correct version strings -- they need regenerating here, once, in the release stream itself:

```
cd "$WSRoot/doc/gen"
./gen_script_man_pages.sh -submit

cd "$WSRoot/doc"
make clean all
p4 rec
cd "$WSRoot/Unsupported/doc"
make clean all
p4 rec
p4 --field Description="Final doc regen for r26.1.0." change -o | p4 change -i
```

Review and submit the resulting changelist, same as Steps 13-14.

`Version` is *also* `isolate`d, and -- unlike the generated docs above -- nothing regenerates it automatically; the populate simply never carried it into the release stream at all. `package_downloads.sh` (Step 19) depends on it for version-stamping the tarball, so seed it now with the same content `main` had when this release was cut (this stream *is* that release, so there's nothing to regenerate, just copy):

```
cd "$WSRoot"
cp "$MainWSRoot/Version" Version
p4 add Version
p4 submit -d "Add Version file to r26.1.0."
```

If a bookmarked link to `main`'s docs shows "MAIN" in a script's version tag rather than a release number, that's expected and correct, not a bug -- it's an honest signal that the reader is viewing the rolling `main`, not a specific release.

**Output**: The release stream's own generated documentation correctly reflects its own release-tagged script versions, not `main`'s.

### Step 19: Package

Package the release by creating the tarball. Done in the **release stream's client** for a GA/patch release, or the **`dev` client with `-P`** for a pre-release build.

```
cd "$WSRoot/tools"
bash package_downloads.sh
```

See the macOS note in "Preparation" above -- this script needs bash 4.x+; if a plain `bash` on your PATH still resolves to macOS's system one, invoke your modern bash explicitly instead (e.g. `/opt/homebrew/bin/bash package_downloads.sh`).

The `package_downloads.sh` script puts tar and zip files into the `downloads` folder (excluded from the package itself, along with `tools/` and `ai_dev_support/`), and removes older versions.

Review and submit the generated changelist manually.  In the output of the script, you should see a message illustrating the exact command you need, something like:

```
To complete packaging, run:
	p4 -s submit -c 31370
```

Note: `downloads/` is `isolate`d on `//p4-sdp/main` -- this changelist stays local to the release stream and never flows via Merge Down/Copy Up. That's deliberate: a pre-release (`-P`) build in `dev` must never leak into `main` via ordinary Copy Up, and each stream's tarball is a build artifact, not source content. This is also why the next step exists.

**Output**: The new SDP tarball exists in the release stream, ready to be published to its permanent, version-independent download location.

### Step 20: Publish the Tarball to `main`

Done in the **`main` client**, immediately after Step 19.

`install_sdp.sh` deliberately downloads the SDP tarball from a fixed, permanent URL under `main` (`.../download/p4-sdp/main/downloads/sdp.Unix.tgz`), not from any specific release stream's own URL -- this mirrors the single fixed download location the Classic depot always had, and means `install_sdp.sh` never needs a URL edit for a future release. Since `downloads/` is `isolate`d, Copy Up (Step 10) does **not** carry the new tarball into `main` -- `main`'s `downloads/` still holds the *previous* release's real tarball at this point, which is exactly why nothing was ever broken earlier in this process: that fixed URL has been resolving to a real, working (if not-yet-current) tarball the entire time. This step is what makes it current:

```
cd "$WSRoot"
mkdir -p downloads
cp "$OtherWSRoot/downloads/"*.tgz "$OtherWSRoot/downloads/"*.zip downloads/
p4 add -t binary+F -f downloads/*.tgz downloads/*.zip 2>/dev/null; p4 edit downloads/*.tgz downloads/*.zip 2>/dev/null
p4 --field Description="Publish r26.1.0's tarball to main's permanent download location." change -o | p4 change -i
```

(Replace `$OtherWSRoot` with the release stream client's root from Step 19, and remove any older-version-named files from `main`'s `downloads/` first, matching `package_downloads.sh`'s own "removes older versions" behavior.)

`downloads/` lives at the stream root, sibling to `tools/`, not inside it -- easy to get wrong since you're likely still sitting in `tools/` from Step 19. Always pass `-t binary+F` explicitly on `add`: if the destination path is ever wrong (so no file exists there yet to inspect), `p4 add` silently falls back to guessing `text`, which is wrong for a tarball and isn't corrected just by fixing the path afterward -- confirmed the hard way while exercising this step for real.

Review the generated pending changelist -- it should contain only the tarball/zip files, with no version-stamped duplicates left behind from prior releases -- then submit.

**Output**: `main`'s fixed, permanent download URL now serves the new release's tarball. At no point during this entire process did that URL ever stop resolving to a real, working tarball -- it simply switched from serving the previous release to serving this one. This completes the public aspects of the release process.

### Step 21: Release in JIRA

Set the date for the release in JIRA, and then execute the Release action.

**Output**: Information in JIRA is correct. This completes the internal tracking of the release process.

### Step 22: Announce the Release

Post an announcement of the release to the internal `#p4-sdp` Slack channel. Include the version string (from `Version`), a link to the Release Notes (`ReleaseNotes.adoc`/`.html`), and a link to the download page.

**Output**: The internal team is aware the release has shipped, without having to notice it independently via JIRA or Workshop.

### Step 23: Post-Release Smoke Tests

Done immediately after release, outside of any release workspace client -- these are manual, real-world exercises, not automation.

1. **Fresh install, from scratch.** Build a fresh VM and follow the documented install-from-scratch procedure using `install_sdp.sh`, exactly as a new customer would. This tests the docs and the script together, not just the script in isolation. After install, exercise a couple of the everyday admin scripts by hand -- e.g. `live_checkpoint.sh` and `daily_checkpoint.sh` -- to confirm the installed result actually works day to day, not just that the installer exited cleanly.

2. **Real upgrade, dogfooded.** Pick an existing live SDP environment and execute the documented upgrade procedure against it for real. This isn't really a test at this point -- it's dogfooding: upgrading a production environment (e.g. the PPN or Public Depot servers) immediately after the release ships, or upgrading a Gen6 lab environment running an older SDP release. Like the fresh install, this exercises the docs and automation together, using the same procedure a real customer would follow.

**Output**: Confidence that the release actually works end-to-end in real environments, as documented, not just in the automated regression suites.

## Hot Fixes and Emergency Bug Fixes

The `dev` -> `main` -> release-stream flow (Steps 9-20 above) is the preferred, intended way for content to reach a release. It is not a rigid one-way gate enforced by tooling, though: release-type streams' `notoparent`/`nofromparent` Options (see Step 17) make copying or merging to/from a release stream a deliberate, manual act -- you have to explicitly flip those Options, do the copy/merge, and flip them back -- but they don't forbid a direct edit in the stream itself. Whether and when to make a direct patch, rather than waiting for the next formal release, is left to developer discretion.

**Hot Fix** (doc/web-page-only, the common case): A direct edit made in a release stream (or in `main`, ahead of the next Copy Up) that touches only `isolate`d, non-shipped content -- typically `doc/*.html`/`doc/*.pdf`, `README.md`, or similar web-facing pages -- and does not affect the packaged tarball. Because `Version` and the generated docs are `isolate`d per stream, a doc-only hot fix in a release stream has no bearing on that stream's `Version` file or its tarball -- nothing needs bumping or republishing. This is the preferred, lower-risk form of exception to the normal flow, and the one hot fixes should stick to whenever possible -- but it's not an absolute rule; see Emergency Bug Fix below for when it isn't enough.

A hot fix made directly in `main` should generally be merged down into `dev` promptly (see Step 9) so it isn't lost or re-diverged later; a hot fix made directly in an already-cut release stream should likewise generally be merged down into `main` (and from there, `dev`) at the developer's discretion. As with `main`-level hot fixes, this is a "should," not a tool-enforced requirement.

**Emergency Bug Fix** (touches shipped software, the rare/exceptional case): A direct edit made in an already-cut release stream that changes something which actually ships in that stream's tarball -- a script, not just a doc page. This bypasses the normal `dev` -> `main` -> release flow on purpose, to get a critical fix out faster than a full new release (e.g. `r26.1.1`) would allow. **The one non-negotiable invariant: if a release stream's shipped content changes, that stream's own `Version` file must be bumped and its tarball republished (repeat Steps 18-20, in that stream) before anyone can rely on the fix.** Never let a release stream's tarball and its `Version` file drift out of sync -- a stale `Version` file describing content that no longer matches the actual tarball defeats the `-V` version-check convention (see the SDP Developer Guide's Script Versioning section) that scripts and support depend on to know what they're actually running.

An Emergency Bug Fix is exceptional by design -- if it starts feeling routine, that's a signal to cut a real Patch Release (see the SDP Developer Guide's Terminology section) instead. For the policy on *when* an Emergency Bug Fix is the right call versus cutting a real Patch Release (weighing how much changed, how severe the bug is, and how long it's been since the release shipped), see the SDP Developer Guide's "Choosing an Emergency Bug Fix vs. a Patch Release" section -- this document covers the mechanics of doing one, not the judgment call of whether to.

**The correct order of operations** (learned by getting it wrong the first time, during the 2026.1 GA cycle): fix the code first, but **update `ReleaseNotes.adoc` before packaging**, not after -- it's easy to jump straight to repackaging/publishing once the code fix looks good, only to realize the release notes still need the entry, forcing a second (avoidable) repackage-and-republish cycle. The full sequence:

1. Fix the code directly in the release stream.
2. Update `ReleaseNotes.adoc`'s existing Change History entry for this release: bump the `Released:` line and the `Jobs Fixed since change X up to Y` range to the changelist containing the fix, and add a new bullet at the top of that release's Jobs Fixed list (highest JIRA number first, per the existing convention). Keep the wording calm and factual -- describe what was wrong and that it was corrected quickly, without alarming language; a same-day fix is a sign the process is working, not something to downplay via omission but also not something to over-dramatize. The `Released:`/`up to` changelist number is necessarily a best-effort estimate, not an exact promise -- **never try to reference an exact future changelist number by reserving a pending changelist first** (e.g. `p4 change -i` to get a number, then referencing it in a doc before submitting). Perforce only guarantees submitted changelist numbers increase monotonically in submission order; a pending changelist can be silently renumbered at submit time if anything else submits on the server in between, breaking the reference. Pick a reasonable number (e.g. the changelist you expect step 5's repackage to land on) and accept that it may end up slightly stale once later, purely-editorial changes follow it -- this is already the norm elsewhere in this file's Change History (e.g. the 2026.1 GA entry's `Released:` line stayed at the code-fix changelist even as later doc-only changes raised the stream's actual head revision).
3. Regenerate `ReleaseNotes.html` **and `ReleaseNotes.pdf`** -- unlike routine development (see "Documentation Builds" policy, PDF-at-release-time-only), an Emergency Bug Fix to an already-released stream *is* a release event, so the PDF gets regenerated too, not just HTML. Also regenerate any script man page whose embedded `-h`/`-man` usage text changed as part of the fix (not just its functional behavior) -- check for this explicitly, since it's easy to fix the behavior and miss that the same stale text is also baked into the shipped usage output. **`ReleaseNotes.html`/`.pdf` and `SDP_Guide.Unix.html`/`.pdf` are two separate regeneration targets, built from two separate `.adoc` files** -- regenerating one does not regenerate the other, and it's easy to do the man-page/`SDP_Guide.Unix` regen (needed because a script's usage text changed) and mentally check off "docs regenerated" without separately regenerating `ReleaseNotes.html`/`.pdf` for the Change History entry from step 2. Confirmed the hard way: an EBF shipped with `ReleaseNotes.adoc`'s source correctly updated but the shipped `ReleaseNotes.html`/`.pdf` still stale, because only `SDP_Guide.Unix.html`/`.pdf` had been regenerated. Also don't assume `doc/*.html`/`*.pdf` are purely web-facing and excluded from the tarball -- check `tools/excludes.downloads.cfg`; unless a directory is explicitly excluded (e.g. `D:tools`, `D:downloads`), `doc/` ships inside the tarball as-is, `ReleaseNotes.html`/`.pdf` included.
4. Bump the release stream's `Version` file to the changelist containing the code fix (man-page regens from step 3 count as part of "the fix"; a later, purely-editorial `ReleaseNotes.adoc`/HTML/PDF-only change does not need its own separate `Version` bump -- `Version` tracks the last change that affects actual script behavior or its packaged usage text, not documentation-only follow-ups).
5. Repackage the tarball.
6. **Verify the fix for real before publishing** -- ideally with the actual scenario the bug affected (e.g. a real install to an old release followed by a real upgrade to the fixed tarball), using the freshly-built local tarball directly (no need to wait for a publish/download round-trip to test it). Publishing before verifying, then discovering the fix needs another pass, means burning a second full package/publish cycle -- verify first if at all possible.
7. Publish the tarball to `main`'s permanent download URL.
8. Confirm the live public download URL actually serves the fix (a fresh `curl`/`tar` round-trip against the real URL, not just trusting the submit).
9. **Merge Down the fix all the way to `dev`** -- release-stream -> `main` -> `dev`. This is the one step it's genuinely easy to forget once the fix is live and verified, but skipping it means `dev` quietly drifts out of sync, and the exact same bug can resurface the next time a release is cut from a stale `dev`. Directionally, since the fix originated at the release-stream end (north of `main`) rather than in `dev` (south of `main`), only the southward Merge Down leg is needed here -- there's no corresponding Copy Up, because nothing originated in `dev` that needs to travel north. Use `p4 merge` (not `-S <stream>` when your client is already dedicated to the target stream -- that confuses the direction; just run plain `p4 merge`/`p4 copy` and let it infer parent/child from the client's own stream). Expect this to also sweep up any other content that independently accumulated in `main` since the last Merge Down (e.g. a backlog of unrelated doc-only changes) -- that's normal, not a sign something went wrong; check each file's actual diff before blindly `-am` resolving, since most will be trivial (RCS keyword lines, `:revdate:` stamps) but a few may have real, unique content on both sides that needs a genuine 3-way resolve. If any stream involved (`dev` or a release stream) has files already checked out by a human when you go to merge, stop and coordinate before proceeding -- don't silently revert or overwrite someone else's in-progress work.

**AI-specific gotchas hit doing this for real:**
- `bot_Claude_Anthropic`'s write access to a release stream can be blocked by the same IP-restricted-protections-plus-VPN-routing issue documented elsewhere in this skill for `test-install_sdp` -- if `p4 edit`/`p4 populate` fails with "no permission" despite an apparently-matching `write` rule, check whether a group-based, IP-*unrestricted* grant exists (or ask for one) rather than chasing the network path.
- Tools like `gen_script_man_pages.sh` and `package_downloads.sh` shell out to plain `p4` internally, which does not inherit an explicit `-p`/`-u` override passed to the outer script -- if the ambient `P4PORT` resolution is ambiguous (see the AI-specific shell-environment notes earlier in this doc), `export P4PORT=...` and `export P4USER=...` as real environment variables before running them, not just via `-p`/`-u` flags on your own commands.
- When running `p4 merge`/`p4 copy` from a client that's already dedicated to the target stream, don't also pass `-S <that-same-stream>` -- it confuses the direction and produces a wrong/misleading error (e.g. "needs copy not merge in this direction" when the real issue is nothing to do with copy vs. merge at all). Just run the plain command with no `-S`; it correctly infers the client's own stream and its parent.
- Step 6's "verify with the freshly-built local tarball" only actually tests the local tarball if you run it through `sdp_upgrade.sh` against a locally-staged copy (or `-local`-mode `install_sdp.sh`). Running plain `install_sdp.sh` from within the extracted local tree, without `-local`, still fetches its own fresh copy of the SDP tarball from `main`'s live download URL -- so a naive "extract and run install_sdp.sh" verification actually re-tests the **still-buggy, not-yet-published** live tarball, not the fix. Confirmed the hard way: a first verification attempt reported success while still showing the old bug, because it silently downloaded the old tarball instead of using the new one sitting right next to it. Use the upgrade-test pattern instead (stage the local tarball into `/opt/perforce/helix-sdp/downloads/sdp.Unix.tgz`, then run `sdp_upgrade.sh` against it) even when what you actually want to validate is a fresh-install scenario.