# SDP Release Process Overview

## Introduction

This document describes the process used to release the Server Deployment Package (SDP). This is intended for those developing the SDP.

## Preparation

A dedicated SDP workspace should be used for releasing the SDP, separate from any you might use for SDP development. This dedicated release workspace should not contain any cruft files such as editor temp files.  The release process mechanisms prevents known-cruft files from being added such as `*~` files and `.DS_Store` files on Mac.  Using a clean dedicated release workspace guarantees no local excess files can sneak into the packaging process.

## Procedure

### Step 1: Plan!

Use [this JIRA Project](https://perforce.atlassian.net/jira/software/c/projects/SDP/issues) to plan your work.  This JIRA instance is only accessible with Perforce corporate SSO.

The SDP product owner determines what the release will be.  For example **2025.1** would be the **SDP 2025.1, GA Release**, while **2025.1.1** would be **SDP 2025.1 Patch 1**.

The SDP shares the same major version as P4 Server (e.g. 2025.1).  Versions in JIRA use the major version number, and append an integer as the patch identifier. The patch number is omitted for the GA release, and increments starting from 1 for each patch after a major release.  While SDP major releases are bound to the P4D release schedule, SDP patches release on an independent schedule.

In JIRA, create Releases to be used for `Fix version` identifiers, such as 2024.2, 2024.2.1, 2025.1, etc.  The name of the release should following YYYY.M for a major release, or YYYY.M.P for a patch release.

For example, in JIRA, there will be a release named 2024.2.3, with a Description of "SDP 2024.2 Patch 3", and a 2025.1 release with a description of "SDP 2025.1."

Create JIRA search filters for each `Fix version`, e.g. `2025.1` for the SDP 2025.1 GA release, or `2024.2.3` for the SDP 2024.2 Patch 3 release. There is also a standing `SDP To Do` filter with all open jobs used during planning.

Also, create JIRA search filters for each `Fix version`, e.g. `SDP 2025.1`. There will also be a fixed `SDP To Do` filter with all open jobs used during planning.

Generally, while some planning occurs early in the release cycle, other elements of planning occur iteratively and continuously throughout the SDP release cycle.

**Output**: The output of this step is a set of JIRA issues associated with `Fix versions`.

### Step 2: Develop!

Develop and test changes in dev branch.  This will occur over time.

When an issue is marked as complete, do the following in JIRA to get it included in the Release Notes:

* Set the `Fix versions` JIRA issue field value to the next release, e.g. `2024.2`.
* Update the `P4Blog` field of the JIRA issue with an indication of what change addressed the issue.  This can be a review URL from The Workshop, e.g. https://swarm.workshop.perforce.com/reviews/29811, or could also be the URL for the changelist in the SDP dev branch where the fix was made.

**Output**: The output of this step is an improved software product and information needed to create release notes.

### Step 3: Target New P4D in SDP Regression Test Suite (Major Releases Only)

This step applies to major releases only; skip this for patch releases.

Update the SDP regression test suite to ensure that it references the new P4 Server release.

Here is a sample of the required change:

```
Change 29229 by tom_tyler@tom_tyler.P4MBPro4.test_sdp_dev on 2022/12/04 04:27:34

	Accounted for change to default P4 Server version.

Affected files ...

... //p4-sdp/dev/dev/Server/test/test_Upgrade.py#17 edit

Differences ...

==== //p4-sdp/dev/Server/test/test_Upgrade.py#17 (text+x) ====

95c95
<     parser.add_argument('--to-version', default="24.2")
---
>     parser.add_argument('--to-version', default="25.1")
```

This code change might possibly break the regression.  It is important to re-verify that the regression test suite is still successful after this change, adapting as needed.

**Output**: This step ensures the new P4 Server version doesn't break SDP regression tests suites, and to raise awareness of any issues that may cause customer issues when adopting the new release.

### Step 4: Verify SDP Regression Tests

Verify that all SDP regression test suites are successful.

All regression test suites target the //p4-sdp/main stream.

#### Regression Test Suite 1: Docker/Jenkins

The first test suite is the Docker-based test suite with a Jenkins server running and polling for changes.

The [SDP regression test suite](https://workshop-jenkins.das.perforce.com:8080/jobs/sdp-dev) is only accessible from the Perforce corporate VPN.

This uses files in `//p4-sdp/main/test/...` (except the `bsw` subdirectory) and `//p4-sdp/main/Server/test`.

#### Regression Test Suite 2: Dedicated Battle School Workshop

The second test suite is the Battle School Workshop test suite, which operates in a dedicated Battle School Workshop Lab Environment.

This uses files in `//p4-sdp/main/test/bsw/...`

Tests are started by calling the `lab qa` command as the `student` user in a dedicated Battle School Lab, and following the instructions that appear in a browser when `lab qa` is run.  (As of SDP 2024.2 Patch 3 and Battle School Gen 6, tweaks are needed to the standard Battle School Lab to sync the extra files needed to support doing a `lab qa` command. The Battle School Gen 7 environment will support running the test suite without additional changes).

The tests in this test suite are defined in `cli_tests.cfg` and `scripted_tests.cfg` in the `/p4/sdp/test/bsw` folder of the SDP, and are initiated with the `run_all_tests.sh` wrapper script.  More info is available in this directory.

#### Regression Test Suite 3: The install_sdp.sh test servers.

A fleet of EC2 instances of various operating ensure that the `install_sdp.sh` script and its dependencies run on a variety of supported Linux distros. This set of standalone servers in AWS region us-east-2 are home to the dedicated test suite for the `install_sdp.sh` script. For optimal realism, these do not run in containers.  The following operating systems are currently represented:

* Ubuntu 20.04
* Ubuntu 22.04
* Ubuntu 24.04
* SuSE 15
* Rocky 9

**Output**: The output of this step is confidence we're not regressing.

### Step 5: Determine Release Type

Determine if the SDP release is to be Major, Patch, or Doc Only.

Since the 2021.2 release of P4D, SDP major releases have been tied to the P4D release schedule, **with SDP releases occurring approximately 10-15 business days after P4D releases**.

Other releases will be Patch releases if there are any functionality changes, or Doc Only releases, where changes since the last release affect only documentation. The SDP documentation is key to its value, worth doing patch updates including only documentation changes.

**Output**: The output of this step is an intended SDP major version number and type, e.g. "2024.2 Patch 3", or "2025.1 GA".

### Step 6: Update Default P4D Version (Major Releases Only)

This step applies to major releases only; skip this for patch releases.

Modify the script `//p4-sdp/dev/p4_binaries/get_p4_binaries.sh` as follows:

* Change the `DefaultP4Version` value to match the current P4D GA release version with a value like `r24.2`.  The binaries for this release should be accessible under https://ftp.perforce.com/perforce/r24.2 (replacing `r24.2` with the new value).

**Output**: The default P4 Server version acquired from our FTP server with the `get_p4_binaries.sh` script reflects the new major version for the P4 Server.

### Step 7: Update P4*.json release list files (Major Releases Only)

This step applies to major releases only; skip this for patch releases.

Edit the *P4*.json release list files used by `get_p4_binaries.sh`.

Starting from the root of your SDP workspace, do:

```
cd p4_binaries
p4 edit P4*.json
for jf in P4 P4D P4Broker P4Proxy; do
   curl -O "https://updates.perforce.com/static/${jf}/${jf}.json"
   p4 diff "${jf}.json"
done
```

Verify the contents of the downloaded *.json files contain valid JSON content. Then submit:

```
p4 submit -d "Updated JSON files" "P4*.json"
```

**REMINDER**:  The `DefaultP4Version` changes *only* for GA releases, not patches. Also, the P4*.json files need only be updated for major releases.

**Output**: The `get_p4_binaries.sh` script has the information it needs to get correct versions on what platforms/architectures are supported by the new P4D version, which is contained in those json files. For example, those files know whether aarch64 (for Graviton processors) is available for a given p4d version.

Update the doc/ReleaseNotes.adoc file, specifically the **Change History** section, by adding a new section titled something like **SDP 2024.1 Patch 1** to include the JIRA issues fixed since the last SDP release or patch.

Run a JIRA report of SDP issues with the target `Fix versions`, e.g. `2024.2.4`, and ensure all issues listed in the report are considered for inclusion in the release notes.

Ensure any JIRA issues to be in the release are closed.  If any issue is not closed, close it to reflect that the work is done if indeed the technical work is complete. Otherwise, either complete the technical work or defer the issue by clearing or changing the value for the `Fix versions` field in JIRA.

We don't need a JIRA issue for every change.  Create JIRA issues to cover the essentials of what changed.  Folks who want the gory details can look at changelists, but folks looking only at the release notes should get a useful summary of what's new in this release or patch, and if appropriate, what to be aware of when taking this release.

Some JIRA issues may be excluded from the public facing release notes if they're not helpful or relevant in helping a customer understand what changed.

This JIRA management work is done during the development, mixed with work from Steps 1 and 2 above. At release time, the check in this step should be a formality to ensure release notes are complete and correct.

**Output**: Our Issue Tracking system has reliable information to be used in Release Notes.

### Step 8: Load Release Tools

Load the latest SDP tools shell environment.

Change to the dir in your SDP workspace corresponding to `//p4-sdp/main/tools`, and do `p4 -s sync` followed by `source ./env.sh` in that directory.

**Output**: The shell environment now has needed tools in the PATH.

### Step 9: Merge Down

Ensure the `dev` branch is updated from main.

This ensures any hot fixes that my have been made directly in `main` branch are considered for incorporation in the formal release.

```
   $ p4 merge -n -b perforce_software-sdp-dev
   All revision(s) already integrated.
```

If there is anything to merge, take care of it in your regular SDP development workspace, not the release workspace. If needed, file new jobs and associate them with changes. Upon completion of any needed merge work.

Note: Any hot fixes should generally be merged down immediately after the hot fix is complete, so that this check is just a formality.  No actual merge work is expected during the release process.  If merge work is required, regression test suites will need to be re-verified.

**Output**: This is the start of the "Merge Down, Copy Up" process, essential to any release process. It ensures any bugs fixed by emergency hot fixes (i.e. those made directly in release streams) don't rear their ugly heads to haunt future releases, including the current one in progress.

### Step 10: Update Doc Versions (Major Releases Only)

This step applies to major releases only; skip this for patch releases.

Update AsciiDoc `revnumber` and `revdate`.

In all *.adoc files, verify that the `:revnumber:` and `:revdate:` tags in the file refer to the correct SDP release version and date the file was last modified -- updating it to the current date.  The *.adoc files appear in the `dev/doc` folder as well as the `Unsupported/dev/doc` folder; ensure both are updated.

The fields look something like this example in each *.adoc file:

```
:revnumber: v2024.2
:revdate: 2024-12-20
```

This can be done like so:

```
Workspace=$(p4 set -q P4CLIENT|cut -d= -f2)
p4 --field Description="Updated revnumber and date for release." --field Jobs= --field Files= change -o | p4 change -i
CL=$(p4 -ztag -F %change% changes -s pending -c $Workspace -m1)

{ for df in $(p4 -ztag -F %depotFile% grep -e :revnumber: "//p4-sdp/main/doc/*.adoc" "//p4-sdp/main/Unsupported/doc/*.adoc"); do f="$(p4 -ztag -F %path% where $df)"; echo -e "\\nFile=$f"; p4 edit -c $CL "$f"; sed -i -E -e "s|^:revnumber:.*|:revnumber: v2024.2|g" -e "s|^:revdate:.*|:revdate: 2024-12-20|g" "$f"; p4 diff "$f"; done; p4 describe -s $CL; } 2>&1 | tee /tmp/docup.log
```

Review documents for any outdated references or content.

At this point all *.adoc files will be opened in the default changelist. Generate a pending changelist:

```
p4 --field Description="Updated rev{number,date} fields in adoc files for release." change -o | p4 change -i
```

Review the generated pending changelist, and submit. Ensure before submitting that the changelist contains only the *.adoc files.

```
p4 -s submit -c $CL
```

**Output**: Help prevent documentation from being outdated.

### Step 11: Update TAG Refs in SDP Upgrade Script

This step applies to any release (major or patch) in which changes to the `p4_vars.template` or `instance_vars.template` files were made. Skip this for patch releases in which no changes to templates were made.

To verify whether changes are were made to the SDP Environment File since the last release or patch, do:

```
LastReleaseCL=$(p4 print -q //p4-sdp/main/Version|cut -d '/' -f 4|cut -d ' ' -f 1)

p4 changes //p4-sdp/main/Server/Unix/p4/common/config/p4_vars.template@$LastReleaseCL,@now

p4 changes //p4-sdp/main/Server/Unix/p4/common/config/instance_vars.template@$LastReleaseCL,@now
```

**If no changes were made, skip to the next step**.  Otherwise, modify the script `//p4-sdp/main/Server/Unix/p4/common/sdp_upgrade/sdp_upgrade.sh` as follows:

* Use the `gen_file_format_check.sh` script to check. If this reports that versions match, **you are done, move on to the next step.**. Otherewise, continue ...
* Open for edit and modify the script: `//guest/perforce_software/sdp/dev/Server/Unix/p4/common/sdp_upgrade/sdp_upgrade.sh`
* Change the `Version=` value, incrementing the 3rd digit by one. So if the current value is `1.6.4`, increment to `1.6.5`.
* Search the file for `TAG:` comments.
* Update `SDPEnvFileNewFormatVersion` if the format of the SDP Environment File `p4_vars` changed since the last release or patch.  Check with 
* Update `SDPInstanceCfgFileNewFormatVersion` if the template for the Instance Config File changed since the last release or patch.
* Update the `SDPTargetMajorVersion` to the new version about to be released (Major Releases Only).
* Submit a new version of `sdp_upgrade.sh` when done if changes were needed to the `*FormatVersion*` or `SDPTargetMajorVersion` strings. If it turned out no changes were needed, revert the file (undoing the version number change).

TIP: These template version tags are important as they affect what format versions are displayed in the log when `sdp_upgrade.sh` runs.  However, the upgrade script always regenerates files using the correct/latest available versions of the templates in the release.

**Output**: The output of this step is that SDP Upgrades using the new version display correct output about updates to generated files such as `p4_vars` and the Instance Vars file(s).

### Step 12: Generate Script Docs

Ensure generated documentation is current.

Starting from the root of your SDP workspace, do:

```
cd doc/gen
./gen_script_man_pages.sh -submit
```

Observe the output.  It may or may not create a new changelist in `//p4-sdp/main/doc/gen/...`, depending on whether the various scripts have changed since docs were last generated for them.

This script runs scripts with the `-man` option and stores that in a file, which is then included in `SDP_Guide.Unix.adoc`.

**Output**: The output of this step is that documentation generated from shell scripts is current with respect to their corresponding source files.

### Step 13: Generate HTML and PDF Docs

NOTE: For the following to work, you will need some utilities, including `asciidoc`, `asciidoc-pdf`, and `markdown`.

Generate HTML and PDF files from the AsciiDoc (`*.adoc`) and Markdown files (`*.md`).

Starting from the root of your SDP workspace, do:

```
cd doc
make clean all
p4 rec
cd ../Unsupported/doc
make clean all
p4 rec
p4 --field Description="Regenerated docs for release." change -o | p4 change -i
```

Review the generated pending changelist. Ensure before submitting that the changelist contains only expected files, generally *.html and *.pdf files.

If all is well the then use the command below with the appropriate changelist number
```
p4 submit -c #####
```

**Output**: All documentation is the consumable formats (HTML, PDF) are current with respect to the source formats (Markdown, AsciiDoc).

### Step 14: Update SDP Version File

Change to the directory in your SDP workspace corresponding to `//p4-sdp/main/tools`, and use a script to determine the release changelist, which will be the latest changelist in the dev branch.

Starting from the root of your SDP workspace, do:

```
cd tools
./preview_changes_and_update_version.sh 2>&1 | tee /tmp/preview.log
```
This script will prompt for the major release version, e.g. `2024.2`.

It will also checkout and display the change it proposes to make to the Version file, something like this:

```
Updating /Users/ttyler/re/p4-sdp/Version file.
//p4-sdp/main/Version#152 - opened for edit
==== //p4-sdp/main/Version#152 - /Users/ttyler/re/p4-sdp/Version ====
1c1
< Rev. SDP/MultiArch/2025.1/37000 (2025/05/15).
---
> Rev. SDP/MultiArch/2025.2/38000 (2005/12/01).
```

Review the `/tmp/preview.log` to make sure no significant changes were excluded from the release notes.

If the diff of the Version file looks reasonable, proceed.

TIP: The date of the release is the date of the last submit to the `dev` branch. Historically and typically this is the same as the date this release process is exercised, but there can be some lag between the last dev branch change and the shipping of the release)

**Output**: The output of this step is that SDP Version file is updated. At this point we know the changelist number and date to be associated with the new SDP release.

### Step 15: Update Release Notes

Change to the dir in your SDP workspace corresponding to `//guest/perforce_software/sdp/dev/doc`.

Open the `ReleaseNotes.*` files for edit. In `ReleaseNotes.adoc`, review the titles of the "What's New" sections and update as needed, removing obsolete info and adding a very short summary of the theme of what's included in this new release.

Then update the **Change History** section, by adding a new section titled something like **SDP 2024.2 Patch 3** to include the list of JIRA issues fixed since the last SDP release or patch.

When editing, follow examples from earlier releases in the file.  In particular, a section of the file will looks something like this snippet, which includes the content of the `Version` file generated in the previous step:

```

== Change History

=== SDP 2024.2 Patch 3:

Released: Rev. SDP/MultiArch/2024.2/31367 (2025/03/23).

Jobs Fixed since change 31193 up to 31367 (B=Bug, D=Doc, F=Feature):

* https://perforce.atlassian.net/browse/SDP-1218[SDP-1218] (B): mkdirs.sh overwrites existing crontab when adding multiple instances.
* https://perforce.atlassian.net/browse/SDP-1217[SDP-1217] (B): Adjust perforce_bashrc template to avoid setting HISTFILE if it is readonly.

... and so on ...
```

To get the list of JIRA issues, run a JIRA report of SDP issues with the target `Fix versions`, e.g. `2024.2.3`, and ensure all issues listed in the report are considered for inclusion in the release notes.

It is also a good idea to run a report of all open JIRA issues, to include those with an unassigned or different value for `Fix versions`, and consider if they could or should be included in the current release.

We don't need a JIRA issue for every change.  Create JIRA issues to cover the essentials of what changed.  Folks who want the gory details can look at changelists, but folks looking only at the release notes should get a useful summary of what's new in this release or patch, and if appropriate, what to be aware of when upgrading to this release.

Some JIRA issues may be excluded from the public facing release notes if they're not helpful or relevant in helping a customer understand what changed.

Much of JIRA management work should be done during the development, mixed with work from Steps 1 and 2 above. At release time, this check should be a careful double-check to ensure release notes are complete and correct.

Then regenerate the ReleaseNotes.{html,pdf} files, and review them visually to ensure they look good and contain necessary info.

```
make ReleaseNotes.{html,pdf}
```

At this point, there should be a changelist with these files checked out:

```
$ ttyler:/Users/ttyler/re/p4-sdp/doc p4 opened
//p4-sdp/main/Version#152 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.adoc#104 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.html#86 - edit default change (text)
//p4-sdp/main/doc/ReleaseNotes.pdf#64 - edit default change (binary)
```

As a final double-check, bing up the JIRA release and make sure all associated issues are marked as done and completed. If any are not done or not indicated as done, we need to make corrective actions. Actions may include:

* Updating JIRA issues status to match reality, e.g. closing an issue if the code has already been delivered and tested.
* Updating reality to match JIRA reported issue status.
* Deferring: change the `Fix versions` of any features that are to be deferred until a future release.

Then submit:

```
p4 submit -d "Updated Version and Release Notes."
```

**Output**: The Version file and Release Notes are updated for the release.

### Step 16: Copy Up

Push the files to main.

Change to the `dev` branch directory in your workspace.  Run `mdcu.sh` (Merge Down/Copy Up) script, which takes no arguments and is not interactive.  That script creates a pending changelist with the Copy Up main.

Massage the generated pending changelist description if desired.

**WARNING FOR PARTIAL RELEASES**: If this is to be a partial release, revert any files that are not ready to be released from the main branch. Also be sure the reverted files correlate to any jobs removed for a partial release.

Next, submit the generated pending changelist manually.  Proceed _immediately_ to the next step.

**Output**: This is the end of the "Merge Down, Copy Up" process, essential to any release process.  At this point, the latest released files and release notes are live and easily viewable on the web site.

### Step 17: Package

Package the release by creating the tarball.

Starting from the root of your SDP workspace, do:

```
cd tools
bash package_downloads.sh
```

After running this script, run `bash package_downloads.sh`.  (For Mac OSX users:  Calling `bash` explicitly is required only on Mac OSX, and assumes you have installed a modern bash somewhere ahead of `/bin` in your PATH, e.g. with `homebrew`. This is because the package script requires bash 4.x+, and Mac restricts bash to 3.x.)

The `package_downloads.sh` script puts tar and zip files into the downloads folder, and removes older versions.

Review and submit the generated changelist manually.  In the output of the script, you should see a message illustrating the exact command you need, something like:

```
To complete packaging, run:
	p4 -s submit -c 31370
```

**Output**: After this step, the new SDP tarball is published and available for downloading, thus enabling upgrades to the new SDP version. This completes the public aspects of the release process.

### Step 18: Release in JIRA

Set the date for the release in JIRA, and then execute the Release action.

**Output**: Information in JIRA is correct. This completes the internal tracking of the release process.
