---
name: install-sdp-test-suite-run
description: Run (or re-run) the SDP Installer Test Suite's full regression across the AWS platform fleet -- exercises install_sdp.sh, mkdirs.sh, and related scripts across every qualified Linux platform. Also covers manual real install-then-upgrade testing (sdp_upgrade.sh), which isn't yet automated. Use whenever asked to run/re-run install_sdp.sh regression tests, or to test an SDP upgrade.
---

# SDP Installer Test Suite Runner

Runs the full `install_sdp.sh` regression suite (the `r`/`uat` aliases, driven by
`cli_tests.cfg` and `test/bsw/run_cli_tests.sh`) across the AWS fleet of platform test
machines. See `InstallSDP-TestServerSetup.md` (in the `//test-install_sdp` stream root) for
the full background on this fleet and how a machine gets added to it; this skill just
codifies running the existing fleet's regression suite so it doesn't have to be re-derived
each session.

**Safety**: this test suite calls `DANGER_CLEAN.sh` repeatedly and is destructive by
design. Only ever run it against the dedicated fleet machines below -- never against a
real customer-facing or production machine.

## Two fleets exist -- only one is correct to use

There are TWO sets of AWS test machines, left over from the Classic-to-Streams depot
migration (see SDP-1170). **Only the new fleet's DVCS remotes point at the real Stream
depots** (`//test-install_sdp/dev/...` and `//p4-sdp/dev_rebrand/...`); the old fleet's
remotes point at stale personal/Classic guest paths (e.g.
`//guest/tom_tyler/sw/main/install_sdp/dev/bin/...`) that stopped being updated long ago.
Running against the old fleet silently tests stale code -- `p4 fetch` reports "No changes
to fetch" and everything appears to pass, but nothing you just changed was actually
exercised. This bit us on 2026-08-31: a full-suite run looked clean (574/0) but had
actually run against code from hundreds of changelists earlier.

**How to tell them apart, in order of reliability:**
1. Wrapper script name: `ssh-test-install_sdp-*.sh` = new/correct fleet.
   `ssh-install_sdp-test*.sh` (note the word order swap) = old/Classic fleet. Easy to
   misread -- check carefully.
2. Authoritative check: `cd` into `/opt/perforce/dev/bin` and `/opt/perforce/dev/sdp` and
   run `p4 remote -o origin`; the `DepotMap` should read
   `//test-install_sdp/dev/... //test-install_sdp/dev/...` and
   `//p4-sdp/dev_rebrand/... //p4-sdp/dev_rebrand/...` (mapping the stream to itself,
   1:1). Anything mapping to a `//guest/...` path is the old fleet or a stale clone --
   don't trust it.

Hostname is **not** a reliable tell as of 2026-08-31: both fleets' machines are now named
`p4d-test-<tag>-01` (the new fleet's AWS-default `ip-...` names were fixed via
`bootstrap_test_machine.sh -host` / manual `hostnamectl` -- see "Fleet setup hygiene"
below -- and happened to reuse the same naming convention as the old fleet). Don't use
hostname to distinguish the fleets; always fall back to check #2 above if the wrapper
script name is ever ambiguous.

If you ever aren't sure which fleet a wrapper script belongs to, check its filename
against the pattern above, or just SSH in and check the remote -- don't assume.

## Retargeting the fleet's SDP source stream

When a new dev-cycle stream becomes the one to test (e.g. after a Merge Down/Copy Up
promotes `dev_rebrand`'s work into `dev`), retarget each fleet machine's `sdp` clone with
`swap_source.sh` (ships in `/opt/perforce/dev/bin`, part of the `test-install_sdp` clone --
see its header comment for full mechanics). It parks the current checkout by suffix and
either restores a previously-parked checkout for the target stream or does a fresh
`p4 clone -r` if none exists yet:

```bash
ssh -o StrictHostKeyChecking=no -l <user> -i <pem> <ip> "sudo bash -c '
source /opt/perforce/dev/bin/dev_env.sh
cd /opt/perforce/dev/bin
export P4_SWAP_USER=ftp
./swap_source.sh p4-sdp/dev
'"
```

Run this once per fleet machine (all 7, in parallel) before the next full-suite run. It
only touches `/opt/perforce/dev/sdp` (the `Target=sdp` case) -- the `bin` clone (the test
suite itself) stays on `test-install_sdp/dev` throughout; retarget it too
(`./swap_source.sh test-install_sdp/dev`) only if the test suite itself has a new stream to
track, which is rare.

**Known gotcha found 2026-09-02**: after retargeting, `test_preflight.sh` can fail with `The
'p4 status' in /opt/perforce/dev/bin was NOT clean; ... .p4ignore` even though nothing about
the retarget touched `bin` at all -- if a fleet machine's `bin` clone predates change 33164
(the `.p4ignore` content fix in `test-install_sdp/dev`), the working file on disk is stale
relative to head and shows up as a bogus reconcile candidate. Fix with a one-time
`p4 sync -f .p4ignore` in `/opt/perforce/dev/bin` on the affected machine(s) -- confirm first
with `p4 diff -f .p4ignore` that the only difference is exactly this, not a real conflict.

## Fleet setup hygiene

`bootstrap_test_machine.sh -host <hostname>` sets both the machine's hostname AND its
timezone to `America/Los_Angeles` (so Public Depot submit timestamps aren't skewed
relative to the target P4 Server's Pacific time -- push/fetch timestamps don't get
translated across time zones, so mismatched zones produce technically-valid but
visually-wrong-looking timestamps). This is optional and easy to forget when bootstrapping
a new machine -- the new fleet's boxes sat with default AWS `ip-...` hostnames and
(most likely) UTC/default timezones until this was caught and fixed on 2026-08-31. If you
bootstrap a new machine, pass `-host p4d-test-<tag>-01` (or run `hostnamectl
set-hostname`/`timedatectl set-timezone America/Los_Angeles` directly) rather than
skipping it.

## Prerequisites

- SSH access to each fleet machine via its PEM + public IP (see "Fleet" below). The PEM
  and per-machine SSH wrapper scripts live at
  `/Users/ttyler/ppn/BattleSchool/aws/pem/us-east-2/ssh-test-install_sdp-*.sh` -- `cat` one
  to get the exact `ssh -l <user> -i <pem> <ip>` invocation for that machine (login user
  varies: `rocky` for Rocky Linux, `ec2-user` for SLES, `ubuntu` for Ubuntu). Do NOT use
  the similarly-named `ssh-install_sdp-test*.sh` scripts -- those are the old/Classic
  fleet; see above.
- Each machine already has `/opt/perforce/dev/bin` (the test suite clone) and
  `/opt/perforce/dev/sdp` (the SDP clone under test) bootstrapped via
  `bootstrap_test_machine.sh` -- see `InstallSDP-TestServerSetup.md` if a machine needs
  (re-)bootstrapping. This skill assumes that's already done.

## Fleet (as of 2026-08-31; re-derive from the wrapper scripts if this looks stale)

| Wrapper script                            | Platform  | Login user |
|--------------------------------------------|-----------|------------|
| `ssh-test-install_sdp-r8x86_64.sh`        | Rocky 8   | `rocky`    |
| `ssh-test-install_sdp-r9x86_64.sh`        | Rocky 9   | `rocky`    |
| `ssh-test-install_sdp-r10x86_64.sh`       | Rocky 10  | `rocky`    |
| `ssh-test-install_sdp-sles15x86_64.sh`    | SLES 15   | `ec2-user` |
| `ssh-test-install_sdp-u22x86_64.sh`       | Ubuntu 22 | `ubuntu`   |
| `ssh-test-install_sdp-u24x86_64.sh`       | Ubuntu 24 | `ubuntu`   |
| `ssh-test-install_sdp-u26aarch64.sh`      | Ubuntu 26 | `ubuntu`   |

No Ubuntu 20 wrapper exists in the new fleet as of this writing (the old fleet had one) --
don't assume it's missing by mistake; just confirm before relying on Ubuntu 20 coverage.

Ubuntu 26 has a **known, pre-existing failure** (POSIX `chown` regression, tracked
upstream) -- see `InstallSDP-TestServerSetup.md`'s Platforms list. A failure there is not
by itself a sign of a new regression; compare against that known issue before treating it
as new. (This was noted against the old fleet; re-confirm it still applies to the new
fleet's Ubuntu 26 box the first time you use it.)

## Running the full suite on one machine

This replicates the interactive `dev` + `uat` aliases (defined in each machine's
`~/.bashrc` and `dev_env.sh`) as a single non-interactive command. **Order matters**:
`dev_env.sh` must be sourced *before* any `p4` command, or `p4` won't be on `PATH` yet and
you'll get `p4: command not found`.

```bash
ssh -o StrictHostKeyChecking=no -l <user> -i <pem> <ip> "sudo bash -c '
set -e
source /opt/perforce/dev/bin/dev_env.sh
export KEEP_GOING=1
cd /opt/perforce/dev/sdp/Server/Unix/setup && p4 fetch
cd /opt/perforce/dev/bin && p4 fetch
./test_preflight.sh && ./r
'" > <scratchpad>/<platform>_run.log 2>&1
```

Notes on this command:
- The two `p4 fetch` calls pull the latest SDP (`//p4-sdp/dev` as of 2026-09-02, see
  "Retargeting" above for how/when this changes) and latest test suite
  (`//test-install_sdp/dev`) respectively -- each of `.../sdp` and `.../bin` is its own
  independent `p4 clone`-managed DVCS repo (see `dev_env.sh`'s comments), so both fetches
  are needed.
- `KEEP_GOING=1` makes the suite run through all test groups even if one fails, instead of
  aborting at the first failure -- needed to get a full pass/fail picture in one run.
- `test_preflight.sh && ./r` mirrors the `uat` alias exactly: don't run the destructive
  suite if the workspace isn't clean.
- Run this via a background Bash task per machine (`run_in_background: true`) -- a full
  regression run takes a while, and you'll want to run the whole fleet in parallel rather
  than one machine at a time.

## Running across the whole fleet

Launch the command above once per machine (substituting that machine's user/pem/ip and a
distinct log file), all as separate background Bash calls in parallel. Wait for each
task-completion notification rather than polling.

## Analyzing results

Each log ends with:

```
Summary:
PASS Test 1 ...
PASS Test 2 ...
...
Scan for unreported errors:
```

```bash
grep -c '^PASS Test' <log>   # pass count
grep -c '^FAIL Test' <log>   # fail count
grep -n '^FAIL Test' <log>   # which ones, to investigate
```

Anything printed under "Scan for unreported errors:" is also worth reviewing -- it's the
`r` script's own heuristic catch-all for error-looking lines that didn't get wrapped in a
`PASS`/`FAIL` marker.

To see the list of individual test groups (e.g. to re-run just one after a fix), run
`./r ls` in place of `./r` in the command above.

## Manual upgrade testing (not yet part of the automated suite)

`./r`/`cli_tests.cfg` exercise `install_sdp.sh` fresh installs across platforms, but do
**not** exercise `sdp_upgrade.sh` -- there is currently no automated upgrade-path coverage
anywhere in this fleet's regression suite. This gap let a real bug ship in SDP 2026.1 GA
(SDP-1394: `install_sdp.sh` and friends hardcoded the planned-for-2026.2
`/opt/perforce/p4-sdp` path instead of 2026.1's `/opt/perforce/helix-sdp`) and a second one
found only by testing the fix itself (SDP-1395: `sdp_upgrade.sh` reported a false "FAILED"
on an otherwise-successful package-structure upgrade). Both were caught the day after GA,
not before, specifically because nothing exercised a real install-then-upgrade cycle.

Until this is automated, do this manually whenever a release touches `install_sdp.sh`,
`sdp_upgrade.sh`, `mkdirs.sh`, `configure_sample_depot_for_sdp.sh`, or anything else in the
package-structure install/upgrade path -- ideally as part of Step 4 (pre-release
verification), not just post-release smoke testing, so a regression like SDP-1394/1395 is
caught before shipping rather than after:

1. **Reset the machine to a clean slate.** From the fleet machine (either fleet -- this
   doesn't depend on the DVCS source-tracking distinction above, since you're testing the
   *shipped tarball*, not the fleet's own dev clone):
   ```bash
   sudo bash /opt/perforce/dev/bin/DANGER_CLEAN.sh -y
   ```
2. **Install a real older release**, matching what an actual customer upgrading today would
   have. Fetch the target release's `install_sdp.sh` from wherever it actually shipped from
   (the frozen Classic depot for anything through 2025.2 Patch 1, e.g.
   `p4 print -q //guest/perforce_software/sdp/main/Server/Unix/setup/install_sdp.sh@<CL>`;
   `//p4-sdp/r<NN.N>` release streams for anything 2026.1+), copy it to the machine, and run
   it in `-demo` mode for speed (storage layout isn't the point of this test):
   ```bash
   ./install_sdp.sh -C > sdp_install.cfg
   ./install_sdp.sh -c sdp_install.cfg -init -y -demo
   ```
   Confirm the resulting structure matches that release's actual convention (e.g.
   `/opt/perforce/helix-sdp` for anything through 2026.1) before proceeding -- don't assume;
   verify with `ls /opt/perforce/`.
3. **Upgrade to the release candidate**, using the same download path a real customer would
   -- the actual published tarball URL for a released version
   (`https://workshop.perforce.com/download/p4-sdp/main/downloads/sdp.Unix.tgz` once
   published), or a freshly-built local tarball copied to the machine directly if testing an
   unpublished release candidate before Step 20:
   ```bash
   cd /opt/perforce/helix-sdp/downloads   # or /opt/perforce/p4-sdp/downloads, per the release
   curl -sL -O <tarball URL>              # or scp a locally-built tarball here instead
   cd /opt/perforce/helix-sdp
   mkdir -p backup && mv sdp backup/sdp.old.$(date +'%Y-%m-%d-%H%M%S')
   tar -xzf downloads/sdp.Unix.tgz && chown -R root:root sdp
   cd sdp/Server/Unix/p4/common/sdp_upgrade
   ./sdp_upgrade.sh -y
   ```
4. **Verify for real**, not just the script's own "SUCCESS" line: check `/p4/sdp/Version`
   matches the target release, confirm the directory structure landed correctly (no stray
   directory named for a not-yet-released naming convention), and if a `p4d` instance is
   actually running, confirm it still answers (`p4 -p ssl:1999 info` or equivalent).

**Old-fleet caveat**: machines on the old (`ssh-install_sdp-test*.sh`) fleet may be in
whatever ad hoc state a prior test left them in -- e.g. one Rocky 8 box was found to be a
broker-only fixture with no real `p4d` instance at all. Still a valid *starting point* for
this procedure (just run `DANGER_CLEAN.sh` first per step 1 above), but don't assume any
given old-fleet machine already has a clean or representative state without checking.