--- name: install-sdp-test-suite-run description: Run (or re-run) the SDP Installer Test Suite's full regression across the AWS platform fleet -- exercises install_sdp.sh, mkdirs.sh, and related scripts across every qualified Linux platform. Also covers manual real install-then-upgrade testing (sdp_upgrade.sh), which isn't yet automated. Use whenever asked to run/re-run install_sdp.sh regression tests, or to test an SDP upgrade. --- # SDP Installer Test Suite Runner Runs the full `install_sdp.sh` regression suite (the `r`/`uat` aliases, driven by `cli_tests.cfg` and `test/bsw/run_cli_tests.sh`) across the AWS fleet of platform test machines. See `InstallSDP-TestServerSetup.md` (in the `//test-install_sdp` stream root) for the full background on this fleet and how a machine gets added to it; this skill just codifies running the existing fleet's regression suite so it doesn't have to be re-derived each session. **Safety**: this test suite calls `DANGER_CLEAN.sh` repeatedly and is destructive by design. Only ever run it against the dedicated fleet machines below -- never against a real customer-facing or production machine. ## Two fleets exist -- only one is correct to use There are TWO sets of AWS test machines, left over from the Classic-to-Streams depot migration (see SDP-1170). **Only the new fleet's DVCS remotes point at the real Stream depots** (`//test-install_sdp/dev/...` and `//p4-sdp/dev_rebrand/...`); the old fleet's remotes point at stale personal/Classic guest paths (e.g. `//guest/tom_tyler/sw/main/install_sdp/dev/bin/...`) that stopped being updated long ago. Running against the old fleet silently tests stale code -- `p4 fetch` reports "No changes to fetch" and everything appears to pass, but nothing you just changed was actually exercised. This bit us on 2026-08-31: a full-suite run looked clean (574/0) but had actually run against code from hundreds of changelists earlier. **How to tell them apart, in order of reliability:** 1. Wrapper script name: `ssh-test-install_sdp-*.sh` = new/correct fleet. `ssh-install_sdp-test*.sh` (note the word order swap) = old/Classic fleet. Easy to misread -- check carefully. 2. Authoritative check: `cd` into `/opt/perforce/dev/bin` and `/opt/perforce/dev/sdp` and run `p4 remote -o origin`; the `DepotMap` should read `//test-install_sdp/dev/... //test-install_sdp/dev/...` and `//p4-sdp/dev_rebrand/... //p4-sdp/dev_rebrand/...` (mapping the stream to itself, 1:1). Anything mapping to a `//guest/...` path is the old fleet or a stale clone -- don't trust it. Hostname is **not** a reliable tell as of 2026-08-31: both fleets' machines are now named `p4d-test--01` (the new fleet's AWS-default `ip-...` names were fixed via `bootstrap_test_machine.sh -host` / manual `hostnamectl` -- see "Fleet setup hygiene" below -- and happened to reuse the same naming convention as the old fleet). Don't use hostname to distinguish the fleets; always fall back to check #2 above if the wrapper script name is ever ambiguous. If you ever aren't sure which fleet a wrapper script belongs to, check its filename against the pattern above, or just SSH in and check the remote -- don't assume. ## Retargeting the fleet's SDP source stream When a new dev-cycle stream becomes the one to test (e.g. after a Merge Down/Copy Up promotes `dev_rebrand`'s work into `dev`), retarget each fleet machine's `sdp` clone with `swap_source.sh` (ships in `/opt/perforce/dev/bin`, part of the `test-install_sdp` clone -- see its header comment for full mechanics). It parks the current checkout by suffix and either restores a previously-parked checkout for the target stream or does a fresh `p4 clone -r` if none exists yet: ```bash ssh -o StrictHostKeyChecking=no -l -i "sudo bash -c ' source /opt/perforce/dev/bin/dev_env.sh cd /opt/perforce/dev/bin export P4_SWAP_USER=ftp ./swap_source.sh p4-sdp/dev '" ``` Run this once per fleet machine (all 7, in parallel) before the next full-suite run. It only touches `/opt/perforce/dev/sdp` (the `Target=sdp` case) -- the `bin` clone (the test suite itself) stays on `test-install_sdp/dev` throughout; retarget it too (`./swap_source.sh test-install_sdp/dev`) only if the test suite itself has a new stream to track, which is rare. **Known gotcha found 2026-09-02**: after retargeting, `test_preflight.sh` can fail with `The 'p4 status' in /opt/perforce/dev/bin was NOT clean; ... .p4ignore` even though nothing about the retarget touched `bin` at all -- if a fleet machine's `bin` clone predates change 33164 (the `.p4ignore` content fix in `test-install_sdp/dev`), the working file on disk is stale relative to head and shows up as a bogus reconcile candidate. Fix with a one-time `p4 sync -f .p4ignore` in `/opt/perforce/dev/bin` on the affected machine(s) -- confirm first with `p4 diff -f .p4ignore` that the only difference is exactly this, not a real conflict. ## Fleet setup hygiene `bootstrap_test_machine.sh -host ` sets both the machine's hostname AND its timezone to `America/Los_Angeles` (so Public Depot submit timestamps aren't skewed relative to the target P4 Server's Pacific time -- push/fetch timestamps don't get translated across time zones, so mismatched zones produce technically-valid but visually-wrong-looking timestamps). This is optional and easy to forget when bootstrapping a new machine -- the new fleet's boxes sat with default AWS `ip-...` hostnames and (most likely) UTC/default timezones until this was caught and fixed on 2026-08-31. If you bootstrap a new machine, pass `-host p4d-test--01` (or run `hostnamectl set-hostname`/`timedatectl set-timezone America/Los_Angeles` directly) rather than skipping it. ## Prerequisites - SSH access to each fleet machine via its PEM + public IP (see "Fleet" below). The PEM and per-machine SSH wrapper scripts live at `/Users/ttyler/ppn/BattleSchool/aws/pem/us-east-2/ssh-test-install_sdp-*.sh` -- `cat` one to get the exact `ssh -l -i ` invocation for that machine (login user varies: `rocky` for Rocky Linux, `ec2-user` for SLES, `ubuntu` for Ubuntu). Do NOT use the similarly-named `ssh-install_sdp-test*.sh` scripts -- those are the old/Classic fleet; see above. - Each machine already has `/opt/perforce/dev/bin` (the test suite clone) and `/opt/perforce/dev/sdp` (the SDP clone under test) bootstrapped via `bootstrap_test_machine.sh` -- see `InstallSDP-TestServerSetup.md` if a machine needs (re-)bootstrapping. This skill assumes that's already done. ## Fleet (as of 2026-08-31; re-derive from the wrapper scripts if this looks stale) | Wrapper script | Platform | Login user | |--------------------------------------------|-----------|------------| | `ssh-test-install_sdp-r8x86_64.sh` | Rocky 8 | `rocky` | | `ssh-test-install_sdp-r9x86_64.sh` | Rocky 9 | `rocky` | | `ssh-test-install_sdp-r10x86_64.sh` | Rocky 10 | `rocky` | | `ssh-test-install_sdp-sles15x86_64.sh` | SLES 15 | `ec2-user` | | `ssh-test-install_sdp-u22x86_64.sh` | Ubuntu 22 | `ubuntu` | | `ssh-test-install_sdp-u24x86_64.sh` | Ubuntu 24 | `ubuntu` | | `ssh-test-install_sdp-u26aarch64.sh` | Ubuntu 26 | `ubuntu` | No Ubuntu 20 wrapper exists in the new fleet as of this writing (the old fleet had one) -- don't assume it's missing by mistake; just confirm before relying on Ubuntu 20 coverage. Ubuntu 26 has a **known, pre-existing failure** (POSIX `chown` regression, tracked upstream) -- see `InstallSDP-TestServerSetup.md`'s Platforms list. A failure there is not by itself a sign of a new regression; compare against that known issue before treating it as new. (This was noted against the old fleet; re-confirm it still applies to the new fleet's Ubuntu 26 box the first time you use it.) ## Running the full suite on one machine This replicates the interactive `dev` + `uat` aliases (defined in each machine's `~/.bashrc` and `dev_env.sh`) as a single non-interactive command. **Order matters**: `dev_env.sh` must be sourced *before* any `p4` command, or `p4` won't be on `PATH` yet and you'll get `p4: command not found`. ```bash ssh -o StrictHostKeyChecking=no -l -i "sudo bash -c ' set -e source /opt/perforce/dev/bin/dev_env.sh export KEEP_GOING=1 cd /opt/perforce/dev/sdp/Server/Unix/setup && p4 fetch cd /opt/perforce/dev/bin && p4 fetch ./test_preflight.sh && ./r '" > /_run.log 2>&1 ``` Notes on this command: - The two `p4 fetch` calls pull the latest SDP (`//p4-sdp/dev` as of 2026-09-02, see "Retargeting" above for how/when this changes) and latest test suite (`//test-install_sdp/dev`) respectively -- each of `.../sdp` and `.../bin` is its own independent `p4 clone`-managed DVCS repo (see `dev_env.sh`'s comments), so both fetches are needed. - `KEEP_GOING=1` makes the suite run through all test groups even if one fails, instead of aborting at the first failure -- needed to get a full pass/fail picture in one run. - `test_preflight.sh && ./r` mirrors the `uat` alias exactly: don't run the destructive suite if the workspace isn't clean. - Run this via a background Bash task per machine (`run_in_background: true`) -- a full regression run takes a while, and you'll want to run the whole fleet in parallel rather than one machine at a time. ## Running across the whole fleet Launch the command above once per machine (substituting that machine's user/pem/ip and a distinct log file), all as separate background Bash calls in parallel. Wait for each task-completion notification rather than polling. ## Analyzing results Each log ends with: ``` Summary: PASS Test 1 ... PASS Test 2 ... ... Scan for unreported errors: ``` ```bash grep -c '^PASS Test' # pass count grep -c '^FAIL Test' # fail count grep -n '^FAIL Test' # which ones, to investigate ``` Anything printed under "Scan for unreported errors:" is also worth reviewing -- it's the `r` script's own heuristic catch-all for error-looking lines that didn't get wrapped in a `PASS`/`FAIL` marker. To see the list of individual test groups (e.g. to re-run just one after a fix), run `./r ls` in place of `./r` in the command above. ## Manual upgrade testing (not yet part of the automated suite) `./r`/`cli_tests.cfg` exercise `install_sdp.sh` fresh installs across platforms, but do **not** exercise `sdp_upgrade.sh` -- there is currently no automated upgrade-path coverage anywhere in this fleet's regression suite. This gap let a real bug ship in SDP 2026.1 GA (SDP-1394: `install_sdp.sh` and friends hardcoded the planned-for-2026.2 `/opt/perforce/p4-sdp` path instead of 2026.1's `/opt/perforce/helix-sdp`) and a second one found only by testing the fix itself (SDP-1395: `sdp_upgrade.sh` reported a false "FAILED" on an otherwise-successful package-structure upgrade). Both were caught the day after GA, not before, specifically because nothing exercised a real install-then-upgrade cycle. Until this is automated, do this manually whenever a release touches `install_sdp.sh`, `sdp_upgrade.sh`, `mkdirs.sh`, `configure_sample_depot_for_sdp.sh`, or anything else in the package-structure install/upgrade path -- ideally as part of Step 4 (pre-release verification), not just post-release smoke testing, so a regression like SDP-1394/1395 is caught before shipping rather than after: 1. **Reset the machine to a clean slate.** From the fleet machine (either fleet -- this doesn't depend on the DVCS source-tracking distinction above, since you're testing the *shipped tarball*, not the fleet's own dev clone): ```bash sudo bash /opt/perforce/dev/bin/DANGER_CLEAN.sh -y ``` 2. **Install a real older release**, matching what an actual customer upgrading today would have. Fetch the target release's `install_sdp.sh` from wherever it actually shipped from (the frozen Classic depot for anything through 2025.2 Patch 1, e.g. `p4 print -q //guest/perforce_software/sdp/main/Server/Unix/setup/install_sdp.sh@`; `//p4-sdp/r` release streams for anything 2026.1+), copy it to the machine, and run it in `-demo` mode for speed (storage layout isn't the point of this test): ```bash ./install_sdp.sh -C > sdp_install.cfg ./install_sdp.sh -c sdp_install.cfg -init -y -demo ``` Confirm the resulting structure matches that release's actual convention (e.g. `/opt/perforce/helix-sdp` for anything through 2026.1) before proceeding -- don't assume; verify with `ls /opt/perforce/`. 3. **Upgrade to the release candidate**, using the same download path a real customer would -- the actual published tarball URL for a released version (`https://workshop.perforce.com/download/p4-sdp/main/downloads/sdp.Unix.tgz` once published), or a freshly-built local tarball copied to the machine directly if testing an unpublished release candidate before Step 20: ```bash cd /opt/perforce/helix-sdp/downloads # or /opt/perforce/p4-sdp/downloads, per the release curl -sL -O # or scp a locally-built tarball here instead cd /opt/perforce/helix-sdp mkdir -p backup && mv sdp backup/sdp.old.$(date +'%Y-%m-%d-%H%M%S') tar -xzf downloads/sdp.Unix.tgz && chown -R root:root sdp cd sdp/Server/Unix/p4/common/sdp_upgrade ./sdp_upgrade.sh -y ``` 4. **Verify for real**, not just the script's own "SUCCESS" line: check `/p4/sdp/Version` matches the target release, confirm the directory structure landed correctly (no stray directory named for a not-yet-released naming convention), and if a `p4d` instance is actually running, confirm it still answers (`p4 -p ssl:1999 info` or equivalent). **Old-fleet caveat**: machines on the old (`ssh-install_sdp-test*.sh`) fleet may be in whatever ad hoc state a prior test left them in -- e.g. one Rocky 8 box was found to be a broker-only fixture with no real `p4d` instance at all. Still a valid *starting point* for this procedure (just run `DANGER_CLEAN.sh` first per step 1 above), but don't assume any given old-fleet machine already has a clean or representative state without checking.