# Session Log -- 2026-09-08

## Summary

**Release day -- SDP 2026.1 GA shipped.** Started by picking up the three Friday-night
carry-forwards (Jenkins-on-`main`, Gen6 post-upgrade `lab 0` retest, Gen7 `main` clean run),
which turned into a full morning of root-cause investigation and real bug fixes (Gen6
lab-reset mechanism traced to a BattleSchool-owned stale-path bug, not an SDP regression;
Gen7 QA lab traced to two genuine, previously-undiscovered `install_sdp.sh` bugs -- a
DVCS-clone `.p4config` rsync-exclude gap and a dangling-symlink `LogLink` bug -- both fixed
and verified via a full clean `qa_build` run plus a 616/616 Installer Test Suite run).

With testing complete, executed the full 22-step (later 23-step) SDP Release Process
end-to-end, one step at a time with the user confirming after each: cut and published
`//p4-sdp/r26.1.0`, closed out the Classic-to-Streams migration (SDP-1170/1385, including a
Swarm project/branch-naming cleanup), fixed a JIRA Resolution-field release-gating gap, and
created a new `sdp-release-process-run` skill to codify the process. Post-release, ran a
real fresh-install smoke test (found and filed SDP-1390/1391, fixed a doc inconsistency),
the user dogfooded the real upgrade across the entire PPN and Public Depot fleets (no
significant issues), submitted the long-staged Classic-depot "We've Moved" banners, added a
missing "announce the release" process step, did the first-ever `//test-install_sdp`
dev->main population, and tore down the AWS labs created during this week's testing. SDP
2026.1 is now fully shipped, released in JIRA, and announced in `#p4-bsw`.

## Jenkins `//p4-sdp/main` build: still unresolved, no word from Robert

User manually clicked "Build Now" overnight (build #6, no config changes since #4) and hit
the identical `Error: can only create exec sessions on running containers: container state
improper` signature reported Friday night. No response yet from Robert (Slack sent Friday
evening, 2026-09-04; today is the first business day after -- US had Monday off for Labor
Day, Robert's UK Monday wasn't a holiday, but no reply has come through yet). Per Friday's
framing, this stays "generally critical path" but the user may still choose to ship without
a green Jenkins run on `main` if confident nothing Jenkins-specific has regressed since it
last worked on `dev`. Decided to hold off on this and pivot to the Gen6 lab-reset
investigation instead. No new diagnostic work done on the Jenkins/podman side today.

## Gen6 lab-reset investigation: confirmed real, root cause found (not a release blocker)

Picked up exactly where Friday left off: does the SDP+P4D upgrade to 2026.1 break the
Gen6 lab's `lab 0` reset mechanism, or was the first lab's failure a fluke? Used the
second lab from Friday's upgrade exercise, `bsw-sdp-upgrade-test2` (us-east-2; confirmed
untouched since Friday -- 5-host fleet + `bsw-lab-ui` jump box, topology and state exactly
as left: SDP+P4D upgrade to 2026.1 completed and verified clean, `lab 0` never yet re-run
post-upgrade on this lab).

**Reproduced the failure, twice.** `lab 0` itself reports success ("Lab 0 is ready", ~20-24
seconds) both times, but the fleet doesn't actually come up: `p4 servers -J` from the
commit side gets `Connect to server failed... TCP connect to perforce:1666 failed:
Connection refused`; on `bos-helix-01` directly, `p4d_1` and `p4broker_1` are both
`systemctl` `failed` (`Main process exited, code=killed, status=9/KILL`), and `/p4/1/bin/`
is completely empty. No OOM event in `dmesg`/`journalctl` -- ruling that out as the kill
cause. The documented Gen6 "run `lab 0` twice" idiosyncrasy (stuck-at-epoch replicas) does
**not** apply here and does not self-heal the problem on a second run.

**Root cause, from `/usr/local/p4battleschool/init/logs/lab_engine.init.0.log` and
`/tmp/reset_sdp.log` on `bos-helix-01`:**

1. Gen6's reset mechanism is fundamentally different from Gen7's. Gen7 uses the current,
   split, customer-facing tools (`DANGER_CLEAN.sh` from `//test-install_sdp` for teardown,
   `install_sdp.sh` from SDP for rebuild). Gen6 instead uses `reset_sdp.sh`, a combined
   tear-down-and-rebuild script that is part of a separate "Helix Installer" codebase
   **maintained by Battle School Workshop itself**, not part of the SDP depot -- this
   predates the install_sdp.sh/DANGER_CLEAN.sh split (which exists specifically so
   customers never touch anything as dangerous as `reset_sdp.sh`).
2. `reset_sdp.sh` always refreshes/re-clones the local SDP tree from its configured branch
   before resetting -- so every `lab 0` run picks up whatever is currently on that branch.
3. This SDP 2026.1 cycle renamed the top-level `helix_binaries/` directory to
   `p4_binaries/` (alongside the `get_helix_binaries.sh` -> `get_p4_binaries.sh` rename).
4. `reset_sdp.sh`'s "modern SDP" binary-staging step (used when the checked-in `p4d` is a
   wrapper script rather than a real binary) still does
   `cp -f -p $BinDir/p4* $SDPHome/helix_binaries/.` -- and `$SDPHome/helix_binaries` no
   longer exists, so the `cp` fails outright (`not a directory`).
5. `mkdirs.sh` (current, unmodified, correctly using the new `p4_binaries` convention)
   then can't find a real binary at the path it expects, aborts with `EXITCODE: 7`, and
   every downstream step fails (`/p4/1/bin/p4_1: No such file or directory` cascading
   through topology config, replica creation, sample-depot load). The two services that
   *were* running pre-reset get torn down as part of the reset's teardown step and never
   restart.
6. Confirmed this isn't a one-off: manually staging real binaries into the new
   `p4_binaries` path let `mkdirs.sh`'s specific failure point move forward, but a second
   `lab 0` run wiped that manual staging again (via `reset_sdp.sh`'s own re-clone) and hit
   the exact same failure at the exact same step.

**Confirmed NOT a release blocker for SDP 2026.1.** Customers never run `reset_sdp.sh` --
that's precisely why it was split into `install_sdp.sh` (safe) and `DANGER_CLEAN.sh`
(dangerous) in the first place. Checked all three real customer/Gen7-path scripts
(`install_sdp.sh`, `DANGER_CLEAN.sh`, `mkdirs.sh`) for any stale `helix_binaries`
reference -- none exist; all three are internally consistent on `p4_binaries`. A real
production Gen6 BSW training lab would also reset against a fixed, already-released SDP
tarball, not a live `main`/`dev` clone -- this only surfaced because tonight's lab was
deliberately tracking bleeding-edge `main` for the pre-release upgrade rehearsal.

**Found and fixed a parallel instance of the same stale-reference class**, while confirming
the customer path was clean: SDP's own `test/bsw/push_bins.sh` (a BattleSchool convenience
script that *is* part of the SDP depot, under `test/bsw/`) already had its Gen7
(`p4c-bos-01`) branch correctly updated to `p4_binaries`, but its Gen6 (`bos-helix-01`)
branch still hardcoded `helix_binaries` -- the same migration, just never completed for
Gen6. Fixed in `//p4-sdp/dev` change 33535, Copy Up (via `p4 copy -S //p4-sdp/dev`, scoped
to just this one file to avoid pulling in an unrelated pending `ReleaseNotes.adoc` diff)
into `//p4-sdp/main` change 33536.

**Filed BSW-114** for the actual `reset_sdp.sh` fix, which is BattleSchool-owned code, not
ours to fix directly: full root-cause writeup, reproduction evidence, explicit "not a
release blocker" reasoning, and a pointer to the SDP-side parallel fix above for reference.
`Subsystem: bsw`, `Severity: B`, `OwnedBy: ttyler`, `CommitRelease: Gen6`.

**Follow-up tweak, per the user:** the BSW jobspec's `Description` field's first line
doubles as the job's title in `p4 jobs` reports/listings -- BSW-114's initial description
ran straight into the detailed explanation instead of leading with a standalone summary.
Fixed by re-submitting the job with `Gen6 lab resets fail after upgrade to SDP 2026.1+.` as
its own first line. Also updated the `bsw-ppn-jobs` skill to document this rule explicitly
(it previously said "first 31 chars are significant," which was both inaccurate and not
actionable) -- submitted as `//BattleSchool/Gen7` change 2764716.

## Two more Gen6 BSW jobs filed (workarounds already in routine use for both)

While in the area, filed two more known Gen6 quirks that already have documented
workarounds (both routinely applied via the `bsw-gen6-lab-create` skill) but had never
been tracked as actual BSW jobs:

- **BSW-115**: Gen6 lab hosts ship with stale/expired baked-in licenses in their AMI
  snapshot -- a lab can't be started/used until this is manually corrected (the
  `bsw_rsync.sh`-via-`bos-helix-01` license-refresh step). Confirmed still present on a lab
  created as recently as 2026-09-04, so this isn't stale AMI history, current Gen6 AMI
  builds still need it.
- **BSW-116**: the very first `lab <N>` run on a brand-new Gen6 lab sometimes leaves some
  replicas permanently stuck at epoch time and never catches up; running `lab <N>` a second
  time reliably fixes it, root cause not investigated. Both jobs: `Severity: C`,
  `OwnedBy: ttyler`, `CommitRelease: Gen6`.

## `ReleaseNotes.adoc` dev->main gap: investigated and fixed

Turned out to be small and real, not the "16-revision gap" it first looked like (dev's and
main's revision numbers simply don't share a base, since main's copy was branched partway
through dev's history). The actual pending content was a single line: dev's change 33492
(2026-09-04, dropping "at least" from the security=4 wording per the user's explicit
decision) had only partly made it to `main` -- `upgrade.sh` itself was already correct in
`main` via a separate earlier Copy Up (33494), but `doc/ReleaseNotes.adoc` (and its
generated `.html`/`.pdf`) still said "unconditionally raises ... to **at least** 4."

Copied up `doc/ReleaseNotes.adoc` from `dev`, regenerated `ReleaseNotes.{html,pdf}` via
`make` in `doc/` (asciidoctor/asciidoctor-pdf), confirmed zero remaining "at least 4" hits
anywhere in `main` (`Unsupported/doc/` was already clean). Submitted as change 33537.

## Gen7 `main` QA lab: re-ran `lab qa_build` -- fails deterministically, root cause narrowed but not fixed

Before re-running, confirmed nothing else had touched this lab since Friday: request queue
empty, `p4bsw.service` active. Copied `trigger_qa_build.sh` up (had to `sudo rm` a stale
read-only copy from Friday first) and triggered a fresh run. (One process hiccup: my own
polling wrapper backgrounded itself in a way that made the tool report "done" prematurely
at the "Request submitted..." line -- the actual remote reset kept running fine per direct
log inspection; re-polled directly against the log instead.)

**Result: fails, reproducibly (2/2 attempts today), at the same point as Friday** --
`test/bsw/run_cli_tests.sh -g BSW_Commit` Test 6, inside `install_sdp.sh` ->
`configure_sample_depot_for_sdp.sh`. Chased down considerably further than Friday's
guess ("leftover partial structure"), which turned out to be wrong:

- The `/opt/perforce/.dev/sdp` DVCS clone is fine -- confirmed correct content
  (`2026.1/33441`) both times, ruling out the stale-clone theory entirely.
- A `mkdirs.1.cfg` diff that looked alarming at first (`DD=/mnt/p4depots` in the sample
  template vs. `DD=mnt/p4depots`, no leading slash, in the generated per-host config) is
  **not a bug** -- `install_sdp.sh` deliberately strips the leading `/` before writing
  these (`# Trim the leading '/' from P4* settings to be compatible with SDP mkdirs.cfg`),
  and `mkdirs.sh` consistently prepends its own `/` everywhere it uses `$DD`/`$DB1`/`$CD`/
  `$LG`. Confirmed `/p4/1`'s resulting symlinks are all correct, absolute, and sane.
- The actual failure: `configure_sample_depot_for_sdp.sh` explicitly verifies p4d
  (`$P4BIN -p "$P4PORT" info -s`, in a proper retry loop) and the broker
  (`$P4BIN -p "$P4BROKERPORT" info -s`, also retried) -- both succeed and print
  "Verified: ... has started." The very next check, `runCmd "$P4BIN -s info -s"
  "Verifying direct connection..."` (no explicit `-p`, so it relies on `$P4BIN`'s own
  default P4PORT resolution rather than the variable already proven to work), fails with
  `Perforce server error: Root directory ... invalid. chdir:
  /opt/perforce/p4-sdp/p4/sdp/.p4root: No such file or directory` followed by a connection
  reset -- an odd, specific error suggesting the connection is resolving to the wrong
  target entirely, not just "not ready yet."
- Manually re-ran the exact same command (`/p4/1/bin/p4_1 -s info -s`, no `-p`) minutes
  later, as `perforce`, from the same working directory the script itself uses -- it
  **succeeded cleanly**, connecting to the live server with no issue at all.

So: this isn't stale content, isn't a real path/structure bug, and doesn't reproduce
on-demand once the fleet has settled -- but it does fail every single time during the
actual automated run, at the exact same line, immediately after the broker starts. Best
working theory: a timing/ordering issue specific to `$P4BIN`'s no-`-p` default P4PORT
resolution (likely P4CONFIG-file-based) racing something written moments earlier in the
same flow, distinct from -- and not caught by -- the retry loops guarding the two checks
immediately before it. Not fixed tonight; this is squarely the item flagged Friday as
critical path to shipping today, so flagging back to the user rather than continuing to
dig blind.

## Note: user is manually testing a third Gen6 lab, `bsw-sdp-upgrade-test3`

User created a new Gen6 lab (`bsw-sdp-upgrade-test3`, jump box public IP `18.218.33.27`)
and is doing manual testing on it directly. Not touched by any automation above --
`bsw-sdp-upgrade-test2` (the one all the `lab 0`/`reset_sdp.sh` investigation above was
done on) is a separate, still-intact lab.

## SDP Guide: Classic-structure sample upgrade now works regardless of depot mount point

User flagged that section 6.3.1.1 ("Sample SDP Upgrade in Classic Structure") hardcoded
`/mnt/p4depots` (the new default). Customers on the older `/hxdepots` default have a habit
of blindly copy/pasting this block, so on release day it would work for zero customers --
but reverting to `/hxdepots` would just shift the same problem to future installs on the
new default. Fixed by detecting the depot mount point at runtime into an `SDPMount`
variable (`SDPMount=$(cd /p4/common && d=$(pwd -P) && echo "${d%/p4/common}")` -- reusing
the exact technique already documented as a manual fallback in this section, now built into
the main procedure instead of being an afterthought). Verified live against two
differently-structured real hosts (a Classic `/hxdepots` lab and an OS-package-structure
host) -- correctly resolved the base mount in both cases, confirming the technique is
mount-name-agnostic. Removed the now-redundant "what if there is no /mnt/p4depots" callout;
added a one-line disclaimer that the sample presumes a standard, undeviated install
structure. `dev` change 33538, Copy Up to `main` 33539 (HTML regenerated in both; PDF
intentionally not touched, per the new PDF policy below).

## New policy: PDFs regenerated only at release time, not during development

Per the user: PDFs are large, heavy files with almost no incremental value day to day;
regenerating one for every doc change wastes disk space for no benefit. HTML stays cheap to
regenerate and should track its source; PDF generation is deliberately deferred to the
release process (`ReleaseProcessOverview.md` Step 14), where it gets caught up all at once.
Added a "Documentation Builds" section to `SDP_DeveloperGuide.adoc` stating this plainly,
plus a cross-referencing note at Step 14 itself. No skills currently discuss doc
regeneration, so none needed updating -- worth remembering if one gets written later.
`dev` change 33540, Copy Up to `main` 33541.

## Upgrade error message clarity: Secure By Default flags, by server role

User's own procedure during a global topology upgrade: always use `-Os` on non-commit
servers, and only choose between `-Os` alone vs. `-Os` + `-opt-out-of-security-update`
together on the commit server (the latter is commit-server-only, per an existing usage
check). The prior wording didn't convey this -- it suggested `-opt-out-of-security-update`
unconditionally regardless of server role, and the abort error phrased `-Os` and
`-opt-out-of-security-update` as alternatives ("use X, or Y"), when the latter actually
requires the former alongside it to have any effect (already correctly described in the
`-man` text, just not in this runtime message). Both the warning (always shown when
crossing the SBD threshold) and the abort error (shown when `security<3` and `-Os` wasn't
given) now branch on `SERVERID == P4MASTER_ID` and describe only the combination valid for
that server's role. `dev` change 33542, Copy Up to `main` in 33544.

## Cruft removal: deprecated-files cleanup now handles empty directories, plus helix_binaries entries

User's call: safe-over-clean remains the default for upgrades (never touch unlisted/customer
files), but explicitly-deprecated SDP-owned cruft is worth cleaning up -- and the existing
mechanism could only remove files, not the (possibly now-empty) directories that held them.
`remove_deprecated_files()` in `sdp_upgrade.sh` now does three passes: classify each listed
entry as file/directory/not-present, remove all listed files first, then `rmdir` (never
`rm -rf`) any listed directories, so a directory is only removed if the file removals
actually emptied it. A directory that can't be removed (e.g. still holds customer files) is
a warning, not an error -- an expected, acceptable outcome, unlike a real file-removal
failure. Updated `deprecated_files.txt`'s header comments to document directory-entry
support, and added entries for the `helix_binaries` directory (renamed to `p4_binaries` this
cycle): the known binary files plus the directory itself. `dev` change 33543, Copy Up to
`main` 33544.

User then asked to verify against their own manual-install lab (`bsw-sdp-upgrade-test3`,
Classic structure) -- checked `/p4/sdp/helix_binaries/` on all 5 fleet hosts (identical
contents everywhere): `get_helix_binaries.sh`, `p4`, `p4broker`, `p4d`, `p4p`. No `.json`
files there (those exist only in the new `p4_binaries` directory, correctly, both current
and staged-for-upgrade). This check paid off: **`get_helix_binaries.sh` itself was missing**
from the deprecated-files entries added above -- without it, the directory would never
actually empty out and the new `rmdir` logic would silently no-op forever. Fixed and
verified the 5 listed entries now exactly match the 5 real files on the live host. `dev`
change 33545, Copy Up to `main` 33546.

## Merge Down flow check: r26.1.0.BETA -> main -> dev, confirmed no-op

Per the user's request, verified the full chain before considering next steps toward
release. `r26.1.0.BETA` is a locked release stream (`changeFlowsToParent`/
`changeFlowsFromParent` both `false` by design -- release streams don't participate in
normal integration flow at all). A direct content comparison (`p4 diff2` between the two
depot trees, bypassing integration history) shows every differing file has main **ahead**
of BETA (higher revision number on main's side) -- Version, generated docs/man
pages/tarballs differ as expected (all `isolate`d per stream by design), and every real
source-code difference (`upgrade.sh`, `sdp_upgrade.sh`, `deprecated_files.txt`,
`push_bins.sh`, `ReleaseNotes.adoc`, `SDP_Guide.Unix.adoc`, etc.) is simply today's work
that never got additionally pushed down into the now-stale BETA rehearsal stream. **Nothing
is uniquely stuck in BETA that's missing from main.** `main` -> `dev`: confirmed a true
no-op (`p4 copy -S //p4-sdp/dev -n` -> "No target file(s) in both client and branch view").
The only remaining diffs between main and dev are 3 `ai_dev_support/` files that exist only
in `dev` -- expected and correct, since `ai_dev_support` is deliberately isolated from the
SDP product streams per the mutual-exclusivity rule (it's this project's own internal
tooling, not SDP release content). Conclusion: `main` is fully caught up and is the correct,
complete source of truth for cutting the real release; `r26.1.0.BETA` can be treated as a
discardable rehearsal snapshot.

## Gen7 `main` QA lab clean run -- RESOLVED, confirmed clean end-to-end

Root cause: the "P4CONFIG-resolution race" flagged earlier today was actually deterministic,
not timing-related. `install_sdp.sh`'s two `-local`-mode rsync steps (deploying the SDP tree,
then Immutable -> Writable) already excluded `.p4root`, `.p4config.local`, and
`.p4ignore.local` (DVCS-clone artifacts) but not the bare `.p4config` file that
`p4 clone`/`p4 fetch` also create. That file contains
`P4PORT=rsh:...p4d ... -r '$configdir/.p4root'` -- a local on-demand p4d pointed at a
`.p4root` that's deliberately excluded. Confirmed a live copy of exactly this poisoned
`.p4config` sitting in both `/opt/perforce/p4-sdp/sdp/` and `/opt/perforce/p4-sdp/p4/sdp/` on
the affected host. Any no-`-p` `p4` command run from inside the installed tree during the
window before the instance-specific P4CONFIG override takes effect gets hijacked into trying
to start that broken local server instead of the real one -- exactly the observed
"Root directory ... invalid ... `.p4root`: No such file or directory" / "TCP receive failed"
failure. **A real, customer-facing bug** in `install_sdp.sh`'s `-local` install mode, not a
BSW/test-harness artifact. Fixed by adding `--exclude=.p4config` at both rsync call sites.
`dev` 33547, Copy Up to `main` 33548.

Logged in as `bot_Claude_Anthropic` to the `/opt/perforce/.dev/sdp` DVCS clone's `origin`
remote, fetched and synced the fix in, and re-ran `qa_build` for real. Install now completes
fully ("SUCCESS: SDP Installation complete with no errors or warnings"), but Test 6 still
initially failed -- for an unrelated, second reason: `install_sdp.log` was a dangling symlink
(target cleaned up since Nov 2025) that never got refreshed. Root cause:
`[[ -e "$LogLink" ]]` dereferences symlinks, so it's false for a dangling one -- skipping the
cleanup block -- and the subsequent `ln -s` (no `-f`) then fails silently on the leftover
link, forever. Fixed by also checking `-L` in the guard. `dev` 33549, Copy Up to `main`
33550. Re-synced and re-ran `qa_build` a third time: **all 6 tests PASS.** Gen7 `main` is now
genuinely clean, fully confirmed live, not just mechanically proven. Followed by a full
7-platform SDP Installer Test Suite run (616/616 passing, after also fixing a stale
`cli_tests.cfg` Test 26 assertion to expect `--exclude=.p4config`, change 33551).

## Ship It: the SDP 2026.1 GA release, executed step-by-step

User: "let's start turning the crank! Ship It!" -- with an explicit request to pause and
await confirmation after each numbered step in `doc/ReleaseProcessOverview.md`, treated as a
standing human-in-the-loop accountability practice, not a one-off pacing request. Created a
new skill, **`sdp-release-process-run`**, to codify this (the one-step-then-stop discipline,
AI-specific shell/workspace-switching gotchas, and judgment calls worth surfacing rather than
silently resolving) -- `dev` change 33553, updated twice more during the run (33559, 33571)
as real gotchas were hit live.

Two process-doc improvements folded in before starting, both now permanent:
- The "Don't release on a Friday (or just before a major holiday)" scheduling caution,
  enshrined into Step 1 per Technical Support's request in the Escalations meeting.
- A preview-first best practice for the "Merge Down, Copy Up" mantra: always `-n` preview a
  merge/copy before running it for real, and get situational awareness on any already-opened
  files that might overlap -- not to avoid merging into opened files absolutely, just to never
  be surprised by it.

Executed Steps 1 through 21 in full, one at a time, with the user confirming after each. Real
bugs and gaps found and fixed live, during actual release execution (not hypothetically):

- **Retargeted SDP-1386** to 2026.2 (Step 2) -- an open issue that wasn't ready for 2026.1.
- **`sed -i -E` macOS/BSD portability bug** in Step 11's own documented revnumber/revdate
  script: `sed -i -E -e "..." "$f"` on macOS's BSD `sed` treats `-E` as `-i`'s required
  backup-suffix argument, silently creating 15 stray `*.adoc-E` backup files with pre-edit
  content (which then showed up as bogus "opened for add" candidates at Step 14). Reverted
  the accidental adds, deleted the stray files, and rewrote the script to the portable
  `sed -e "..." "$f" > "$f.tmp" && mv "$f.tmp" "$f"` form (no `-i` at all) in both `dev` and
  `main` (`main` change 33560).
- **Deferred a deeper `.adoc` content review** (outdated references across all 15 doc files)
  to SDP 2026.1.1, as a tracked task rather than a JIRA issue, per the user's call --
  reasonable to defer, not worth blocking release day.
- Cut the real release stream, **`//p4-sdp/r26.1.0`**, via templated `p4 stream -o` +
  `p4 populate -r -S`, tagged `r26.1.0.33565`. Final regen (Step 18), packaging (Step 19,
  `sdp.Unix.tgz`/`sdp.Windows.zip`), and publishing the tarball to `main`'s permanent download
  URL (Step 20, change 33570 -- caught and fixed an incomplete first pass that only copied the
  versioned-named files, missing the bare `sdp.Unix.tgz`/`sdp.Windows.zip` that
  `install_sdp.sh` actually downloads from).
- **`Version`** bumped to `Rev. SDP/MultiArch/2026.1/33562 (2026/09/08).`
- **Release Notes** (Step 16): Jobs Fixed list ordered by JIRA number, descending (highest
  first, per explicit correction mid-edit), "What's Coming Next in SDP 2026.2?" filled in
  (OS Package work, retiring Docker/Podman/Python suite, AI-skills-as-product-content).

## Swarm/P4 Code Review cleanup: SDP-1385 and the branch-naming rebrand

Researched (official Swarm docs + public Swarm source) and confirmed to the user's
satisfaction: Swarm's mainline-branch detection matches by branch **Name** (case-insensitive,
against `config.php`'s `mainlines` list, first alphabetical match wins) and **not** `id`;
`id`/`name` are independently persisted with no derivation link; and a review's branch
association is a persisted field on the review record itself, not recomputed from current
branch definitions. Net: renaming a Swarm branch's `name` is safe and reversible, and does not
orphan existing reviews.

On that basis, the user retired the redundant `p4-sdp` Swarm project (created during the early
Classic-to-Streams migration phase, since superseded) and renamed branches in
`perforce-software-sdp` so the new Streams branch names are the natural, undecorated stream
names (`main`, `dev`, `r26.1`, `dev_rebrand`), with the old Classic-era branches renamed
`OldMain`/`OldDev`/etc. to get them out of the way of Swarm's alphabetical-match ordering.
Verified live post-rename: the Overview page correctly renders the new 2026.1 `main` content,
and open reviews on the old Classic branches remained accessible. On that basis, marked
**SDP-1385 complete**, and with it, **SDP-1170** (the whole Classic-to-Streams migration
effort) as complete.

## JIRA release gating: the Resolution-field gap

Closing out SDP-1170/1385 surfaced a real snag: JIRA's "Release 2026.1" dialog reported "10
unresolved work items" despite every one of them showing Status = Closed. Diagnosed via JQL
(`resolution = Unresolved`) that this project's "Close" transition moves `status` to `Closed`
but never sets `resolution` -- and JIRA's release-blocking check is driven by `resolution`
being empty, independent of `status`/`statusCategory`. Not a JIRA admin change, just the first
time anything actually checked. Fixed by explicitly setting `resolution: Done` on all 10
issues; confirmed 0 remain unresolved. **User then executed the actual "Release 2026.1"
action in JIRA** -- SDP 2026.1 is now formally released in JIRA.

Captured both this gotcha and the user's ask to remind whoever runs this step to **uncheck
"Create release notes"** in JIRA's Release dialog (JIRA's own release-notes feature isn't used
for SDP; the real ones live in `ReleaseNotes.adoc`) into the `sdp-release-process-run` skill.

## Step 22 (renumbered from the old Step 22; see below): Post-Release Smoke Tests

**Fresh install, from scratch.** Since Gen6/Gen7 lab testing had already gotten thorough
coverage, did this one for real: launched a plain Rocky Linux 9.8 EC2 instance (not part of
the BSW fleet, no test-suite software), attached 3 unformatted EBS volumes matching
`SampleStorageSetup-AWS.adoc`'s starting-state scenario, formatted/mounted them, downloaded
the actual published `sdp.Unix.tgz` from the live download URL (confirmed
`2026.1/33562`), and ran `install_sdp.sh` through the documented procedure exactly as a new
customer would. p4d came up fully functional (SSL by default, checkpoints wrote cleanly).
Found and fixed/filed three real issues along the way, confirmed on a second, untouched Rocky
9.8 instance to rule out one-off flukiness:

- **SDP-1390** (filed, targeted 2026.1.1): `neovim`/`screen`/`systemd-coredump` fail to
  install on stock Rocky/RHEL 9 -- they require EPEL, which `install_sdp.sh` never enables.
  Reproduced deterministically on both VMs.
- **SDP-1391** (filed, targeted 2026.1.1): `ExtraP4PackageList[yum]` is empty, so the
  "Perforce Package Repository" extras step literally runs `yum install -y  < /dev/null` (no
  package name) -- yum rejects it with a usage dump, logged as the misleading
  "Not all Perforce packages installed successfully" warning. Cosmetic/non-fatal but
  confusing log noise on every RHEL-family install.
- **Doc inconsistency, fixed directly** (not JIRA, trivial): `SampleStorageSetup-AWS.adoc`
  mounted sample volumes at `/p4depots`/`/p4db`/`/p4logs`, while `install_sdp.sh`'s actual
  generated config defaults to `/mnt/p4depots`/`/mnt/p4db`/`/mnt/p4logs`. Fixed all 9
  occurrences in `dev`, change 33572.
- One GPG-trust dnf failure (all 30 standard packages failing on the "Problem repository:
  perforce" complaint) was seen once on the first VM but did **not** reproduce on the second
  -- treated as a probable cold-boot/networking fluke, not filed.

Both EC2 instances and their ephemeral security group were torn down immediately after
testing.

**Real upgrade, dogfooded.** User upgraded the entire PPN and Public Depot server fleets
(both SDP and P4D) against the real, documented upgrade procedure. **No significant issues** --
only the two minor items already tracked below (the `helix_binaries` leftover-JSON cleanup
gap, and the SDPInfo per-instance motd idea).

## Two more real-world findings from the PPN/Public Depot upgrade

- **`deprecated_files.txt` gap, fixed immediately.** Observed post-upgrade: `helix_binaries/`
  wasn't removed, because it wasn't actually empty -- `P4.json`/`P4D.json`/`P4Broker.json`/
  `P4Proxy.json` (written by `get_helix_binaries.sh`) were never in the known-files list, so
  the three-pass `rmdir` logic silently left the directory in place. Added all 4 to
  `deprecated_files.txt`. No JIRA (too trivial); `dev` change 33577.
- **SDP-1392 filed** (Story, unversioned): enhance the `SDPInfo` trigger (`sdp_info.sh`) to
  look for a per-instance `motd.${SDPInstance}.txt` first, falling back to the existing
  shared `motd.txt`, for smoother multi-instance-site defaults. Flagged an open
  implementation question in the ticket: the script currently has no way to know its own
  `SDPInstance` at all (no instance param passed by the trigger, doesn't source `p4_vars`) --
  whoever picks this up needs to solve that first.

## Classic depot: "We've Moved" banners submitted (SDP-1170 follow-through)

The Round-3 Classic-depot changes -- staged since earlier in the week, gated on the real GA
happening -- were submitted now that it has. Adds `WeHaveMoved.md`, a "this location has
moved" banner to `README.md` and every generated doc HTML page, a one-line `warnmsg` in the
Classic `install_sdp.sh`/`upgrade.sh` pointing at the new home (their actual 2025.2-Patch-1
install/upgrade behavior is unchanged), and removes the now-redundant `doc/README.html`.
Nothing in the Classic area is deleted or broken -- it remains supported per SDP's normal N-2
window (until SDP 2028.2 ships). Submitted with the `BYPASS_TRIGGER` marker required for this
frozen depot path, change 33578.

While in the area, found and fixed a genuine shipped placeholder: `ReadMe.md` under
`Server/Unix/p4/common/site/` had a literal `EDITME-UpdateThisURL` in its Helm Management
System (HMS) link -- present in the actual 2026.1 GA tarball. Fixed independently in `dev`
(33574), `main` (33575), and directly in the live `r26.1.0` release stream as a doc-only Hot
Fix per `ReleaseProcessOverview.md`'s own Hot Fix definition (33576, no tarball regen needed).

## New process step: announcing the release (gap found, fixed for future releases)

User asked whether the release process already covered posting to the internal `#p4-sdp`
Slack channel -- it didn't, at all, anywhere in the 22 steps. Added as a new **Step 22:
Announce the Release** in `dev` (between the old Step 21 "Release in JIRA" and the old Step
22 "Post-Release Smoke Tests", which is now renumbered Step 23), change 33573. Doesn't apply
retroactively to 2026.1's already-shipped docs -- rides along for 2026.1.1. (The user
separately posted the actual 2026.1 GA announcement to `#p4-bsw` by hand tonight.)

## `//test-install_sdp` dev -> main: first-ever population, plus a real permissions puzzle

Per the user: unlike the SDP project's routine, frequent Merge Down / occasional Copy Up
rhythm, `//test-install_sdp/main` had literally never been populated -- zero files, zero
history. This was a first-time `p4 populate -S //test-install_sdp/dev`, not an incremental
Copy Up.

Hit a genuine, non-obvious permissions puzzle along the way: `bot_Claude_Anthropic`'s
`p4 populate`/`p4 add` attempts against `//test-install_sdp/main/...` failed with "no
permission," despite a `write group AI_Agents ... //test-install_sdp/...` rule that looked
like it should cover it. Root cause, confirmed via `p4 info`: that `write` grant is
IP-restricted to `108.86.9.164`, but the actual TCP connection to the Public Depot server was
arriving as `10.151.120.30` (a VPN-related routing quirk on the user's end -- a plain
`ifconfig.me` lookup still showed the expected public IP, but the Perforce-bound traffic
specifically was taking a different path). The user fixed this properly, not by chasing the
network path: added `bot_Claude_Anthropic` to the `SDP_Developers` group with an
IP-unrestricted `write` grant on `//test-install_sdp/main/...`, which sidesteps the whole
VPN/NAT question. Populated successfully once that landed: change 33579, 28 files branched.

## Gen6 BSW lab cleanup

Identified the labs created during this week's upgrade-testing cycle via
`lslab.sh -r us-east-2` filtered to the `sdp-upgrade-test` naming pattern:
`sdp-upgrade-test`, `test2`, `test3`, `test4` (9 labs total in the region; these 4 were ours).
Previewed each with `rmlab.sh <lab> -n` first, then tore all 4 down for real. Confirmed via a
follow-up `lslab.sh` that all 4 are gone (9 labs -> 5, remaining 5 unrelated to this cycle).
Two other Gen6-tagged AWS resources exist but were correctly left alone: a colleague's
unrelated lab (`mzinthefer`, June 2026) and the long-standing `bsw-ttyler-SDP_QA-Gen6-*` lab
(2024-vintage) that gets reused, not recreated, across cycles.

## Open / carried forward

- **Jenkins podman crash on `//p4-sdp/main`**: never actually root-caused. Worked around by
  supplementing with the manual Gen6/Gen7 lab exercises and the fresh-install/upgrade
  dogfooding above; the release shipped without a green Jenkins run on `main`. Still worth a
  nudge to Robert separately, since the underlying container-state issue is unresolved.
- **`//p4-sdp/r26.1.0.BETA`**: the pre-release rehearsal stream is no longer needed now that
  the real `r26.1.0` has been cut, and hasn't been cleaned up. User's call tonight: probably
  just leave it be for now (it's cheap, structurally harmless clutter), but floated the idea
  of eventually hiding rehearsal-type streams like this via Protections once there are enough
  release streams accumulated that they start looking cluttered in listings. Not decided,
  not urgent -- worth revisiting years down the road, not now.
- **Deep `.adoc` content review**: deferred to SDP 2026.1.1 as a direct task (not JIRA), per
  the user's call during Step 11 of the release process.
- **SDP-1390/1391/1392**: filed, unresolved. 1390/1391 targeted at "2026.1 Patch 1"; 1392
  left unversioned pending triage.