# Session Log -- 2026-09-09

## Summary

Day after SDP 2026.1 GA shipped. Started with a routine investigation of a customer's
`refresh_P4ROOT_from_offline_db.sh` log (SDP 2025.1/31674) that surfaced one real,
still-live bug (SDP-1393, `verify_sdp.sh`'s spurious `grep: off: No such file or
directory` with `-L off`) and confirmed a second suspected bug was already fixed. Also
created a new `sdp-jira-jobs` skill capturing conventions accumulated during the release
cycle, and retroactively applied its new Summary/Description convention to SDP-1390
through SDP-1393.

The day's real event: the user found a genuinely critical, still-shipping bug --
`install_sdp.sh` and several related scripts hardcoded the *planned 2026.2* OS package
directory name (`/opt/perforce/p4-sdp`) instead of the current, correct 2026.1 name
(`/opt/perforce/helix-sdp`). What started as reviewing the user's own quick patch (change
33608) turned into a full sweep finding the bug was far more widespread than initially
fixed, a second, independently-discovered upgrade bug (SDP-1395, a false "FAILED" status
on successful upgrades), two full Emergency Bug Fix cycles (fix -> verify via a real
install-then-upgrade test -> Release Notes -> repackage -> publish -> verify live), a full
Merge Down chain (`r26.1.0` -> `main` -> `dev`) to get the fixes back into ongoing
development, a post-merge doc-regen sweep across all three streams (catching a third,
smaller stale-doc bug along the way), and a real upgrade-test run on the new
`install_sdp.sh` test fleet to close the loop on a test-coverage gap this whole incident
exposed. Also fixed a stale "SDP and P4D Version Compatibility" section in the Release
Notes (should have been updated at GA, wasn't) while in the area.

The new automated coverage immediately paid for itself: a full run of the Helix Install
Test Suite across the whole EC2 fleet turned up a **fourth** instance of the same bug
class (SDP-1396, `opt_perforce_sdp_backup.sh` hardcoding the same premature path
independently of the already-fixed `SDPPackageBase` variable), reproducing identically on
every platform except Rocky 8. Fixed and verified live in `dev`/`main`; not yet in the
shipped `r26.1.0` tarball -- whether that warrants a third same-day Emergency Bug Fix is
an open decision for the user.

## SDP-1393: verify_sdp.sh's spurious "grep: off" warning

User shared a real customer's `refresh_P4ROOT_from_offline_db.sh` log (SDP 2025.1/31674,
2026-08-24) with two things that looked like bugs: `grep: off: No such file or directory`,
and `mv: replace ... overriding mode 0440` prompts during the root/offline_db symlink
swap.

**Bug 1, confirmed still live**: `verify_sdp.sh`'s two end-of-run summary blocks
(warnings-only and errors+warnings cases) unconditionally do
`grep ^Warning: "$Log"` / `grep -E '^(Error|Warning):' "$Log"`. When called with `-L off`
(exactly how `refresh_P4ROOT_from_offline_db.sh`'s `switch_db_files()` calls it), `$Log`
retains the literal string `"off"`, so this tries to grep a nonexistent file named `off`.
Confirmed via `p4 print ...@31674` that this exact unguarded line is unchanged since 2025.1
and still present in current `main`. Filed as **SDP-1393**, targeted at "2026.1 Patch 1".

**Bug 2, already fixed**: the `mv` calls in `backup_functions.sh`'s `switch_db_files()`
(the "Switching root and offline_db links" step) lacked `-f` at 2025.1/31674, matching the
mode-override prompts in the customer log exactly. Confirmed current `main` already has
`-f` on every one of these calls -- fixed at some point between 2025.1 and now, untracked
but done. Nothing to file.

## New skill: sdp-jira-jobs, and a retroactive convention fix

User asked for a new convention: every SDP JIRA issue's Description should start with the
Summary duplicated verbatim as its first line (blank line, then the full explanation),
Summary targeted at ~70 characters. Reason: in case a future JIRA -> P4Jobs integration
gets built, it would likely rely on this convention (mirroring how a native P4 job's own
`Description` field already works).

Created a new skill, **`sdp-jira-jobs`** (mirroring `bsw-ppn-jobs`'s structure but for
JIRA), documenting this plus the P4Blog/Resolution-field gotchas and fixVersion naming
pattern (`"2026.1 Patch 1"` for what people call "2026.1.1") accumulated during the
release cycle. Submitted as `//test-install_sdp/dev` change 33604. Also fixed a real gap
while there: the new `sdp-release-process-run` skill (created the night before) had never
actually been added to the master `SKILLS_INDEX.md` -- fixed alongside adding the new
skill's own entry, change 33605.

Retroactively applied the new Summary/Description convention to **SDP-1390 through
SDP-1393** (all filed before the convention existed). While doing so, found and fixed a
real encoding glitch: SDP-1391's Summary had a literal `<` HTML entity instead of a
plain `<`. Fixed the Summary field directly, and added a new "stick to plain ASCII" rule
to the skill (change 33606) citing this exact incident as the reason.

## SDP-1394 and SDP-1395: the premature /opt/perforce/p4-sdp path, and the false-FAILED upgrade bug

User found this by inspecting their own quick emergency patch (change 33608, fixing
`sdp_upgrade.sh`/`opt_perforce_sdp_backup.sh`/its systemd unit) and asked for a broader
scan for any other accidental dependency on the not-yet-existing `/opt/perforce/p4-sdp`
structure (the OS package directory rename planned for 2026.2, not 2026.1).

**Scope turned out to be much larger than the initial patch.** Found the same
`SDPPackageBase="$PerforcePackageBase/p4-sdp"` bug in three more files the initial fix
missed -- critically, **`install_sdp.sh` itself**, the actual customer-facing installer,
meaning every fresh package-structure install of 2026.1 was creating the wrong top-level
directory. Also `mkdirs.sh` and `configure_sample_depot_for_sdp.sh`. Additionally found:
`install_sdp.sh`'s `BackupScript` variable hardcoded the same wrong path directly (feeding
into SELinux `fcontext`/`restorecon` calls that would silently target a nonexistent path),
and `sdp_upgrade.sh` auto-set `MigrateSDPToPackageDirs=1` whenever it detected an existing
`helix-sdp` structure, which would trigger an unwanted migration attempt during a routine
upgrade. Filed as **SDP-1394**.

Fixed all of it directly in `r26.1.0` (changes 33608 by the user, 33609 the rest), mirrored
into `main` (change 33610, after confirming `main`/`dev` had never received any of it).
Regenerated affected man pages in both streams (`mkdirs.sh`'s embedded `-man` text had the
same stale path baked in), bumped `r26.1.0`'s `Version`, repackaged, and republished to
`main`'s permanent download URL -- verified live via a fresh public-URL download.

**While verifying SDP-1394 with a real install-then-upgrade test** (installed genuine SDP
2025.2 Patch 1 via `install_sdp.sh`, fetched from the frozen Classic depot at the exact GA
changelist, then upgraded to the fixed 2026.1 tarball), found a **second, independent
bug**: `sdp_upgrade.sh`'s STEP 9 unconditionally chowned `$SDPPackageBase/p4_binaries` (a
top-level directory that only exists for `-local`-mode pre-staged binaries) -- for a
normal install, the real `p4_binaries` lives nested at `$WritableSDPDir/p4_binaries`,
already covered by a preceding recursive chown. Root cause: this cycle's
`helix_binaries` -> `p4_binaries` rename was a literal find-and-replace of the 2025.2 line
(which correctly referenced a real top-level directory back then) without accounting for
the location semantics changing. Result: every normal package-structure upgrade reported a
false **"SDP Upgrade FAILED"** even though the upgrade fully succeeded underneath. Filed as
**SDP-1395**.

Given we were still under 24 hours from GA and didn't want a bad upgrade experience to be
anyone's first impression of the release, fixed this the same way, but improved the process
based on the first round's mistakes (see below): fix in `r26.1.0` (change 33622, removed the
redundant/broken chown line entirely, since the real directory is already covered) ->
mirrored to `main` (change 33623) -> updated Release Notes (changes 33626/33627, calm
"corrected the same day" wording, no alarming language) -> regenerated HTML **and PDF**
(changes 33628/33629, since an EBF to a shipped release is a real release event) -> bumped
`Version` to 33622 -> repackaged -> **verified with a real install-then-upgrade test using
the freshly-built local tarball, before publishing** -> published -> verified live.

Both SDP-1394 and SDP-1395 closed in JIRA with `resolution: Done` and `P4Blog` set to their
respective r26.1.0 fix changelists.

## Process improvement: the EBF flow, learned by doing it wrong once

The first round (SDP-1394) published before updating Release Notes, requiring a second,
avoidable repackage-and-republish cycle once the gap was noticed. Documented the correct
order of operations in `ReleaseProcessOverview.md`'s Emergency Bug Fix section (both `dev`
change 33633 and `main` change 33634): fix code -> update Release Notes -> regenerate HTML
**and PDF** (unlike routine dev, an EBF to a shipped release is a real release event) ->
regenerate any man page whose embedded usage text changed -> bump `Version` -> repackage ->
**verify with a real test before publishing** (using the local tarball directly, no need to
wait for a publish/download round-trip) -> publish -> confirm live. Also documented two
AI-specific gotchas hit for real: release-stream write access blocked by the same
VPN-routing/IP-restricted-protections issue as `test-install_sdp` (group-based unrestricted
grant is the fix, not chasing the network path), and tools that shell out to plain `p4`
internally (`gen_script_man_pages.sh`, `package_downloads.sh`) needing `P4PORT`/`P4USER`
exported as real environment variables, not just passed as `-p`/`-u` flags to the outer
script.

## The full Merge Down chain: r26.1.0 -> main -> dev

User corrected a terminology/mechanics point mid-flow: for changes originating in an
already-cut release stream (the "top"/"north" of the flow, in the user's phrasing -- release
streams are north of `main`, dev streams are south), only a **Merge Down** is needed (all
the way to `dev`), not a Copy Up -- there's nothing in `dev` that needs to travel north,
since nothing originated there.

**r26.1.0 -> main**: required temporarily flipping `r26.1.0`'s `notoparent` stream Option to
`toparent` (approved by the user after the auto-mode classifier blocked the piped
`p4 stream -o | sed | p4 stream -i` command as risky), then `p4 merge` (not `p4 copy`, since
`main` might have diverged) from `main`'s client, resolved cleanly (8 files: 7 had only
RCS `$Id:`/`$Change:` keyword differences, `ReleaseNotes.adoc` had the real P4D Version
Compatibility content), regenerated ReleaseNotes HTML/PDF, submitted (change 33637), flipped
the Option back.

**main -> dev**: found the user had a pending, unsubmitted change (33607) in `dev` --
their own earlier, partial version of the exact same SDP-1394 fix, on the same files this
merge would touch, with a note "we should scan for other possible places" (exactly what
turned up the additional bugs). User reverted it themselves (I lack `admin` access needed
for `p4 revert -C <client>` on another user's client). Then hit a real mechanics gotcha:
running `p4 merge -S //p4-sdp/dev` while already on a client dedicated to that exact stream
produced a misleading "needs copy not merge in this direction" error -- the redundant `-S`
confuses direction detection. Plain `p4 merge` (no `-S`) worked correctly. The merge swept
up not just today's fixes but a **backlog of ~15 unrelated files** with only trivial
`:revdate:` differences dating back to Step 11 of the GA release, never previously merged
down -- confirmed each was trivial before resolving, then `-am` auto-resolved everything
cleanly (0 conflicting), preserving `dev`'s own unique content (Step 22 "Announce the
Release", the AWS storage mount-point fix) alongside `main`'s incoming changes. Submitted as
change 33641 (23 files).

Documented this as a new, explicit step 9 in the EBF order-of-operations (dev change 33642,
main change 33643): "Merge Down the fix all the way to `dev`" -- flagged as the step that's
genuinely easy to forget once a fix is live, plus the north/south-of-`main` directional
framing and the `-S`-redundant-flag gotcha.

## Post-merge doc regen sweep, all three streams

Checked for doc regen needs after the merge chain, per the user's request (PDF regen
allowed in `r26.1.0`/`main`, explicitly **not** in `dev`):

- **`dev`**: `make`'s timestamp-based dependency check flagged ~13 HTML files as stale
  (mostly a P4-sync-resets-mtimes artifact, not real content changes). Regenerated all
  flagged files, then checked real `p4 diff -sa` content diffs before submitting --
  reverted the one genuine no-op (`README.html`), kept the 10 with real diffs (the
  `:revdate:`-driven backlog files plus `ReleaseNotes.html`).
- **`main`**: found `SDP_Guide.Unix.html`/`.pdf` and `TheSiteDirectory.html` (rendered from
  `Server/Unix/p4/common/site/ReadMe.md`) were stale after the r26.1.0->main merge.
  Regenerated and submitted (change 33644); reverted `SDP_Legacy_Upgrades.Unix.html`/`.pdf`
  after confirming the HTML had zero real diff (the PDF-only "diff" was pure embedded
  timestamp metadata noise).
- **`r26.1.0`**: same staleness check found **a third real bug still in the live shipping
  tarball** -- `TheSiteDirectory.html` still had the `EDITME-UpdateThisURL` placeholder from
  before the `ReadMe.md` HMS-link fix (submitted days ago), because it was never
  regenerated when that fix landed. Confirmed via the exclusion config and a live tarball
  extraction that this file genuinely ships. Regenerated (change 33645), repackaged
  (change 33646), republished to `main` (change 33647), verified live.

## Test-coverage gap, closed for real

The whole SDP-1394/1395 incident traced back to a real gap: all upgrade testing this
release cycle exercised the Gen6 BSW lab's `reset_sdp.sh`-based flow, never a real
`install_sdp.sh` install followed by a real `sdp_upgrade.sh` upgrade.

Tried the **old** `install_sdp.sh` test fleet first (`ssh-install_sdp-test-r8x86_64.sh`,
word-order-swapped naming) as an opportunistic pre-existing-state fixture -- found it was a
broker-only partial fixture (no real `p4d` instance), still useful as one test scenario but
not sufficient alone. Did a full, controlled test on my own ad hoc EC2 VMs instead: real SDP
2025.2 Patch 1 install (fetched from the frozen Classic depot) -> real upgrade to the fixed
2026.1 tarball, for both SDP-1394 and SDP-1395 -- this is what actually caught SDP-1395 in
the first place.

Then closed the loop on the **new**, correct fleet (`ssh-test-install_sdp-*` wrapper
scripts, machine `p4d-test-r8-01`): found it already had leftover contamination (a
`p4-sdp` directory despite running 2025.2, from earlier ad hoc testing), reset it cleanly
with `DANGER_CLEAN.sh`, did a real 2025.2 Patch 1 install, then upgraded to the final
published 2026.1 tarball -- clean success, confirming both fixes and that this fleet is a
viable, repeatable venue for real upgrade testing going forward, not just fresh-install
testing.

## SDP and P4D Version Compatibility: also stale

While fixing SDP-1395's Release Notes entry, user separately noticed `ReleaseNotes.adoc`'s
"SDP and P4D Version Compatibility" section was stale -- still listed 2025.2 as the latest
release and included P4D 2024.1 (past its End of Maintenance date). Fixed in `r26.1.0`
(change 33635, doc-only, no tarball changes needed) to list 2026.1 through 2024.2, updated
the "As of" date and example sentence, then real-merged into `main` as part of the larger
r26.1.0->main Merge Down above.

Added new guidance to `ReleaseProcessOverview.md` Step 16 (dev change 33639, main change
33640), marked "(Major Releases Only)": review/update this section each major release,
using the P4 Server (P4D) End of Maintenance lifecycle table
(https://portal.perforce.com/s/article/Helix-Core-Maintenance-Lifecycle-Helix-Core-Server-P4D)
as the source of truth for which P4D versions to list, not an assumed fixed-length window.

## Test-coverage gap: now documented as a repeatable procedure, not just demonstrated once

Followed up same day: documented the manual install-then-upgrade procedure (proven out live
above) as a permanent, repeatable part of two places, so the gap doesn't quietly reopen:

- **`install-sdp-test-suite-run` skill** (change 33648): new "Manual upgrade testing (not
  yet part of the automated suite)" section -- `DANGER_CLEAN.sh` -> install a real older
  release (fetched from wherever it actually shipped from) -> upgrade to the release
  candidate's actual tarball -> verify `Version`, directory structure, and (if applicable)
  that `p4d` still answers. Notes the old fleet's ad hoc-state caveat (one Rocky 8 box there
  is a broker-only fixture, not a full `p4d` instance) and that this is deliberately *not yet
  automated* into `cli_tests.cfg`.
- **`ReleaseProcessOverview.md` Step 4** (dev change 33649, main change 33650): flagged that
  Regression Test Suite 3's automated coverage is install-only and never exercises
  `sdp_upgrade.sh` -- the exact gap that let SDP-1394/1395 ship -- and pointed to the new
  skill section, recommending it run pre-release (here) rather than only as part of Step 22's
  post-release smoke tests, for any release touching the package-structure install/upgrade
  path.

## Automating the upgrade-test into cli_tests.cfg, and a real gap that surfaced along the way

Same day, user asked to actually automate this into `//test-install_sdp/dev/cli_tests.cfg`
(not just document a manual procedure) and validate it on the real EC2 fleet. Confirmed via
`AskUserQuestion` that this meant the fleet's own `cli_tests.cfg` (Step 4's Regression Test
Suite 3), not the separate, differently-scoped `//p4-sdp/dev/test/bsw/cli_tests.cfg` used by
the Gen7 lab's `qa`/`qa_build` mechanism (that file tests functional checks against an
already-running instance -- `verify_sdp.sh`, `p4verify.sh`, `ccheck.sh` -- not
`install_sdp.sh`/`sdp_upgrade.sh` at all).

**Found something much bigger while adding the test**: `cli_tests.cfg` itself hardcoded
`/opt/perforce/p4-sdp` in 56 places -- setup steps for `-local` installs staging binaries
in the (now-wrong) directory, and output assertions expecting the wrong path in
`OnlineInstall*`/`PreflightAbort`/`DryRun` tests. After the SDP-1394 script fix, running
this suite as-is would have failed almost everywhere. Fixed all 56 (left the two genuine
`.p4-sdp.home_dir_backup` filename references alone -- a fixed identifier, unrelated to
package-structure naming). Change 33652.

Added a new **`Upgrade`** test group: installs the real SDP 2025.2 Patch 1 as a genuine
pre-upgrade baseline, then upgrades to the current `dev` tree (the release candidate) via
the documented procedure, with explicit regression guards for both SDP-1394 (no premature
`p4-sdp` directory) and SDP-1395 (`sdp_upgrade.sh` must report real success).

**Testing it for real caught a bug in the test itself.** First run through the actual
harness (`./r Upgrade` -- note: positional argument, not `-g <name>`; `./r -g Upgrade`
silently matches zero tests) failed 6 of 15, all cascading from one root cause: the
Upgrade group's own setup step tried `p4 print` against the Classic depot from within
`/opt/perforce/dev/sdp`, but that directory's `.p4config.local` is the DVCS clone's own
local on-demand-p4d shadow config (`P4PORT=rsh:...`), not a real connection -- and the test
harness's own sandboxed execution environment has no other P4 connection info available
("TCP connect to perforce:1666 failed"). Fixed by checking in a static fixture
(`test_fixtures/install_sdp.2025.2-Patch1.sh`, the real fetched 2025.2 Patch 1 script) and
copying it locally at test-time instead of a live cross-depot fetch -- more robust for
automated testing anyway, not just a workaround. Change 33653. Re-ran through the real
harness: **all 15 tests pass**, genuine end-to-end execution confirmed (real
`sdp_upgrade.sh` preflight/upgrade output visible in the log, not a trivial pass).

## DANGER_CLEAN.sh compatibility for the new Upgrade test group

New general rule from the user: every test group must leave the machine in a state fully
cleanable by `DANGER_CLEAN.sh` (files, OS users, services) so groups can run in any order
-- the one accepted exception is OS package installations, never cleaned between runs.
The new `Upgrade` group (change 33653) violated this: it used `/root/install_sdp_old`
(not covered by `DANGER_CLEAN.sh`'s `SDPInstallDir="/root/install_sdp"`) and let
`sdp_upgrade.sh` write its own log to a bare, uncovered `/root/sdp_upgrade.<timestamp>.log`.
Fixed (change 33655) by renaming to `/root/install_sdp` (matching convention, already
covered) and adding `-L /root/install_sdp/sdp_upgrade.log`. Verified via a real
`DANGER_CLEAN.sh -y` run that both are now gone. This was the first test group to exercise
`sdp_upgrade.sh` at all, so this particular litter case had never come up before. Also
manually cleaned up pre-fix debris left over from iterative testing on `p4d-test-r8-01`.

Separately flagged, not fixed (out of scope, pre-existing, unrelated to today's changes):
hundreds of accumulated `gen_sudoers.<timestamp>.log` files in `/root/` on `p4d-test-r8-01`
dating back to August 13, from `install_sdp.sh`'s own `gen_sudoers.sh` call on every test
run, never cleaned by `DANGER_CLEAN.sh`. Either `gen_sudoers.sh` needs to clean up after
itself or `DANGER_CLEAN.sh` needs to know about this path -- user's call, later.

## Full Helix Install Test Suite run, and a fourth bug (SDP-1396)

With the `Upgrade` group now added and `DANGER_CLEAN.sh`-safe, ran the full suite across
the whole EC2 platform fleet (Ubuntu 22/24/26-aarch64, Rocky 8/9/10, SLES 15) to validate
today's changes didn't regress anything and that the new coverage holds up broadly.

**Every platform except Rocky 8 came back with the same 12 failures**, all in
`BackupRecoveryMultiInstance` (Tests 234, 235, 244-253; Ubuntu 24 additionally failed Test
232, a related symptom of the same cause -- the journal window not containing the
completion line because the backup never truly succeeded). Root cause: **a fourth
instance of the SDP-1394 bug class**, this time in `opt_perforce_sdp_backup.sh` itself --
it independently hardcodes the literal string `"opt_perforce_p4-sdp.$ThisHost"` for its
backup/staging directory names and the tar command's internal structure (lines 436, 440,
490, 1067, 1069, plus five spots in its own `-man` usage text), completely separate from
the `$SDPPackageBase` variable that SDP-1394 already fixed correctly. Live evidence from
the Ubuntu 22 run: `Backups are here: /mnt/p4depots/backup/opt_perforce_p4-sdp.p4d-test-u22-01`
-- the wrong, not-yet-existing 2026.2 name, even on a fresh install using the
already-fixed `install_sdp.sh`. A broader grep also caught 4 more stale comment-only
references (not functional bugs, but same class) in `mkdirs.sh`/`install_sdp.sh`,
including one that wrongly said the required `-local`-mode subdirectories were "p4-sdp and
downloads" instead of "sdp and downloads".

Fixed in `dev` (change 33658; plus a separate change 33659 for ~11 already-correct-at-the-
-source but stale-generated HTML files found opened alongside it, catch-up regen for the
already-submitted SDP-1379 Helix-to-P4 rebrand text -- confirmed safe via `p4 diff`/`p4
opened` against the source `.adoc`/`.md` files before submitting, not something I broke).
Copied Up to `main` (change 33660 for the fix plus the rest of an outstanding dev->main
backlog: `ReleaseProcessOverview.md`, `SampleStorageSetup-AWS.adoc`,
`deprecated_files.txt`; change 33661 for the man-page/`SDP_Guide.Unix.html`+`.pdf` regen).
Used `p4 integrate` (not `copy`/`merge`) for this Copy Up after both stream-aware commands
refused with contradictory direction errors (`copy`: "cannot 'copy' over outstanding
'merge' changes"; `merge`: "needs 'copy' not 'merge' in this direction") -- a real
mechanics gotcha, apparently triggered by today's earlier Merge Down having touched the
same files. No Merge Down back to `dev` was needed afterward (`p4 istat` showed
`parentChange` already caught up, since the doc-regen-only files in main's follow-up
change are outside `dev`'s branch view).

Verified live: re-ran just the `BackupRecoveryMultiInstance` group against the freshly-
fixed `dev` tree on the Ubuntu 22 fleet machine (synced via `p4 fetch`/`p4 sync` in the
DVCS clone) -- all tests pass, backup now correctly uses
`/mnt/p4depots/backup/opt_perforce_helix-sdp.<host>`.

Filed as **SDP-1396**, fixVersion "2026.1 Patch 1" (matching SDP-1395's precedent -- not
yet fixed in the shipped `r26.1.0` tarball), closed with `resolution: Done` and `P4Blog`
pointing at the `dev` fix changelist.

**Separate, minor, non-reproducing finding**: Rocky 8's run also failed Tests 12-15
(`ccheck.sh`/`opt_perforce_sdp_backup.sh`/`configure_new_server.sh -V` all reporting
`show_versions: command not found`, sourced from `/p4/common/lib/utils.lib`). Manually
re-ran the exact same commands on the same machine immediately after and they worked fine
-- `show_versions` is defined correctly, `-V` prints the version as expected. Almost
certainly a timing/race artifact from Rocky 8 being the slowest platform in the fleet (the
user's own prior observation) combined with tests kicking off simultaneously across
platforms; not investigated further since it didn't reproduce and is unrelated to today's
fix.

## Third Emergency Bug Fix, same day: SDP-1396 into r26.1.0

User's call: "Let's done one more EBF! We're still (just barely) less than 24h from when
we first shipped last night." Applied the same procedure as the SDP-1394/1395 rounds,
following the now-documented order of operations:

1. **Fix** in `r26.1.0` (change 33663): same `opt_perforce_helix-sdp` correction to
   `opt_perforce_sdp_backup.sh` plus the 4 comment fixes in `mkdirs.sh`/`install_sdp.sh`.
2. **Release Notes** entry first (change 33664), calm wording matching the SDP-1394/1395
   style ("briefly referenced... corrected the same day").
3. **Man pages + HTML/PDF regen** (changes 33665 doc/gen + 33665 continued for
   `SDP_Guide.Unix.html`/`.pdf` -- man regen was needed this time, unlike SDP-1395, since
   `opt_perforce_sdp_backup.sh`'s `-man` usage text changed).
4. **Version bump** to 33665 (change 33666) -- matching the SDP-1394 precedent of tracking
   through the man-regen changelist when one was needed, rather than stopping at the bare
   code-fix number.
5. **Repackage** (change 33667, pending).
6. **Verify with a real test before publishing** -- and this caught a real process gotcha:
   the first verification attempt ran `install_sdp.sh` directly from the extracted local
   tarball's own tree, which does *not* use local files -- `install_sdp.sh` always fetches
   its own copy of the SDP tarball from `main`'s permanent download URL unless `-local`
   mode is used, so the first attempt actually re-verified the **still-buggy, not-yet-
   published** live tarball instead of the new one. Caught by noticing the backup still
   reported the wrong path immediately after "successful" install. Fixed by following the
   documented upgrade-test pattern instead: installed a same-network baseline, then
   staged the freshly-built local tarball directly into `/opt/perforce/helix-sdp/downloads/`
   and ran `sdp_upgrade.sh` against it (the same technique already proven for testing
   unpublished release candidates) -- confirmed live: backup correctly reports
   `/mnt/p4depots/backup/opt_perforce_helix-sdp.<host>`, and the upgrade itself succeeded
   cleanly (re-confirming SDP-1395 still holds too).
7. **Publish**: submitted the package (33667), republished to `main`'s permanent download
   URL (change 33668) following `ReleaseProcessOverview.md` Step 20's documented procedure
   exactly (remove old version-stamped files, add new ones `-t binary+F`, edit the two
   fixed-name files).
8. **Confirm live**: fresh `curl` of the real public download URL confirmed `Version`
   33665 and zero occurrences of the bad path.
9. **Merge Down all the way to `dev`**: `r26.1.0` -> `main` (temporarily flipped
   `notoparent`->`toparent` again, `p4 merge -S //p4-sdp/r26.1.0` from `main`'s client,
   4 files, 0 conflicts, flipped back; change 33670), man/doc regen in `main` (change
   33672), then `main` -> `dev` (plain `p4 merge`, 4 files, 0 conflicts; change 33673),
   plus `dev`'s own `ReleaseNotes.html` regen (HTML only, per dev policy; change 33674).

Updated SDP-1396 in JIRA: description and P4Blog now point at the `r26.1.0` fix (change
33663), matching the SDP-1394/1395 convention of citing the release-stream fix rather than
the `dev` change once one exists.

**User caught a real gap in the above: the Release Notes fix never actually shipped.**
Step 3's regen only touched `SDP_Guide.Unix.html`/`.pdf` -- a completely separate build
target from `ReleaseNotes.html`/`.pdf` (built from `ReleaseNotes.adoc` via the Makefile's
generic `.adoc` -> `.html`/`.pdf` rule, no relation to `SDP_Guide.Unix.adoc`). Confirmed via
`tar -tzf` on the already-published tarball: `doc/ReleaseNotes.html`/`.pdf` genuinely ship
inside it (nothing in `tools/excludes.downloads.cfg` excludes `doc/`), and the shipped copy
had zero occurrences of "SDP-1396" -- the Change History entry from step 2 never made it
into the actual generated output, let alone the published tarball.

Fixed properly: first attempt tried reserving a pending changelist number (`p4 change -i`)
so `ReleaseNotes.adoc`'s `Released:`/`Jobs Fixed up to` line could cite its own exact final
changelist number. **User flagged this as unsafe process guidance**, even though it
happened to work this time: Perforce only guarantees *submitted* changelist numbers
increase monotonically in submission order, so a pending changelist can be silently
renumbered at submit time if anything else submits on the server in between -- relying on a
reserved pending number in documentation is fragile by design, not just this once. Correct
lesson: accept the existing, already-established tolerance for a slightly-stale `Released:`
number (visible elsewhere in the same file's Change History), and instead fix the actual
bug -- regenerating the *right* file. Regenerated `ReleaseNotes.html`/`.pdf` for real
(change 33675, bundled with the `Released:`/`up to` line correction and a matching
`Version` bump, since the reserved number happened to match this time), repackaged (33676),
verified in the local tarball this time (`SDP-1396` present, `Released:` line matches,
zero bad-path occurrences), republished (33677), confirmed live via a fresh `curl`, then
re-ran the full Merge Down chain a second time for this correction (`r26.1.0` -> `main`
changes 33679/33681, `main` -> `dev` changes 33682/33683).

Updated `ReleaseProcessOverview.md` (dev change 33684, Copy Up to main 33685) with the two
real lessons: **(1)** `ReleaseNotes.html`/`.pdf` and `SDP_Guide.Unix.html`/`.pdf` are
separate regeneration targets and don't assume `doc/*.html`/`*.pdf` are excluded from the
tarball without checking `excludes.downloads.cfg`; **(2)** never reserve/reference a pending
changelist number in documentation -- accept approximate numbering instead. Also documented
the `install_sdp.sh`-fetches-its-own-copy-unless-`-local` verification gotcha hit while
testing this same round (see previous section) as a new "AI-specific gotcha" bullet.

## Open / carried forward

- **Non-JIRA task, explicitly deferred, not critical path**: ponder a policy for when an
  Emergency Bug Fix may update an already-shipped release in place (bump `Version`'s CL, add
  a Jobs Fixed entry) vs. requiring a real new patch release. User was fine doing it this way
  today specifically because the bugs were caught within 24 hours of GA; explicitly does not
  want this to become the default move for a fix discovered a month out. Candidate factors:
  how much changed, how severe the bug was, how long it took to be detected. Saved as a
  memory note (`project_emergency_fix_inplace_policy`), not yet acted on.
- **Old `install_sdp.sh` test fleet**: confirmed at least one machine
  (`ssh-install_sdp-test-r8x86_64.sh`) is a broker-only partial fixture, not a full `p4d`
  instance -- one of presumably several different ad hoc states across that fleet's
  machines, not yet inventoried.
- **`gen_sudoers.*.log` litter** on the `install_sdp.sh` test fleet (see above), not yet
  fixed -- user's call on `gen_sudoers.sh` self-cleanup vs. a `DANGER_CLEAN.sh` extension.
- ~~SDP-1396 not yet fixed in `r26.1.0`~~ -- **resolved tonight**: user approved a third
  same-day EBF; fixed, verified live, and Merge Downed through `main`/`dev` (see above,
  including the Release Notes regen-target correction caught by the user mid-flight).
- **`SDP_Guide.Unix.html`/`.pdf`, `TheSiteDirectory.html`, man-page regens**: all three
  streams were left in a clean, fully-submitted, mutually-consistent state as of the last
  commit tonight (`Released:`/`Version` all read 33675 in `r26.1.0`/`main`/`dev`) -- no
  known drift remaining to fix next time, but worth a quick `p4 diff -sa`-style sanity
  check at the start of the next session before assuming that's still true, per the usual
  P4-sync-mtime-reset caveat.

## Next session: starting points

- **Further testing**: the SDP-1396 fix itself was verified live (real install + real
  upgrade using the locally-built tarball, both on the EC2 fleet and via the published
  `main` URL) and the targeted `BackupRecoveryMultiInstance` re-run passed cleanly on
  Ubuntu 22 against the fixed `dev` tree -- **no further testing is strictly required**
  before considering this incident closed. That said, a fresh full Helix Install Test
  Suite run across the whole EC2 fleet (now that `dev` carries all of today's fixes,
  including the Release Notes correction) would be a reasonable, low-cost way to get final
  confidence before moving on to Patch 1 planning, since it hasn't been re-run end-to-end
  since the SDP-1396 fix landed in `dev` (only the one targeted group was re-verified).
  The Rocky 8 `show_versions: command not found` flake (Tests 12-15, non-reproducing) is
  worth a passive eye if it shows up again, but isn't worth chasing proactively.
- **SDP 2026.1 Patch 1 planning**: checked JIRA's fixVersion "2026.1 Patch 1" (id 46052)
  before writing this and found it's **not a curated list** -- it returns 15+ issues,
  including the 3 actually-intended-for-Patch-1 bugs from tonight (**SDP-1390** EPEL-
  dependent packages fail on stock Rocky/RHEL 9, **SDP-1391** empty
  `ExtraP4PackageList[yum]` produces a bogus yum command, **SDP-1393** `verify_sdp.sh`'s
  spurious `grep: off` warning with `-L off`), plus **SDP-1395 and SDP-1396** (which
  shouldn't still be tagged for it -- both already shipped same-day via Emergency Bug Fix,
  so their fixVersion is stale and should be cleared), plus a long tail of older, unrelated
  backlog items (SDP-1382, 1354, 1353, 1352, 1284, 1274, 1179, 1050, 1016, 889) that predate
  tonight and have apparently been using this fixVersion as a loose catch-all over time.
  **First real step next session: a genuine triage pass** to decide what actually belongs
  in Patch 1's scope before treating this fixVersion's contents as a plan, plus clearing
  the stale SDP-1395/1396 tags.
- **Non-JIRA tracked tasks to review** (see full list immediately above): the EBF
  in-place-vs-patch-release policy question (`project_emergency_fix_inplace_policy`
  memory), the old `install_sdp.sh` test fleet's ad hoc/uninventoried state, and the
  `gen_sudoers.*.log` litter on that fleet (`DANGER_CLEAN.sh` extension vs. script
  self-cleanup). None are urgent; good candidates to knock out or consciously defer again
  while Patch 1 scope is still being decided.