# Pleasantries

Greetings, Agent!

## Task For Today

My first task for you is just to review the information and identify anything that's confusing or unclear up to TASK 1.3.  More detailed will be needed to go beyond 1.3, but that's the first milestone.

## Background Information - SDP Classic to streams Migration

We are in the process of migrating the P4 Server Deployment Package (SDP) from its current home in the "Classic" (non-Streams) legacy directory structure in the Public Depot (a P4 server publicly accessible) to a modern structure in a Stream depot.  The SDP 2025.2 Patch 1 release was shipped recently from the Classic depot structure in the Perforce Public Depot server (P4PORT=public.perforce.com:1666) in `//guest/perforce_software/sdp/...`. That will be the last SDP release to ship from that Classic directory structure. That entire structure is now locked down as read-only.  The "new world" is the //p4-sdp stream depot.  There is a //p4-sdp/dev stream that is to be a long-lived development stream. That is also a //p4-sdp/dev_rebrand stream, parented by //p4-sdp/dev, where lots of work is currently going on.  That branch will eventually go away just after we do a final Copy Up from //p4-sdp/dev_rebrand to //p4-sdp/dev.  The //p4-sdp/main steam is defined but unpopulated right now. A future //p4-sdp/r26.1 stream will be created for the SDP 2026.1 release when it ships (hopefully in only another week or two, but could take longer).

The branch spec SDP_Classic_to_Streams has a Description field which contains a draft plan for the SDP Classic to Streams migration.  Review that as a reference, but beware it may be a bit outdated.

## Background Information - The Planned SDP 2026.1 Release

The plan is to make a relatively small set of functional changes between SDP 2025.2 Patch 1 (currently released) and the new SDP 2026.1 release. There will be a lot of churn adapting to Streams and not a lot of time, so only a few bug fixes and feature changes will be done for SDP 2026.1. However, many files will be affected to a change in versioning style -- so even though it's not a complex change, most files in the package will have content updates due to this change.  We won't be implmenting these just yet, but for high-level context the changes that will make it into the SDP 2026.1 release are:

* SDP-1379 (Feature): Rebranding from Helix back to P4 for SDP.
* SDP-1350 (Feature): Adapt to P4D 2026.1 Secure By Default (SBD) initiative, including warning about behavior changes.
* SDP-1323 (Feature): Changed default P4 version in get_p4_binaries.sh to r26.1.
* SDP-1320 (Feature): New best practice: Set server.startup.autorestart=1 for P4D is 2026.1+.
* SDP-1286 (Feature): Update default auth.id=p4_<CommitServerID>; ServerID=commit.<OrgName>.<SDPInstance>.
* SDP-1170 (Feature): Migrate SDP from Classic to Streams: //guest/perforce_software/sdp/... -> //p4-sdp/...
* SDP-1161 (Feature): Replace SDP script version numbers with RCS Keywords.
* SDP-799 (Feature): Capture SDP version in the output of scripts.

Recap:
1. SDP 2025.2 Patch 1 shipped, it's the last release from Classic.
2. The SDP_Classic_to_Streams branch spec contains the plan.
3. The World awaits the SDP 2026.1 GA release, which will be the first to release from the new Streams structure.

While the set changes planned for 2026.1 is modest, there is a big impact on the three SDP test suites.

## Background Information - SDP Test Suites

There are 3 SDP test suites:

### Docker/Podman Test Suite

The Docker/Podman test suite is the original SDP test suite.

When you need more information on that, see: [SDPTestSuite-DockerPodman.md](SDPTestSuite-DockerPodman.md).

### Battle School Workshop (BSW) Test Suite

A standing and dedicated P4 Battle School Workshop (BSW) lab environment is used for executing some SDP tests.

When you need more information on that, see: [SDPTestSuite-BattleSchool.md](SDPTestSuite-BattleSchool.md).

### SDP Installer Test Suite

You'll want to read this background now; it'll be relevent for the working session today: [SDPTestSuite-SDPInstaller.md](SDPTestSuite-SDPInstaller.md).

## Task For Today

### TASK 1:  Formalize and Update the SDP Test Suite for install_sdp.sh

#### TASK 1.1: Launch EC2 Instances for a new Test Fleet.

There are 8 machines in the current fleet.  I'd like to essentially duplicate the fleet, but dropping the EOL Ubuntu 20 platform, so we'll have 7 new VMs.

I'd like you to create a fleet of 7 AWS EC2 instances.  When these were originally launched manually, it was slow and painstaking work to select an appropriate AMIs to represent each platform, using the AWS Console and searchig.  For this purpose, only zero-cost AMIs are acceptable (that is, zero additional licensing costs for software), with one exception: SLES 15 is exempt from the zero-cost requirement.  We'll keep reusing the current SLES 15 AMI even though its licensing cost isn't currently known -- SLES has few customers, but some are important ones (e.g. nuclear power plant operators), so keeping SLES on the supported platform list is worthwhile as long as it isn't too costly.  Determining the actual cost is a task for another day.  I have 2 ideas on how to short-circuit that effort this time:

1. Figure out what AMIs we're using for the current fleet, and simply re-use the same ones.  If it helps to know the EC2 instance ID, you can get that by logging into a machine using its ssh wrapper script and running the command 'ec2id'.
2. Perhaps there's some kind of standard list of "best AMI to use for this vanilla OS testing" we could use?
3. I'm open to any suggestions you might have for selecting AMIs.

We generally use the 't' (burstable) instance family at the 'small' size -- 'small' rather than 'micro', since we've had reliability/throttling issues with 'micro'.  Use the latest generation available (t3, t4, t5, t6, etc. -- whichever is newest), since AWS generally improves price/performance with newer generations.  For aarch64 platforms, use the 'g' suffix (e.g. t4g.small).  Cost matters, but it's not the top priority -- usability and reliability come first.

We'll have a whole new set of ssh scripts, for this, so I want to create them with a slightly different naming convention to easily separate them from current ones.

The current pattern is: ssh-install_sdp-test[<N>]-<PlatformTag>.sh

Let's use this for the new scripts: ssh-test-install_sdp-<PlatformTag>.sh

Since we'll have new EC2 instances, we won't need the <N> numbers. (Those may appear down the road if we need to abandon machines for whatever reason).

When this task is done, we'll have a bunch of new EC2 instances with nothing on them except the vanilla OS install, and a set of ssh wrapper scripts to get to them easily.

The local directory where the current pem files and ssh wrapper scripts live, /Users/ttyler/ppn/BattleSchool/aws/pem/us-east-2, is itself a P4 workspace mapped to the //BattleSchool/main/aws/pem/us-east-2 folder on the PPN server.  Once we've created the new ssh wrapper scripts and confirmed they work to access the new EC2 instances, we'll need to add and submit them in that P4 workspace too.

#### TASK 1.2: Update the SDP Installer Test Suite

The SDP Installer Test Suite is currently versioned here:

//guest/tom_tyler/sw/main/install_sdp/dev/bin/...

That software path will now be frozen and deprecated, in part because it pulls the SDP from Classic Depot paths, and SDP is moving to its new home.  Likewise, the SDP Installer Test suite is moving to a new home.  These days the SDP Installer Test Suite is essential to the SDP release process, and I feel the time has come to move it to a more official location than my personal area in the Public Depot server.  To that end, I created a new //test-install_sdp stream depot.  This depot has a //test-install_sdp/main stream (currently empty) and a //test-install_sdp/dev stream.

At this moment as we are starting this project, //test-install_sdp/dev/... was populated with //guest/tom_tyler/sw/main/install_sdp/dev/bin/..., like this:

   p4 populate -b test-install_sdp_Classic_to_Streams -d "Initialized //test-install_sdp/dev with p4 populate -b test-install_sdp_Classic_to_Streams"

This resulted in change @33144.

Some things we'll need to change:

* We'll need to change scripts and docs to pull the SDP Installer Test Suite software from the new home in the Streams depot.

* We'll need a new capability: "Source Swapping"

Currently, the SDP Installer Test Suite software pulls both itself and the SDP from fixed sources in version control, i.e. fixed P4 paths.  The SDP is pulled from it's Classic dev branch (//guest/perforce_software/sdp/dev/...) and the SDP Installer Test Suite is pulled from the obscure path //guest/tom_tyler/sw/main/install_sdp/dev/bin/...  I want to make it so that, once a machine is setup, it is straightforward to change the source for each.  For the SDP, the user could specify any stream in the //p4-sdp depot, e.g. //p4-sdp/main, //p4-sdp/dev, //p4-sdp/dev_rebrand.  For the SDP Installer Test Suite, it could be pulled from //test-install_sdp/dev or //test-install_sdp/main.

For initial configuraiton, we'll pull the SDP Installer Test Suite from //test-install_sdp/dev (main isn't populated yet!), and we'll pull SDP from //p4-sdp/dev_rebrand (because, as of this writing, the action is happening in dev_rebrand rather than dev).  (For background //p4-sdp/main is the mainlne, //p4-sdp/dev is parented by main, and //p4-sdp/dev_rebrand is parented by dev.)

I want it to be very easy change the SDP Installer Test Suite to change the source it pulls from.  That would require a "re-clone", and we'd need to move the initially cloned local directories on the EC2 instances aside, but be able to move them back in place shoudl the user want to swap back and forth.

The initally cloned local directories on the machines are:

For SDP: /opt/perforce/dev/sdp
For SDP Installer Test Suite: /opt/perforce/dev/bin

If we do a swap and move them aside, we should move them to something that includes a short form of the source stream, e.g. something like:

/opt/perforce/dev/sdp.p4-sdp_dev
/opt/perforce/dev/sdp.p4-sdp_dev_rebrand

/opt/perforce/dev/bin.test-install_sdp_dev
/opt/perforce/dev/bin.test-install_sdp_main

So the '/opt/perforce/dev/sdp' and '/opt/perforce/dev/bin' paths will be the default/current and actively used sources. If source swapping occurs, the swapped-out paths will be those above.  If the user swaps back, those paths would be moved back into place after the current folder is moved aside to a path indicate what it's source was.

A key idea here is that the initial 'p4 clone' command and subsequent 'p4 fetch' and 'p4 push' commands rely on state infomration contained in the local repo structure.  If we switch the source, we need to keep all that intact for a possible switch back.  If the user switches to something that hasn't existed before on this particulare VM, then it'll just do a fresh 'p4 clone' with to pull from the new source.

Implementing this capability will require a good understanding of P4-native DVCS features, e.g 'p4 clone', 'p4 fetch', 'p4 push', etc, as well as how they're used and reference in the script and docs.  The current cloning mechanism uses fixed paths; we might want to add remote specs to make things easier and less error-prone for users, e.g. 'p4 clone -p public.perforce.com:1666 -r <RemoteSpecWeSpecify>' instead of '-f <//somePath/...>'.

#### TASK 1.3: Install the updated SDP Installer Test Suite on the fleet.

After we've updated, we'll install the updated software in the new fleet server machines.  (We may neeed to iterate 1.2 and 1.3...).

### TASK 2: Prepare SDP-QA-Gen7

The SDP-QA_Gen6 lab environment has served well, but it is tied to the old ways.

We need to prepare the SDP-QA-Gen7.  The background information in that section will be useful when we get to that task, but lots more informaiton will be needed, like how to access lab environments and pull from various version control sources once in the environments.  That taks is still down the road.

### TASK 3: Upgrade the Docker/Podman Test Suite

We'll need to setup new Jenkins builds and do some retooling here.

### TASK 4: Implement the SDP 2026.1 Features.

We have the list of JIRA issues above. We'll of course need a lot more detail to get those implemented.

### TASK 5: Redesign the SDP Reelase Process.

The SDP Release Process will need to be updated.

The current file in the //p4-sdp/dev stream that describes the SDP Release Process is outdated -- it describes how the release process worked when SDP was in the Classic structure.

The file is here: //p4-sdp/dev/tools/ReleaseProcessOverview.md

That will need to be reviewed and overhauled. In particular, now that we'll be in streams, we'll take advantage of Streams workflows and features.

## Let's Get Started

Reminder: My first task for you is just to review the information and identify anything that's confusing or unclear up to TASK 1.3.