SDP-564

Details
Comments -

Status: Closed
Project: perforce-software-sdp
Severity: C
Reported By: C. Thomas Tyler
Reported Date
Modified By: swarm-user
Modified Date
Owned By: tom_tyler
Dev Notes: This job evolved.
Originally there was going to be a new /p4/Version
file, but this idea was dropped. The /p4/sdp/Version is clear enough.

The spirit of this job, which is to standardize and simplify SDP version
checking, is upheld. It was determined that adding yet antoher copy of
the Version file would not help with this.

This does mean the SDP 'tarball extraction' folder that is symlinked from
/p4/sdp is now required on all SDP machines. This is reasonable and it
does match how SDP is commonly deployed. There is a failure mode where
copying SDP from one machine to another (e.g. via rsync) might miss this,
however the verify_sdp.sh will catch that.
Component: setup
Type: Feature

27100	Eliminated extra /p4/Version file. The standard for determining the SDP version for 2020.1 and forward will be the file /p4/sdp/Version. This is more clear. Job SDP-564 evolved slightly (per DevNotes in the job). The goal remains the same, to standardize the method of determining the SDP version for SDP 2020.1+. It was deemed that having an extra copy in /p4/Version will not help with that, and instead would introduce more failure modes and possibilities for out-of-sync files. This does mean the 'tarball extraction' sdp folder that is symlinked to from /p4/sdp is now a critical part of the SDP installation. This is normal and as documented, though there have been cases where SDP is copied from one machine to another in some incomplete way, e.g. rysnc of /p4/common but not /hxdepots/sdp and the symlink to it from /p4/sdp. However, the verify_sdp.sh will catch that form of misconfiguration.
26962	mkdirs.sh v4.1.0: * Accounted for directory structure change of Maintenance to Unsupported. * Added standard command line processing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced '-test' mode to simulate /hx* mounts. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. * Added '-s <ServerID>' option to override REPLICA_ID. * Added '-S <TargetServerID>' used for replicas of edge servers. * Added '-t <server_type>' option to override SERVER_TYPE. * Added '-M' option to override mount points. * Added '-f' fast option to skip big chown/chmod commands, and moved those commands near the end as well. verify_sdp.sh v5.9.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab: Skip crontab check. Use this if you do not expect crontab to be configured, perhaps if a different scheduler is used. - license: Skip license related checks. - version: Skip version checks. - excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH. * Added VERIFY_SDP_SKIP_TEST_LIST setting ton instance_vars.template, to define a standard way to have verify_sdp.sh always skip certain tests for a site. * Extended '-online' checks to check for bogus P4MASTERPORT, a common config error. Update test_SDP.py: * Adjusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function. === Upgrade Process Changes === Made /p4/common/bin/p4d/p4/p4broker/p4p shell script rather than binary. This changes the way SDP new binaries are staged for upgrade. For safety, exes are now staged to a director outside the PATH, the /p4/sdp/exes folder. A new 'get_latest_exes.sh' script simplifies the task of pulling executables from the Perforce FTP server. This can be used 'as is' for environments with outbound internet access, and is useful in any case to describe now to acquire binaries. This addresses an issue where a p4d binary staged for a future upgrade might be called before the actual upgrade is performed. upgrade.sh v4.0.0: * All preflight checks are now done first. Added '-p' to abort after preflight. * Added '-n' to show what would be done before anything is executed. * Minimalist logic to start/stop only servers that are upgrade, and apply upgrades only as needed. * Staging of exes for upgrade is now separate from /p4/common/bin * Improved in-code docs, added '-h' and '-man' options. * Retained pre/post P4D 2019.1 upgrade logic.
26900	mkdirs.sh v4.0.0: * Added deployment of Version file to /p4/Version. A future upgrade_sdp.sh in the next SDP release will make use of this. * Accounted for directory structucre change of Maintenancd to Unsupported. * Added standard command line procssing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. verify_sdp.sh v5.7.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab - version - and license Update test_SDP.py: * Adusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function.
27099	Eliminated extra /p4/Version file. The standard for determining the SDP version for 2020.1 and forward will be the file /p4/sdp/Version. This is more clear. Job SDP-564 evolved slightly (per DevNotes in the job). The goal remains the same, to standardize the method of determining the SDP version for SDP 2020.1+. It was deemed that having an extra copy in /p4/Version will not help with that, and instead would introduce more failure modes and possibilities for out-of-sync files. This does mean the 'tarball extraction' sdp folder that is symlinked to from /p4/sdp is now a critical part of the SDP installation. This is normal and as documented, though there have been cases where SDP is copied from one machine to another in some incomplete way, e.g. rysnc of /p4/common but not /hxdepots/sdp and the symlink to it from /p4/sdp. However, the verify_sdp.sh will catch that form of misconfiguration. #review-27100
26982	mkdirs.sh v4.1.0: * Accounted for directory structure change of Maintenance to Unsupported. * Added standard command line processing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced '-test' mode to simulate /hx* mounts. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. * Added '-s <ServerID>' option to override REPLICA_ID. * Added '-S <TargetServerID>' used for replicas of edge servers. * Added '-t <server_type>' option to override SERVER_TYPE. * Added '-M' option to override mount points. * Added '-f' fast option to skip big chown/chmod commands, and moved those commands near the end as well. verify_sdp.sh v5.9.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab: Skip crontab check. Use this if you do not expect crontab to be configured, perhaps if a different scheduler is used. - license: Skip license related checks. - version: Skip version checks. - excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH. * Added VERIFY_SDP_SKIP_TEST_LIST setting ton instance_vars.template, to define a standard way to have verify_sdp.sh always skip certain tests for a site. * Extended '-online' checks to check for bogus P4MASTERPORT, a common config error. Update test_SDP.py: * Adjusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function. === Upgrade Process Changes === Made /p4/common/bin/p4d/p4/p4broker/p4p shell script rather than binary. This changes the way SDP new binaries are staged for upgrade. For safety, exes are now staged to a director outside the PATH, the /p4/sdp/exes folder. A new 'get_latest_exes.sh' script simplifies the task of pulling executables from the Perforce FTP server. This can be used 'as is' for environments with outbound internet access, and is useful in any case to describe now to acquire binaries. This addresses an issue where a p4d binary staged for a future upgrade might be called before the actual upgrade is performed. upgrade.sh v4.0.0: * All preflight checks are now done first. Added '-p' to abort after preflight. * Added '-n' to show what would be done before anything is executed. * Minimalist logic to start/stop only servers that are upgrade, and apply upgrades only as needed. * Staging of exes for upgrade is now separate from /p4/common/bin * Improved in-code docs, added '-h' and '-man' options. * Retained pre/post P4D 2019.1 upgrade logic.
26954	mkdirs.sh v4.1.0: * Accounted for directory structucre change of Maintenancd to Unsupported. * Added standard command line procssing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced '-test' mode to simulate /hx* mounts. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. * Added '-s <ServerID>' option to override REPLICA_ID. * Added '-S <TargetServerID>' used for replicas of edge servers. * Added '-t <server_type>' option to override SERVER_TYPE. * Added '-M' option to override mount points. * Added '-f' fast option to skip big chown/chmod commands, and moved those commands near the end as well. verify_sdp.sh v5.9.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab - version - license - version * Added VERIFY_SDP_SKIP_TEST_LIST setting ton instance_vars.template, to define a standard way to have verify_sdp.sh always skip certain tests for a site. * Extended '-online' checks to check for bogus P4MASTERPORT, a common config error. Update test_SDP.py: * Adusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function. === Upgrade Process Changes === Made /p4/common/bin/p4d/p4/p4broker/p4p shell script rather than binary. This changes the way SDP new binaries are staged for upgrade. For safety, exes are now staged to a director outside the PATH, the /p4/sdp/exes foler. A new 'get_latest_exes.sh' script simplifies the task of pulling executables from the Perforce FTP server. This can be used 'as is' for environments with outbound internet access, and is useful in any case to describe now to acquire binaries. This addresses an issue where a p4d binary staged for a future upgrade might be called before the actual upgrade is performed. upgrade.sh v4.0.0: * All preflight checks are now done first. Added '-p' to abort after preflight. * Added '-n' to show what would be done before anything is executed. * Minimalist logic to start/stop only servers that are upgrade, and apply upgrades only as needed. * Staging of exes for upgrade is now separate from /p4/common/bin * Improved in-code docs, added '-h' and '-man' options. * Retained pre/post P4D 2019.1 upgrade logic. #review-26962