Release Notes for SDP - Helix Server Deployment Package Helix Server Components Introduction This document notes changes to the SDP. There are two flavors of the SDP, Unix and Windows. At a high level, the core function of the SDP is similar on Unix and Windows: To maintain an offline copy of the Helix databases, and perform routine offline checkpoints in preparation for backups. At a detailed level, there are differences in both implementation and behavior on the various platforms. Enhancements are needs-driven, and not necessarily applied across Unix and Windows platforms. -------------------------------------------------------------------------- Upgrading the SDP on Unix To incorporate a new version of the SDP into an existing SDP installation, first stage the deployment as in this example, assuming you have put the SDP distribution tar file in /hxdepots. Working as user root on a Perforce server: cd /hxdepots # If an sdp folder exists here, move it aside, e.g. to # sdp.MOVED_2014-09-11 tar -xzpf sdp.Unix.2014.2.10649.tgz chown -R perforce:perforce sdp.Unix.2014.2.10649.tgz /hxdepots/sdp cd /hxdepots/sdp/Server/Unix/p4/common/bin # Copy 'p4', 'p4d', 'p4broker', 'p4web', 'p4p', etc. (only # those currently used) into this 'bin' folder. cd /hxdepots/sdp/Server/Unix/setup cp mkdirs.cfg mkdirs.cfg.orig vi mkdirs.cfg # Review and adjustments to mkdirs.sh suit your environment. ./mkdirs.sh 1 -test 2>&1 | tee mkdirs.1.log Using 'mkdirs.sh' script with the '-test' flag as illustrated above will deploy to the SDP to /tmp/p4 rather than /p4 area. If working on the Production server directly, DO NOT forget to use the '-test' flag. -------------------------------------------------------------------------- Upgrading the SDP on Windows: Install the SDP onto a different machine - configure sdp_master_config.ini - run create_env.py Manually copy resulting files to the appropriate places in your existing installation. -------------------------------------------------------------------------- Versioning the SDP: We strongly recommend that files in the following locations are versioned in a local Helix server: On Unix: /p4/1/bin /p4/common On Windows: C:\p4\1\bin C:\p4\common C:\p4\config NEW: Helix DVCS and Streams features enable a new method for updating the SDP long-term, greatly simplifying the process of bringing in updates from The Workshop into your environment, and then merging into your local as-deployed SDP (locally configured and possibly customized). Once configured, it entails a straightforward 'p4 fetch' and 'p4 merge' process. For more information, see the hms_ts_setup.sh ship documentation. This documentation is not complete as of this release, and is targeted for new deployments. While the concepts can be retrofitted into any existing SDP installation, some custom procedures are required, as the scripts do not account for evolution of the SDP structure. Contact consulting@perforce.com for more information. -------------------------------------------------------------------------- CHANGE HISTORY: Released: Rev. SDP/MultiArch/2019.2/25938 (2019/08/05) Jobs Fixed since change 25923 up to 25938 (F=Feature, B=Bug): SDP-414 (B): Fixed issue with safety check in edge processing scripts. Released: Rev. SDP/MultiArch/2019.2/25923 (2019/08/05) Jobs Fixed since change 25594 up to 25923 (F=Feature, B=Bug): SDP-398 (F): Disallow replicas from being initially mandatory per P4D 2019.1 requirement. SDP-400 (B): Fixed p4login bug where P4PORT value can have a value of Unset. SDP-401 (F): Enanced replica_status.sh to check archive replication. SDP-403 (B): Fixed root-owned symlinks in /p4/N after running mkdirs.sh. SDP-404 (F): Update edge scripts to support 2019.1+ changes to edge servers. SDP-405 (F): Enhanced load_checkpoint.sh to handle case-insensitive checkpoints. SDP-410 (B): [R]CMDEXITCODE not set when [r]run() called with 'grep string' parameter. SDP-413 (F): Add script to request checkpoint on edge/replica. Released: Rev. SDP/MultiArch/2019.2/25594 (2019/05/02) Jobs Fixed since change 25480 up to 25594 (F=Feature, B=Bug): SDP-356 (F): Reduce SDP Core to contain only software supported by Perforce Support. SDP-369 (F): Added flags to p4verify.sh to verify by type like shelved. SDP-374 (B): Fixed off-by-three-orders-of-magnitude error in proxy cleanup script. SDP-378 (F): Removed journalPrefix as command line paramter during journal rotation, deferring to db.config values. SDP-380 (F): Removed journalPrefix as command line paramter during journal rotation, deferring to db.config values. SDP-383 (F): Added '-a' flag meant to be equivalent of most common set of flags. SDP-384 (B): Fixed failure to cleanup old logs in sdp_sync.sh on Ubuntu Linux. SDP-392 (B): upgrade.sh must handle new 19.1 features SDP-393 (B): Fixed perms on purge_revisions.sh and journal_watch.sh. SDP-394 (F): Added support for wildcards/globbing to purge_revisions.sh scripts. SDP-395 (F): Added support for 2019.1 upgrades and testing of same. Released: Rev. SDP/MultiArch/2019.1/25480 (2019/04/11) Jobs Fixed since change 25386 up to 25480 (F=Feature, B=Bug): SDP-382 (B): Fixed missing 'exit 1' in p4d_base after 'p4d -xvU' check failure. SDP-387 (F): Test harness should run verify_sdp.sh and look for errors. SDP-390 (B): Removed crontab refs to deleted scripts. SDP-391 (B): In mkdir.sh, fixed root ownership of generated crontab file. Released: Rev. SDP/MultiArch/2019.1/25386 (2019/03/21) Jobs Fixed since change 25374 up to 25386 (F=Feature, B=Bug): SDP-379 (B): Fixed mkdirs.sh issue with SSL-enabled installs due to chown issue. Released: Rev. SDP/MultiArch/2019.1/25374 (2019/03/21) Jobs Fixed since change 25320 up to 25374 (F=Feature, B=Bug): SDP-302 (F): Parallelized checkpoint processing to reduce duration. SDP-375 (B): Fix ownership and missing symlink issues in mkdirs.sh. Released: Rev. SDP/MultiArch/2019.1/25320 (2019/03/08) Jobs Fixed since change 25309 up to 25320 (F=Feature, B=Bug): SDP-372 (B): Fixed breaking test - logic around $CN was misunderstood. Released: Rev. SDP/MultiArch/2019.1/25309 (2019/03/07) Jobs Fixed since change 25276 up to 25309 (F=Feature, B=Bug): SDP-370 (B): Fix issue with mkdirs.sh -test mode. SDP-371 (B): Fixed install_sdp_perl to work with updated P4Perl. Released: Rev. SDP/MultiArch/2019.1/25276 (2019/03/06) Jobs Fixed since change 25238 up to 25276 (F=Feature, B=Bug): SDP-368 (B): Fix syntax errors for SDP_ADMIN_PASSWORD_FILE in backup_functions.sh. Released: Rev. SDP/MultiArch/2019.1/25238 (2019/03/02) Jobs Fixed since change 23583 up to 25238 (F=Feature, B=Bug): SDP-198 (B): DOC: SDP Guide needs more detail on SDP-controlled replica setup. SDP-247 (B): Need a validation script to check current status, especially after an upgrade from older version of SDP. SDP-255 (B): Update SDP documentation to deal with SSL enabled servers. SDP-256 (B): Update documentation to include mkdirs.cfg instead of updating mkdirs.sh SDP-269 (F): Optimize journalPrefix values for master, edge, standby, replica. SDP-296 (B): Fixed bug where updates to os_tweaks.sh dropped support for RHEL 6.5. SDP-297 (B): Fixed handling of unlaod depots in p4verify.sh SDP-298 (B): Avoid checking for shelved files in spec depot. SDP-299 (F): Enhanced HMS failover for standby/journalcopy replicas to use 'p4 admin end-journal' SDP-300 (F): Set rpl.journalcopy.location=1 for p4d 2018.1+ servers. SDP-301 (B): Crontab spamming emails if configurable track=1 SDP-304 (B): Fixed issues with hms status for SSL-enabled servers, etc. SDP-305 (F): Enhancemnts in libcore.sh bail(), errmsg(), initlog(), and stoplog() functions. SDP-306 (B): Moved cset of P4JOURNAL so it's done only on masteror edge server. SDP-307 (F): Changed hms remote spec for the Public Depot to be the default 'origin'. SDP-308 (F): Set rpl.journalcopy.location=1 for 2018.1+ standby replicas. SDP-311 (B): Avoid unnecessary removal of rdb.lbr in db swap. SDP-313 (F): Batch of validate_sdp.sh enhancements. SDP-314 (B): Fixed scary but otherwise harmless error in Cmd.pm SDP-315 (F): Reformat protections table comment format during upgrades. SDP-316 (F): Added script to rotate live journal due low disk space SDP-317 (B): Fixed issue preventing failover from an edge servder. SDP-318 (B): Removed HMS references to /usr/local/bin/bash. SDP-319 (F): Use 'p4 monitor terminate' rather than 'kill' for idle pids. SDP-325 (F): Enhanced rejectList setting for P4EXP. SDP-326 (F): Enhanced rejectList setting for P4EXP. SDP-327 (B): DOC: must I create /hx*? SDP-328 (B): DOC: mismatch between setup instructions and setup script SDP-329 (B): DOC: folder permissions SDP-330 (B): DOC: to where should we copy the tgz file? SDP-331 (B): DOC: lack of clarity on folder ownership. SDP-332 (B): DOC: setting executable bit on p4 etc. SDP-333 (B): DOC: reorder the create user/group sequence on Linux. SDP-334 (B): DOC: no comment on mkdirs.N.cfg, referenced in the mkdirs.sh script. SDP-335 (B): DOC: no comment on ssl certificate generation. SDP-337 (B): The upgrade.sh script, used to update your p4/p4d executables and corresponding database, starts with the message: SDP-338 (F): DOC: server upgrades and checkpoint SDP-339 (B): DOC: non-existent script referenced in docs SDP-341 (B): Critical recreate_db_checkpoint.sh bug with shared /hxdepots shared. SDP-342 (B): Fixed issue with P4Triggers.py impacting CheckCaseTrigger.py, etc. SDP-345 (B): Implemented '-r' flag to clean up old logs in global_replica_status.sh. SDP-347 (F): Enhance CaseCheckTrigger to work even if defaultChangeType is set to restricted. SDP-348 (F): Enhanced CheckCaseTrigger.py to enable bypass. SDP-350 (F): Support SELinux ("Security Enhanced Linux)") in enforcing mode. SDP-353 (F): Avoid doing 'p4d -cset' for P4JOURNAL, but preserve safety feature. SDP-362 (B): p4verify.sh -remote ignores SDP_RECENT_CHANGES_TO_VERIFY. SDP-366 (F): Optimize display of Support messaging. SDP-367 (B): p4verify.sh should also specify the -t (transfer) option for edge servers like it currently does for replicas. SDP-49 (B): SDP Guide: inconsistent user name used in setup instructions. Page 9 of the pdf, step 1 says create user 'p4admin'. The rest of the guide that I've read, and the mkdirs.sh script, all refer to user 'perforce'. SDP-73 (F): Add Evil Twin detector/prevention trigger. Released: Rev. SDP/MultiArch/2018.1/23583.p2 (2019/01/23) Jobs Fixed since change 23583.p1 up to 23583.p2 (F=Feature, B=Bug): SDP-366 (F): Optimize display of Support messaging. SDP-341 (B): Critical recreate_db_sync_replica.sh bug with shared /hxdepots shared. Released: Rev. SDP/MultiArch/2018.1/23583.p1 (2018/07/13) PATCH release fixes SDP-341. SDP-341 (B): Critical recreate_db_checkpoint.sh bug with shared /hxdepots shared. Released: Rev. SDP/MultiArch/2018.1/23583 (2018/02/08) Jobs Fixed since change 23504 up to 23583 (F=Feature, B=Bug): SDP-287 (B): Fixed mkdirs.sh bug fixing ownership of wrong dir. 2nd fx for SDP-287. Released: Rev. SDP/MultiArch/2018.1/23504 (2018/01/19) Jobs Fixed since change 23354 up to 23504 (F=Feature, B=Bug): SDP-285 (B): The use of "tee" in p4d_base causes start to hang on some systems with 2016.2 p4d SDP-286 (B): Fixed: In configure_new_server.*, super user not logged in after restart. SDP-287 (B): Fixed mkdirs.sh bug fixing ownership of wrong dir. SDP-290 (F): Added SDP-standard Ruby to PATH. SDP-291 (F): Added workflow support triggers. SDP-292 (F): Enhancements to p4dstate.sh. Released: Rev. SDP/MultiArch/2017.4/23354 (2017/12/08) Jobs Fixed since change 23329 up to 23354 (F=Feature, B=Bug): SDP-281 (B): Fixed journalPrefix logic in mkdirs.sh Released: Rev. SDP/MultiArch/2017.4/23329 (2017/12/05) Jobs Fixed since change 23041 up to 23329 (F=Feature, B=Bug): SDP-265 (B): sync_replica.sh and recreate_db_sync_replica.sh should avoid rsync compression SDP-266 (B): Fixed logging issue with sync_replica.sh and recreate_db_sync_replica.sh. SDP-267 (B): Fixed issue detecting an edge server; wrong EDGESERVER value. SDP-268 (F): Added basic comments to all functions in backup_functions.sh. SDP-269 (F): Optimize journalPrefix values for master, edge, standby, replica. SDP-270 (F): Added trigger script to create Swarm reviews (and unit test). SDP-271 (F): Refactor to push common arg_parse into P4Triggers. SDP-272 (F): Enhance p4login info in error output, added -d debug flag. SDP-273 (F): New RequireJob pre-submit trigger. SDP-274 (F): Added trigger to validate contents YAML files. SDP-276 (B): Make mkrep.sh work when /p4/hms instance does not exist. SDP-277 (F): Added license time remaining check script. SDP-278 (F): Enhance upgrade.sh run on machines with only broker/proxy. SDP-279 (F): Added filetype.bypasslock=1 configurable, recommended for Swarm. SDP-280 (F): Updated Support coverage info and enhanced other info in SDP README file. Released: Rev. SDP/MultiArch/2017.3/23041 (2017/10/24) Jobs Fixed since change 23003 up to 23041 (F=Feature, B=Bug): SDP-143 (F): Enhance init scripts to display usage with no args. SDP-259 (B): Fixed logging issue in upgrade.sh - p4broker/p4p start/stop not captured. SDP-260 (F): Simplified stop_p4d() in backup function due to logic now in p4d_base. SDP-261 (B): Fixed minor issue message about rotating journal escaping log capture. SDP-262 (F): Take advantage of '--daemonsafe' arg in P4D 2017.1. SDP-263 (B): Fixed login count in p4login script. Released: Rev. SDP/MultiArch/2017.3/23003 (2017/10/19) Jobs Fixed since change 22201 up to 23003 (F=Feature, B=Bug): SDP-223 (B): Fix EDGESERVER check in p4d_base regression. SDP-224 (B): Fix os_tweaks.sh to work on RHEL/CentOS 6.9. SDP-225 (B): Updated hms_ts_setup.sh to adapt to mkdirs.cfg changes. SDP-226 (B): Fixed typo in variable name in libcore.sh. SDP-227 (B): install_sdp_perl.sh: Added DVCS environment isolation safety feature. SDP-228 (F): install_sdp_perl.sh: Now uses r17.1 API. SDP-232 (F): Re-establish autologinprompt default behavior. SDP-233 (B): Fix bug in sdp_sync.sh related to args passed to rrun function. SDP-234 (B): Change Unix default server.id to be unique to support remote depots SDP-236 (B): Avoid issues interacting with incompatible local Perl. SDP-237 (B): Scripts using libcore.sh display a harmless but scary error after all processing completes. SDP-239 (F): Renamed crontab templates files to be template.job.p4s rather than *.template. SDP-240 (B): Fixed cosmetic logging bug affecting several scripts. SDP-241 (B): Update systemd templates to add new wait targets. SDP-242 (B): Corrected filetypes on some scripts to chmod +x. SDP-248 (F): Provide an override for diff between proxy listen and target ports. SDP-249 (F): Extended P4IGNORE for more Mac files. SDP-250 (F): Add value for db.monitor.shared configurable. SDP-251 (F): Tweaked versions for which net.tcpsize=0 is set. SDP-252 (B): SetWsOptionsAndView.py trigger should exit 0 when p4 client used w/ -t