Release Notes for SDP - Helix Server Deployment Package Helix Server Components Introduction This document notes changes to the SDP. There are two flavors of the SDP, Unix and Windows. At a high level, the core function of the SDP is similar on Unix and Windows: To maintain an offline copy of the Helix databases, and perform routine offline checkpoints in preparation for backups. At a detailed level, there are differences in both implementation and behavior on the various platforms. Enhancements are needs-driven, and not necessarily applied across Unix and Windows platforms. -------------------------------------------------------------------------- Upgrading the SDP on Unix To incorporate a new version of the SDP into an existing SDP installation, first stage the deployment as in this example, assuming you have put the SDP distribution tar file in /depotdata. Working as user root on a Perforce server: cd /depotdata # If an sdp folder exists here, move it aside, e.g. to # sdp.MOVED_2014-09-11 tar -xzpf sdp.Unix.2014.2.10649.tgz chown -R perforce:perforce sdp.Unix.2014.2.10649.tgz /depotdata/sdp cd /depotdata/sdp/Server/Unix/p4/common/bin # Copy 'p4', 'p4d', 'p4broker', 'p4web', 'p4p', etc. (only # those currently used) into this 'bin' folder. cd /depotdata/sdp/Server/Unix/setup vi mkdirs.sh # Review and adjustments to mkdirs.sh suit your environment. ./mkdirs.sh 1 -test Using 'mkdirs.sh' script with the '-test' flag as illustrated above will deploy to the SDP to /tmp/p4 rather than /p4 area. If working on the Production server directly, DO NOT forget to use the '-test' flag. -------------------------------------------------------------------------- Upgrading the SDP on Windows: Install the SDP onto a different machine - configure sdp_master_config.ini - run create_env.py Manually copy resulting files to the appropriate places in your existing installation. -------------------------------------------------------------------------- Versioning the SDP: We strongly recommend that files in the following locations are versioned in your local Helix server: On Unix: /p4/1/bin /p4/common On Windows: C:\p4\1\bin C:\p4\common C:\p4\config NEW: Helix DVCS and Streams features enable a new method for updating the SDP long-term, greatly simplifying the process of bringing in updates from The Workshop into your environment, and then merging into your local as-deployed SDP (locally configured and possibly customized). It entails a straightforward 'p4 fetch' and 'p4 merge' process. For more information, see the hms_ts_setup.sh ship documentation. This documentation is not complete as of this release, and is targeted for new deployments. While the concepts can be retrofitted into any existing SDP installation, some custom procedures are required, as the scripts do not account for evolution of the SDP structure. Contact consulting@perforce.com for more information. -------------------------------------------------------------------------- CHANGE HISTORY: Released: Rev. SDP/MultiArch/2016.2/21489 (2017/01/11) Jobs Fixed since change 21480 up to 21489 (F=Feature, B=Bug): job000677 (B): Added Suffix field to spec depot in new servers. Released: Rev. SDP/MultiArch/2016.2/21480 (2017/01/11) Jobs Fixed since change 21379 up to 21480 (F=Feature, B=Bug): job000670 (F): Added server.start.unlicensed configurable. job000671 (F): Work around RHEL/CentOS 7 command completion issue. job000672 (B): Fixed install_sdp_python.sh issue with building P4Python for SSL-enabled server. job000673 (F): Added script to lowercase a directory tree. job000674 (F): Added SSL support in hms_ts_setup.sh. job000675 (B): Fixed bug in upgrade.sh introduced when truncate journal was changed to use p4 admin journal. job000676 (F): Enhanced SSL support in install_sdp_python.sh to account for varied OpenSSL installations. Released: Rev. SDP/MultiArch/2016.2/21379 (2016/12/20) Jobs Fixed since change 21369 up to 21379 (F=Feature, B=Bug): Released: Rev. SDP/MultiArch/2016.2/21369 (2016/12/19) Jobs Fixed since change 21328 up to 21369 (F=Feature, B=Bug): job000668 (F): Simplified SSL augoten cert generation. job000669 (B): Re-login after setting auth.id, restart at end of script. Released: Rev. SDP/MultiArch/2016.2/21328 (2016/12/16) Jobs Fixed since change 21239 up to 21328 (F=Feature, B=Bug): job000660 (F): Added sample /Users/perforce/.mailrc. job000661 (F): SDP-ified log, moving p4triggers.log to ${LOGS}. job000662 (F): Updated to hms tight ship setup script, still WIP. job000664 (F): Added script to lowercase a dir tree. job000665 (F): Added initial HMS tight ship script. job000666 (F): Standardized doc format to docx. job000667 (F): Enhanced to support enterprise environments with two metadata volumes. Released: Rev. SDP/MultiArch/2016.2/21239 (2016/12/06) Jobs Fixed since change 21193 up to 21239 (F=Feature, B=Bug): job000657 (F): Enable auth.id for P4D 2016.1+ servers. job000658 (F): Suppress known-harmless errors in p4verify.sh. job000659 (F): Enhance install_sdp_p* scripts to remove hard-coded dependency on an instance 1. Released: Rev. SDP/MultiArch/2016.2/21193 (2016/12/2) job000638 (F): Take advantage of new 2016.2 configurables. job000649 (F): Add CentOS 7 support to os_tweaks.sh job000650 (F): Update SDP Python/Perl install to use 'http:' rather than 'ftp:' in URLs. job000652 (B): Correct cp path for ssl dir. job000653 (F): Enhance Windows SDP install to server.depot.root. job000654 (F): Enhance to support 2 volumes for root/offline_db. Modified directory structure and scripts to support storing root and offline database on two different volumes. This is actually a preferred configuration now for performance and safety. Released: Rev. SDP/MultiArch/2016.2/21123 (2016/11/22) Jobs Fixed since change 21103 up to 21123 (F=Feature, B=Bug): job000651 (B): Fix shebang line for case check trigger. job000652 (B): Correct cp path for ssl dir. Released: Rev. SDP/MultiArch/2016.2/21103 (2016/11/21) Jobs Fixed since change 21101 up to 21103 (F=Feature, B=Bug): job000648 (F): Added doc-only hms setup scripts. Released: Rev. SDP/MultiArch/2016.2/21101 (2016/11/21) Jobs Fixed since change 21033 up to 21101 (F=Feature, B=Bug): job000646 (F): Add CheckCaseTrigger.p to the SDP. job000647 (F): Various doc update for hms and failover. Released: Rev. SDP/MultiArch/2016.2/21033 (2016/11/10) Jobs Fixed since change 20995 up to 21033 (F=Feature, B=Bug): job000639 (B): mkrep.sh: Add missing journalcopy to standby replicas. job000640 (B): hms: Fixed issues with script hang on local failover. job000641 (F): hms: Replaced hard-coded master ServerID value with configured value. job000643 (B): Fix p4verify.sh bug when -recent flag is used. Released: Rev. SDP/MultiArch/2016.2/20995 (2016/11/07) Jobs Fixed since change 20972 up to 20995 (F=Feature, B=Bug): job000634 (B): p4verify.sh is currently broken - use of an unbounded variable ($d); use of the -recent option fails as for verify -S you cannot specify a range (only @= is supported) ; stripping out archive,unload+remote depots also not working. get_verofy_rev_range used depot name incorrectly in changes job000636 (B): Tweaked maint scripts to handle oddly named groups. job000637 (F): Added new CN setting for /p4/common to test folder paths. Released: Rev. SDP/MultiArch/2016.2/20972 (2016/11/01) Jobs Fixed since change 20856 up to 20970 (F=Feature, B=Bug): job000616 (F): Add -recent flag to p4verify.sh to verify recent changes. job000626 (B): p4login.*.log files are not being cleaned up by the standard clean-up mechanism (remove_old_logs in backup_function.sh). job000627 (F): Allow for multiple depot volumes. job000628 (B): Fixed missing LOGFILE varialbe issue in p4verify.sh. job000629 (B): Fix formatting of logs to use simple HTML line breaks for Windows SDP. job000630 (B): Various test suite improvements. job000631 (B): fixed issue with broken def'n of P4PORT with multiple SSL prefixes. job000632 (F): Enhanced two-factor auth mechanims. job000633 (F): Tweak to slightly reduce mkdirs.sh configuration. Released: Rev. SDP/MultiArch/2016.2/20856 (2016/10/04) Jobs Fixed since change 20805 up to 20856 (F=Feature, B=Bug): job000618 (F): Enhanced logic to favor init scripts in /p4/N/bin over /etc/init.d. job000619 (B): Fixed quoting bug in p4login. job000620 (F): hms checks for cfg files in /p4/common/config. job000621 (B): Fixed hms bug re: wrong master host discovered in failback testing Released: Rev. SDP/MultiArch/2016.2/20805 (2016/10/02) Jobs Fixed since change 20790 up to 20805 (F=Feature, B=Bug): job000611 (B): Fixed ssh hang issues with hms scripts. job000612 (F): Implemented global/central start stop. job000613 (F): mkrep.sh: Added more replica types and journalcopy support. job000615 (F): Implement checkauth feature in HMS broker_wrapper. Released: Rev. SDP/MultiArch/2016.2/20790 (2016/09/30) Jobs Fixed since change 20755 up to 20790 (F=Feature, B=Bug): job000606 (B): Minor p4 login fix to show P4TICKETS output reliably. job000608 (B): Fix mkrep.sh to use $P4BIN instead of p4. job000609 (F): Added basic p4login-super.sh script. job000610 (B): Added hms dir to mkdirs.sh. Released: Rev. SDP/MultiArch/2016.2/20755 (2016/09/29) Jobs Fixed since change 20559 up to 20755 (F=Feature, B=Bug): job000545 (F): Add option to install_sdp_python.sh to build P4Python with SSL. job000595 (B): Fix attempted login count in p4login. job000597 (B): Fixed order-of-operations issue that caused some log files to be lost on rotation. job000598 (B): Cosmetic fix to broker init script with 'status all' run w/no config files. job000600 (F): Add Helix Management System (HMS) files to SDP. job000601 (F): Enhance p4login to avoid hang issue with auth.id set. job000602 (F): Further test suite docker environment updates. job000603 (F): Add Two Factor Authentication sample auth script. job000604 (B): Correct crontab to run sync_replica.sh every day. job000605 (B): Fix bugs in convert_label_to_autoreload.py. Jobs Fixed since change 20491 up to 20559 (F=Feature, B=Bug): job000590 (B): Fix recent regression with 'restart' option in init scripts. Released: Rev. SDP/MultiArch/2016.1/20491 (2016/09/09) Jobs Fixed since change 20460 up to 20491 (F=Feature, B=Bug): job000589 (B): Fix special case env insulation bug running init scripts as root. Released: Rev. SDP/MultiArch/2016.1/20460 (2016/09/08) Jobs Fixed since change 20395 up to 20460 (F=Feature, B=Bug): job000584 (B): Fixed a typo inserted in mkdirs.sh. job000588 (B): chmod +x fix on some scripts. Released: Rev. SDP/MultiArch/2016.1/20395 (2016/09/04) Jobs Fixed between changelists 20387 and 20395 (F=Feature, B=Bug): job000584 (B): Fixed a typo inserted in mkdirs.sh. Released: Rev. SDP/MultiArch/2016.1/20387 (2016/09/04) Jobs Fixed between changelists 20348 and 20387 (F=Feature, B=Bug): job000579 (B): Fixed several issues in email.sh maint script. job000582 (B): Fixed using with runRemoteCmd getting error if USER not defined in non-core SDP script. job000583 (B): Removed references to legacy names for checkpoint scripts (non-functional change). Jobs Fixed between changelists 20028 and 20348 (F=Feature, B=Bug): job000545 (F): Add option to install_sdp_python.sh to build P4Python with SSL. job000564 (F): Add p4sanity_check.sh script from Battle School. job000565 (B): Fix missing echo of command into log file in p4verify.sh. job000567 (F): Change adminspass to use hidden, and now instance-specific, password files job000568 (F): Add mkrep script to simplify replica creation. job000569 (F): Update p4verify.sh to verify shelves on master/replicas. job000570 (F): Add script to convert labels to autoreload. job000571 (F): Add Evil Twin Detector maintenance script. job000575 (F): Add script to simplify adoption of OS-level best practices recommended by Support. job000577 (B): Eliminate syntax error if user forgets start/stop/status arg; just display status. job000578 (F): Use pid to shutdown p4d rather than 'p4 admin stop' to enhance reliability. Released: Rev. SDP/MultiArch/2016.1/20028 (2016/08/03) Jobs Fixed between changelists 19888 and 20028 (F=Feature, B=Bug): job000548 (F): Defend against environment hijacking of P4ENVIRO. job000557 (F): Revive external auth LDAP group mirror scripts. job000550 (B): sdp_sync.sh v2.1.7: Fixed environment bug (missing defs), HMS support. job000553 (F): Simplify login ticket handling for sophisticated topologies. job000556 (F): Prevent cross-instance contamination if p4_vars is loaded multiple times. Released: Rev. SDP/MultiArch/2016.1/19888 (2016/07/07) Jobs Fixed between changelists 19838 and 19888 (F=Feature, B=Bug): job000274 (B): SDP trigger breaks with Streams workspaces in P4V. job000547 (B): Daily/weekly_backup can fail leaving lock file present requiring manual intervention job000548 (F): Defend against environment hijacking of P4ENVIRO. Released: Rev. SDP/MultiArch/2016.1/19838 (2016/06/26) Jobs Fixed between changelists 19768 and 19838 (F=Feature, B=Bug): job000274 (B): SDP triggers break with Streams workspaces in P4V. job000546 (F): Add defense against known-misbehaving client versions. Released: Rev. SDP/MultiArch/2016.1/19768 (2016/06/24) Jobs Fixed between changelists 19661 and 19768 (F=Feature, B=Bug): job000541 (F): Change to best-practice configurables; auth.autologinprompt=0. job000543 (B): p4master_run operates incorrectly as root. Released: Rev. SDP/MultiArch/2016.1/19661 (2016/06/08) Jobs Fixed between changelists 19410 and 19661 (F=Feature, B=Bug): job000538 (F): Add KEEPJNLS to allow keeping more jnls than ckps. job000539 (F): Rename replica scripts to be consistent with earlier rename of checkpoint scripts. job000540 (F): Enhance error checkong and logging replica handling scripts. job000541 (F): Change to best-practice configurables; auth.autologinprompt=0. Released: SDP/MultiArch/2016.1/19410 (2016/05/17) Jobs Fixed between changelists 18958 and 19410: job000510: Work with license check in P4D 2016.1. job000512: Change p4verify.sh to use -S to verify shelves on a replica instead job000513: Removed the HOST_IP settings from mkdirs and instance_vars since it job000514: Dockerify SDP test suite. job000515: Add replica cleanup script on Windows. job000516: Get p4verify.sh to check SHAREDDATA on replicas. job000517: Avoid chmod on a replica if it shares data with the master. job000518: Use 'p4 admin journal' on master to rotate journal. Released: SDP/MultiArch/2016.1/18958 (2016/04/08) Jobs Fixed between changelists 18619 and 18958: job000484: Proxy status init script can show proxy is up when it is not. job000489: Moved ckp_running.txt to LOGS dir. job000490: Fix missing LOGFILE= in replica_cleanup.sh. job000491: De-emphasized changeType=restricted. job000492: Update Perl/P4Perl Python/P4Python job000493: Support multiple broker configs. job000494: Enhanced p4_vars.template to support non-p4d hosts. job000495: SDP-ify P4DTG init script. job000496: Enhanced p4deleteuser.py to check for shelves, double-deletes. Released: SDP/MultiArch/2016.1/18619 (2016/03/10): Jobs Fixed between changelists 17297 and 18619 job000427 on 2015/12/15 by ttyler *closed* 'Enhanced offline_db 'stale data' Jobs Fixed between changelists 16784 and 17297: job000436: Support $MAILTO values with'#' char in gen_default_broker_cfg.sh. job000446: sdp_sync.sh: Fix 'null crontab' bug. job000447: p4d_base: When stopping p4d, send stderr to /dev/null. job000448: global_replica_status.sh: Support instance-specifc host config. job000449: Support email addresses starting with '#'. Released: SDP/MultiArch/2016.1/16784 (2016/01/12) job000434: Add documentation for sdp_sync.sh to SDP Guide. job000436: Support $MAILTO values with'#' char in gen_default_broker_cfg.sh. -------------------------------------------------------------------------- For more information on jobs, change history details, etc, see: https://swarm.workshop.perforce.com/projects/perforce-software-sdp