USAGE for reset_sdp.sh v4.11.8: reset_sdp.sh [-c <cfg>] [-no_ssl] [-no_sd] [-no_cron] [-no_ppr] [-no_systemd] [-no_firewall] [-no_tweaks] [-no_sudo|-ls] [-v] [-fast|-dapi] [-t <ServerType>] [-s <ServerID>] [-ts <TargetServerID>] [-tp <TargetPort>] [-lp <ListenPort>] [-se] [-H <hostname>] [-T <timezone>] [-local|-B] [[-d <sdp_dir>] | [-b <branch>[,@cl]]] [-p <dir1>[,<dir2>,...]>] [-i <helix_installer_branch>] [-D] [-X|-R] [-M] or reset_sdp.sh -C > settings.cfg or reset_sdp.sh [-h|-man] SAFETY NOTICE: This script SHOULD NEVER EXIST on a Production Perforce server. It is useful for bootstrapping a new Production server machine as well as demo hardware, but this script should be removed after it has successfully executed on a Production server. DESCRIPTION: This script simplifies the process of testing an SDP installation, repetitively blasting all process by the 'perforce' user and resetting the SDP from the ground up, blasting typical SDP folders each time. By default, it installs the Perforce Helix Core server (P4D) with a P4Broker, and installs the Perforce Sample Depot data set used for training and PoC installations. With command line options, it can also install a proxy (P4P) server or a server running only a P4Broker (e.g. in a DMZ). It is helpful when bootstrapping a demo server with a sample data set, complete with broker, and optionally Perl/P4Perl and Python/P4Python. This script handles many aspects of installation. It does the following: * Creates the OS user that will run the Helix Core p4d process, the 'perforce' user by default, using the 'useradd' command, unless that account already exists. If a non-local account is to be used, that should be created first before running this script. If the account is created using 'useradd', the password will be set to match that of the admin P4USER, which is also 'perforce' by default (matching the OSUSER). * Creates the home directory for the OSUSER user, if needed. * Creates and enables systemd *.service files if neeeded. * Does needed SELinux/systemd configuration if semanage and restorecon are available. * Adds OS packages as needed for P4Perl/P4Python local builds (if -dapi is specified). Following installation, it also does the following to be more convenient for demos, and also give a more production-like feel: * Grants the perforce user sudo access (full or limited). * Creates default ~perforce/.bash_profile and .bashrc files. * Connects to the Perforce Package Repository (APT and YUM only). * Adds firewalld rules for Helix server and broker ports ('firewalld' only; there is no support for the 'ufw' or other firewalls). * Installs crontab for ~perforce user. PLATFORM SUPPORT: This works on Red Hat Enterprise Linux, CentOS, and Mac OSX 10.10+ thru Mojave platforms. It works on RHEL/CentOS 6.4-7.6, SuSE Linux 12, and likely on Ubuntu 18 and other Linux distros with little or no modification. This script currently supports the bin.linux26x86_64 (Linux) and bin.maxosx1010x86_64 (Mac OSX/Darwin) architectures. This script recognizes SysV, Systemd, and Launchd init mechanisms, though does not currently support Launchd on OSX. For Mac OSX, note that this requires bash 4.x, and the default bash on Mac OSX remains 3.x as of OSX Mojave. For operating on Mac, the /bin/bash shebang line needs to be adjusted to reference a bash 4 version, e.g. /usr/local/bin/bash if installed with Homebrew. REQUIREMENTS: The following OS packages are installed (unless '-fast' is used): * Yum: bc cronie curl gcc gcc-c++ mailx make openssl openssl-devel rsync tar wget which zlib zlib-devel * AptGet: bc build-essential cron libssl-dev make zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev rsync * Zypper: bc cronie curl gcc gcc-c++ make openssl openssl-devel wget which zlib zlib-devel rsync Development utilities such as 'make', the 'gcc' compiler, and 'curl' will be available (unless running with '-fast'). In addition, if the Perforce Package Repository is added, these additional packages are installed: * Yum: perforce-p4python3 * AptGet: perforce-p4python3 * Zypper: None, as the Perforce Package Repository does not support the Zypper package management system (e.g. as used on SuSE Linux). OPTIONS: -c <cfg> Specify a config file. By default, values for various settings such as the email to send script logs to are configure with demo values, e.g. P4AdminList@p4demo.com. Optionally, you can specify a config file to define your own values. For details on what settings you can define in this way, run: reset_sdp.sh -C > setings.cfg Then modify the generated config file settings.cfg as desired. The generated config file contains documentation on settings and values. If no changes are made to the generated file, running with '-c settings.cfg' is the equivalent of running without using '-c' at all. -C See '-c <cfg>' above. -no_ssl By default, the Perforce server is setup SSL-enabled. Specify '-no_ssl' to avoid using SSL feature. -no_sd By default, the Perforce Sample Depot data set is loaded. Specify '-no_sd' to skip loading the Sample Depot. -no_cron Skip initialization of the crontab. -no_ppr Skip addition of the Perforce Package Repository for YUM/APT repos. By default, the Package Repository is added. -no_sudo Specify that no updates to sudoers are to be made. WARNING: If systemd/systemctl is used to manage Perforce Helix services, the OSUSER that operates these services ('perforce' by default) requires sufficient sudo access to start and stop services. Using this option may result in an unusable service being created. If this option is used, consider also using '-no_systemd'. Alternately, it is appropriate to use this option if the machine it operates on was based on a machine image that pre-creates the OSUSER with sufficient sudo accces. This option is mutually exclusive with '-ls'. -ls Specify that only limited sudo is to be granted. By default full sudo access is granted to the OSUSER by adding this file: /etc/sudoers.d/perforce with these contents: perforce ALL=(ALL) NOPASSWD: ALL If '-ls' is specified, limited sudoers access is provisioned, with just enough access to execute commands like: systemctl <start|stop|status> p4d_* systemctl <start|stop|status> p4dtg_* systemctl <start|stop|status> p4broker_* systemctl <start|stop|status> p4p_* For more detail, see the template file: perforce_sudoers.t This option is mutually exclusive with '-no_sudo'. -v Specify '-v' to run the verify_sdp.sh script after the SDP installation is complete. If '-v' is specified and the verify_sdp.sh script is available in the SDP, it is executed. If the Sample Depot is loaded, the '-online' flag to the verify_sdp.sh script is added. If '-no_cron' is specified, the corresponding '-skip cron' option is added verify_sdp.sh. If '-no_sd' is specified, the '-skip' tests also exclude the offline_db and p4t_files checks in verify_sdp.sh. -fast Specify '-fast' to skip package installation using the package manager (yum, apt-get, or zypper). The '-fast' flag should not be used if you plan to deploy or develop triggers that use P4Python or P4Perl, such as the SDP CheckCaseTrigger.py. -dapi Specify '-dapi' (derived API) to attempt install of the SDP derived APIs if they are available in the SDP package. Those scripts are no longer included in the SDP 2020.1+. This option may be deprecated in a future release. Pragmatically, this option is only useful if local builds are needed for Perl, Ruby, associated derived APIs such as P4Perl. For Python and P4Pytyon in particular, this option is not needed if you are operating on a platform for which the perforce-p4python3 package is available. The '-fast' and '-dapi' options are mutually exclusive. -local By default, various files and binaries are downloaded from the Perforce Workshop and the Perforce FTP server as needed. If the server machine on which the Helix Installer is to be run cannot reach the public internet or if using files from external sites is not desired, the '-local' flag can be used. With '-local', needed files must be acquired and put in place on the server machine on which this script is to be run. Any missing files result in error messages. The '-local' argument cannot be used with -B. For '-local' to work, the following must exist: 1. Helix Binaries Helix binaries must exist in /hxdepots/helix_binaries: * /hxdepots/helix_binaries/p4 * /hxdepots/helix_binaries/p4d * /hxdepots/helix_binaries/p4broker * /hxdepots/helix_binaries/p4p 2. Server Deployment Package (SDP) The SDP tarball must be acquired an put in place here: * /hxdepots/downloads/sdp.Unix.tgz 3. Sample Depot Tarball The Sample Depot appropriate to your platform must exist unless the the '-no_sd' ('No Sample Depot') is used. * /hxdepots/downloads/sampledepot.mac.tar.gz (on Mac/OSX if case-insensitive) * /hxdepots/downloads/sampledepot.tar.gz (on UNIX/Linux or case-sensitive Mac) 4. Helix Installer With '-local', the 'reset_sdp.sh' script and all related files must be acquired and placed in /hxdepots/reset. See EXAMPLES below for sample of acquiring files for use with '-local' mode. -B Specify '-B' to blast base SDP dirs, for a clean start. Otherwise without '-B', downloaded components from earlier runs will be used (which should be fine if they were run recently). The '-B' flag also replaces files in the /hxdepots/reset directory, where this script lives, with those downloaded from The Workshop (the versions of which are affected by the '-i <helix_installer_branch>' flag, described below). The '-B' flag also blasts the /tmp/downloads and /tmp/p4perl directories, used by install_sdp_python.sh and install_sdp_perl.sh, if they exist. -no_firewall Specify '-no_firewall' to skip updates to firewall. By default, if on a system for which the host-local firewall service available is one this script handles (currently firewalld but not ufw), then the firewall service is updated to open appropriate ports for the Perforce Helix services installed. -no_systemd Specify '-no_systemd' to avoid using systemd, even if it appears to be available. By default, systemd is used if it appears to be available. This is helpful in operating in containerized test environments where systemd is not available. This option is implied if the systemctl command is not available in the PATH of the root user. -no_tweaks Skip execution of the SDP operating system tweaks script, os_tweaks.sh. -t <ServerType> Specify the type of server. Valid values are: * p4d_master - A master/commit server. * p4d_replica - A replica with all metadata from the master (not filtered i any way). * p4d_filtered_replica - A filtered replica or filtered forwarding replica. * p4d_edge - An edge server. * p4d_edge_replica - Replica of an edge server. Also set TargetServerID. * p4broker - An SDP host running only a p4broker, with no p4d. * p4proxy - An SDP host running only a p4proxy, with no p4d no p4d. -s <ServerID> Specify the ServerID. A ServerID is required if the ServerType is any p4d_* type other than p4d_master. -ts <TargetServerID> Specify the Target ServerID. Set this only if ServerType is p4d_edge_replica. The value is the ServerID of edge server that this server is a replica of, and must match the ReplicatingFrom: field of the server spec. -tp <TargetPort> Specify the target port. For p4broker and p4proxy only. -lp <ListenPort> Specify the port to listen on. For p4broker and p4proxy only. -se Specify -se to simulate email. This generates a mail simlator script: /p4/common/site/bin/mail -H <hostname> Set the hostname. This is only supported on systems that support the 'hostnamectl' command. The hostname is set by doing: hostnamectl set-hostname <hostname> If the corresponding 'Hostname' setting is defined in the configuration file and this '-H <hostname>' flag is used, the command line option will override the config file. -T <timezone> Set the timezone. This is only supported on systems that support the 'timedatectl' command. The timezone is set by doing: timedatectl set-timezone <timezone> If the corresponding 'Timezone' setting is defined in the configuration file and this '-T <timezone>' flag is used, the command line option will override the config file. -p <dir1>[,<dir2>,...]>] Specify a comma-delimited list of directories under /p4/common to preserve that would otherwise be removed. Directories must be specified as paths relative to /p4/common, and cannot contain spaces. For example, the value '-p config,bin/triggers' would preserve the /p4/common/config and /p4/common/bin/triggers directories. Directories specified are moved aside to a temporary working area before the SDP folders are removed. After installation, they are moved back via an 'rsync' command with no '--delete' option. This means any files that overlap with the stock install are replaced by ones that originally existed, but non-overlapping files are not removed. This is intended to be useful for developing test suites that install server under /p4/common/bin, e.g. Component Based Development scripts which install under /p4/common/bin/cbd would use '-p bin/cbd'. -d <sdp_dir> Specify a directory on the local host containing the SDP to deploy. Use the special value '-d default' to use the /sdp directory (as per the Docker-based SDP Test Suite environment). The directory specified by '-d' is expected to contain either: * an SDP tarball (sdp.Unix.tgz) file, or * an already-extracted SDP directory, which must include the SDP Version file. -b <branch>[,@cl] The default SDP install method is to use the latest released SDP tarball representing the main branch in The Workshop (public.perforce.com:1666). The latest tarball can be found on this server, consistently named sdp.Unix.tgz. This file appears alongside a version-tagged file named something like sdp.Unix.2019.2.25938.tgz. These appear here: https://swarm.workshop.perforce.com/projects/perforce-software-sdp/files/downloads Specify '-b' to use a different branch, typically '-b dev'. This changes the install method from a tarball to using a 'p4 clone' command using Helix native DVCS features to fetch the very latest unreleased files from the branch at any point in time. This is mainly of interest when testing the SDP or previewing specific up and coming features. If '-b' is specified with the optional @cl syntax, where @cl specifies a changelist with files shelved on the given branch, a form of unshelving is done, enabling a workflow for testing shelved changes with the Helix Installer. So for example, specify '-b dev,@23123' to clone from the dev branch, and then followed by a fetch of shelved changelist @23123, which is expected to have files shelved in the dev branch. DEBUGGING OPTIONS: -i <helix_installer_branch> Specify the branch of the Helix Installer to use. This affects the URL from which Helix Installer files in /hxdepots/reset are pulled from The Workshop. The default is main; an alternative is '-i dev'. -D Set extreme debugging verbosity. -X Extreme reset. This removes the user account for the configured OSUSER ('perforce' by default) and blasts all SDP-related directories at the start of script operations, including the home directory for the configured OSUSER and any configurd system services. This also clears firewall rules ('firewalld' only). Using '-X' does not blast the Helix Installer downloads or helix_binaries directories, and thus is compatible with either the '-B' or '-local' options. This also does not undo the setting of the hostname or the timezone with either the command line ('-H <hostname>' and '-T <timezone>') or configuration file settings. -R Specify '-R' to reset. The cleanup logic is the same as with -X. Unlike -X, with -R, processing stops after the cleanup is done. -M Specify '-M' to allow multiple runs. This is useful for running a series of tests. This option disables the safety feature that self-disables this script after it completes. to prevent accidentally running this script again, e.g. after real data has been migrated to this machine after using this script to bootstrap for production. HELP OPTIONS: -h Display short help message. -man Display this full manual page. --help Alias for -man. EXAMPLES: === FAST INSTALLATION (skipping package updates) === su - mkdir -p /hxdepots/reset cd /hxdepots/reset curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/main/src/reset_sdp.sh curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/main/src/r chmod +x reset_sdp.sh r ./r Note that the 'r' wrapper script calls the reset_sdp.sh script with a pre-defined of flags optimized for fast operation. The 'r' wrapper also handles log capture, writing to the file 'reset_sdp.log'. === COMPREHENSIVE INSTALLATION === su - mkdir -p /hxdepots/reset cd /hxdepots/reset curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/main/src/reset_sdp.sh chmod +x reset_sdp.sh ./reset_sdp.sh 2>&1 | tee reset_sdp.log === CONFIGURED INSTALLATION === su - mkdir -p /hxdepots/reset cd /hxdepots/reset curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/main/src/reset_sdp.sh chmod +x reset_sdp.sh ### Generate a default config file: ./reset_sdp.sh -C > settings.cfg ### Edit settings.cfg, changing the values as desired: vi settings.cfg ./reset_sdp.sh -c settings.cfg 2>&1 | tee log.reset_sdp === LOCAL INSTALL === The following sample commands illustrate how to acquire the dependencies for running with '-local' on a machine that can reach the public internet. The resulting file structure, with paths as shown, would need to be somehow copied to the machine where the this reset_sdp.sh script is to be run. This can be used to facilitate operation on a machine over an "air gap" network. $ mkdir -p /hxdepots/helix_binaries $ cd /hxdepots/helix_binaries $ curl -s -O https://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4 $ curl -s -O https://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4d $ curl -s -O https://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4broker $ curl -s -O https://ftp.perforce.com/perforce/r23.1/bin.linux26x86_64/p4p $ mkdir /hxdepots/downloads $ cd /hxdepots/downloads $ curl -s -O https://ftp.perforce.com/perforce/tools/ $ curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/downloads/helix_installer.tgz $ tar -xzf helix_installer.tgz $ rsync -a hi/src /hxdepots/reset === SDP DEV BRANCH TESTING === The Helix Installer can be used to test SDP changes shelved to the SDP dev branch in The Workshop. The following example illustrates testing a change in shelved changelist 23123: su - mkdir -p /hxdepots/reset cd /hxdepots/reset curl -s -O https://swarm.workshop.perforce.com/projects/perforce_software-helix-installer/download/main/src/reset_sdp.sh ./reset_sdp.sh -b dev,@23123 2>&1 | tee log.reset_sdp.CL23123 After the first test, an iterative test cycle may follow on the same shelved changelist. For each test iteration, the shelved changelist is first updated in the workspace from which the change was originally shelved, e.g. with a command like 'p4 shelve -f -c 23123'. Then a new test can be done by calling reset_sdp.sh with the same arguments. The script will re-install the SDP cleanly, and then re-apply the updated shelved changelist. === SDP TEST SUITE SUPPORT === The Helix Installer can install the SDP in the Docker-based SDP Test Suite. In that environment, the directory /sdp appears on the test VMs, shared from the host machine. To deploy that SDP, use the '-d <sdp_dir>' flag, something like this: ./reset_sdp.sh -d /sdp -fast 2>&1 | tee log.reset_sdp.test
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#45 | 30496 | C. Thomas Tyler | Updated Version, README.md, Release Notes, and doc files Release. | ||
#44 | 30459 | C. Thomas Tyler |
Change URLs in curl calls in docs and code to avoid Apache redirects. Also removed '-s' option from curl calls. |
||
#43 | 30412 | C. Thomas Tyler | Updated docs. | ||
#42 | 30399 | C. Thomas Tyler | Adapted to SDP default value of 'localhost' for DNS name of commit server. | ||
#41 | 30398 | C. Thomas Tyler | Add '-no_enable' option to avoid enabling installed services. | ||
#40 | 30397 | C. Thomas Tyler |
Adjust location of generated crontab file to: /p4/common/etc/cron.d/crontab.<user>.<short_hostname>.<instance>. |
||
#39 | 30393 | C. Thomas Tyler | Updated docs for release. | ||
#38 | 30299 | C. Thomas Tyler | Updated docs. | ||
#37 | 30213 | C. Thomas Tyler |
Added '-si <SDPInstance>' as a command line option, so there is one less reason to need to do the more complex 'Configured' installation, especially for a proxy or stand-alone broker. #review-30214 |
||
#36 | 30045 | C. Thomas Tyler | 2023.2 release; changed default version to 2023.2. | ||
#35 | 29698 | C. Thomas Tyler |
Changed 'curl' commands referencing Workshop to use '-L'. Updated stable version numbers in README.md to use new versions with corrected URLs. |
||
#34 | 29661 | C. Thomas Tyler | Updated doc and sample to match latest version of reset_sdp.sh. | ||
#33 | 29615 | C. Thomas Tyler |
Set default to r23.1. Removed '-k' from curl calls. |
||
#32 | 28928 | C. Thomas Tyler | Updated doc file. | ||
#31 | 28662 | C. Thomas Tyler |
Added support for SELinux in enforcing mode. Enhanced documentation, including notes about SELinux handling. |
||
#30 | 28656 | C. Thomas Tyler | Updated docs to latest. | ||
#29 | 28582 | C. Thomas Tyler |
reset_sdp.sh v4.9.0: Added '-no_sudo' option. Added '-no_firewall' option. Fixed minor issue where 'systemctl' commands try to execute when systemctl isn't avail. Added test coverage in test_hi.sh. |
||
#28 | 28266 | C. Thomas Tyler |
Improved support for proxy-only hosts. #review-28267 @rwillyoung |
||
#27 | 28125 | C. Thomas Tyler | Regenerated document. | ||
#26 | 27553 | C. Thomas Tyler |
Spell check only. Non-functional changes. Docs udpated. Fixed typos in comments and messages, including generated settings.cfg files. |
||
#25 | 27391 | C. Thomas Tyler | Updated docs. | ||
#24 | 27305 | C. Thomas Tyler | Added rsync package to all platforms. | ||
#23 | 27302 | C. Thomas Tyler | Updated command line summary. | ||
#22 | 26606 | C. Thomas Tyler | Non-functional changes to maintain shellcheck compliance. | ||
#21 | 26576 | C. Thomas Tyler |
Fixed order-of-operations issue due to recent changes in SDP; the logic to implement 'sudo' as required to call 'systemd' wrappers now requires the sudo to be called before the call to configure_sample_depot.sh. Refined firewall handling logic. |
||
#20 | 26565 | C. Thomas Tyler | Set password for OSUSER to same as P4USER if OSUSER needs to be created. | ||
#19 | 26557 | C. Thomas Tyler |
Changed default Helix Core to r20.1. Updated docs. |
||
#18 | 26022 | C. Thomas Tyler | Updated doc page. | ||
#17 | 25915 | C. Thomas Tyler |
Fixed issue with firewalld configuration on RHEL/CentOS 7 systems. Changed '-X' extreme cleanup to now clear firewalld rules. Added missing mention of firewalld config in documentation. |
||
#16 | 25870 | C. Thomas Tyler | Updated Version/Release Notes and docs for release. | ||
#15 | 25819 | C. Thomas Tyler | Updated docs. | ||
#14 | 25753 | C. Thomas Tyler |
Added code to call SDP os_tweaks.sh. Regenerated command summary file. |
||
#13 | 25392 | C. Thomas Tyler | Update default SDP release. | ||
#12 | 25382 | C. Thomas Tyler |
Updated default SDP release for tar installs to sdp.Unix.2019.1.25374.tgz. |
||
#11 | 25370 | C. Thomas Tyler |
Added '-c <cfg>' and '-C' flags to use (and with '-C' generate) a config file (similar to the '-C' and '-c <cfg>' flags with p4broker). Updated docs accordingly. Added missing 'chmod +x' step in man page in docs. Enabled testing on Mac OSX to get past the package manager test, since yum/apt-get/zypper aren't expected to be on Mac OSX. |
||
#10 | 25282 | C. Thomas Tyler | Fixed order-of-operations issue for creating ~perforce/.bash* files. | ||
#9 | 25248 | C. Thomas Tyler | Filled in SDP 2019.1 changelist number, 25238. | ||
#8 | 25247 | C. Thomas Tyler | Additional cleanup. | ||
#7 | 25240 | C. Thomas Tyler |
Simplified to use a single /hxmetadata volume, as this is the most common configuration. Small sites don't need multiple volumes, and big sites have an array of hardware options, only some of which require a two-volume solution for metadata to maintain performance. Changed default replica ServerID from p4d_fr_bos to p4d_ha_bos, to promote modern 2018.2 high availability replicas that can be created with the mkrep.sh in SDP 2019.1+. The replica is not actually created with the Helix Installer, just referenced as a sample ServerID name. Extended '-B' flag to cleanup temp dir used by install_sdp_python.sh. Changed behavior to bail on failure to cleanup temp dirs, as this could result in wasted time building outdated versions of P4Python, etc. Added '-i <helix_installer_branch>' debugging flag to support testing of Helix Installer from the dev branch. Moved this and the '-D' flag to a new doc section titled DEBUGGING OPTIONS. Added '-X' eXtreme reset debugging option. Updated some internal variable names. |
||
#6 | 25054 | C. Thomas Tyler |
Updated SDP default tarball to 2018.1.23583.p2. Corrected Swarm release to 2018.3; there is no 2018.4! |
||
#5 | 24913 | C. Thomas Tyler | Updated version for release. | ||
#4 | 24863 | C. Thomas Tyler | Updated Version, release notes, and docs for release. | ||
#3 | 23364 | C. Thomas Tyler | Updated Version to release 2017.4.23363. | ||
#2 | 23360 | C. Thomas Tyler | Updated Version, relnotes and doc for release. | ||
#1 | 23281 | C. Thomas Tyler | Added doc page. |