#------------------------------------------------------------------------------ # Instance specific variables are set or calculated in this file. This file # is not intended to be sourced directly. It is sourced when the main SDP # environment file, /p4/common/bin/p4_vars, is sourced. This p4_N.vars file # is sourced as in the following example, where N is the SDP instance name, # with a value of 1 by default: # # source /p4/common/bin/p4_vars N # # A line like the above can optionally be added in the ~/.bashrc file # of the perforce user on the Linux system. This is recommended so that # the SDP Environment is loaded properly during initial login. # #------------------------------------------------------------------------------ # When editing values in this file keep in mind the following: # # * Do not change values for which the comments say 'DO NOT CHANGE'. # * Do not add code or logic of any kind. # * Limit changes strictly to change the right-side of assignments (i.e. text # to the right of the '=' sign). # * Be careful not to introduce whitespace characters on either side of # the '=' sign. # # If custom logic is needed, e.g. to add host-specific behaviors or configure # multiple active brokers with different ports, add that custom logic in custom # local files that appear in the /p4/common/site directory. Files in the # 'site' directory will not be not affected by SDP upgrades. Also, files in the # 'site' directory are inherently custom and not officially supported by # Perforce Technical Support. # # If needed, custom logic goes into these files: # * /p4/common/site/config/p4_vars.local - Global/Common # * /p4/common/site/config/p4_N.vars.local - Instance-specifc (N=instance). # # This file contains comments of the form: # shellcheck disable=SCNNNN. # DO NOT CHANGE OR MOVE THESE COMMENT LINES. #------------------------------------------------------------------------------ # Disable shellcheck requirement for shebang line, as this file is sourced # in (from p4_vars) rather than executed as a script. # shellcheck disable=SC2148 #------------------------------------------------------------------------------ # shellcheck disable=SC2034 # SDP_INSTANCE_VARS_FORMAT defines the version of the format of this file, # used when the SDP is upgraded. DO NOT CHANGE SDP_INSTANCE_VARS_FORMAT. SDP_INSTANCE_VARS_FORMAT=1.13 # This file is processed by mkdirs.sh to update REPL_* values as appropriate. # It can subsequently be edited if required. # Various scripts send email to $MAILTO. This can contain comma-delimited list # of email addresses (no spaces). The addresses specified here are generally # those of individual Perforce Helix administrators, and/or to an email # distribution list of administrators. export MAILTO=REPL_MAILTO # Specify the email address from which emails are sent in $MAILFROM. # This must be exactly one email address. The value typically looks something # like P4Admin@example.com. # NOTE: Many modern email services, such as Office365 and Google Workspaces, # may require that the domain name in the email address from which emails are # sent be the correct domain for your company. For example, if your company's # domain for email is example.com, the email server may require that any emails # sent from that domain also end in @example.com. export MAILFROM=REPL_MAILFROM # This is the P4USER that is the super user. This user must have a unlimited # ticket valid on the master Perforce Helix server machine. export P4USER=REPL_ADMINUSER # Service user definition. DO NOT CHANGE P4SERVICEUSER. export P4SERVICEUSER=svc_${SERVERID} # P4MASTER_ID is server.id of Master for this instance export P4MASTER_ID=REPL_MASTER_ID # If this instance uses SSL, set SSL_PREFIX to ssl:, ssl4:, ssl6:, # ssl46:, or ssl64:. See: https://portal.perforce.com/s/article/1180 # The trailing colon is required. export SSL_PREFIX=REPL_SSLPREFIX # Set the numeric portion of the P4PORT value used for p4d. export P4PORTNUM=REPL_P4PORT # The P4PORT value as defined here is only for use on the server machine. For # safety and clarity, it intentionally excludes a hostname component, and # looks like 'ssl:1999' or just '1999', but not something like # 'ssl:some_host:1999'. The intent is to ensure that the P4PORT value cannot # accidentally target another machine due to DNS and/or /etc/host hacks. # This P4PORT value is calculated. DO NOT CHANGE P4PORT. export P4PORT=${SSL_PREFIX}${P4PORTNUM} # Set the value for P4BROKERPORT to Unset if the broker is not # used. export P4BROKERPORTNUM=REPL_P4BROKERPORT # This is the broker port. To test broker connections, you # can use a command like: p4 -p $P4BROKERPORT info # DO NOT CHANGE P4BROKERPORT. export P4BROKERPORT=${SSL_PREFIX}${P4BROKERPORTNUM} # These $P4D_FLAGS show when using 'ps' to scan the process table. # Other environment variables and 'p4 configure' settings can affect # p4d even if not listed on the command line. # DO NOT CHANGE P4D_FLAGS. export P4D_FLAGS="-p $P4PORT -r $P4ROOT -J $P4JOURNAL -L $P4LOG -q -d --pid-file" # Default values of the following - may be overwritten below # P4MASTERHOST=DNS Name (or IP address) of the commit server # for this instance. If DNS is not available, /etc/hosts # entries may be used to simulate DNS. export P4MASTERHOST=REPL_P4MASTERHOST # The P4MASTERPORT is the P4PORT value of the commit server, used # by replicas, proxies, and standalone brokers. This incorporates # teh DNS name of the commit server. # DO NOT CHANGE P4MASTERPORT. export P4MASTERPORT=${SSL_PREFIX}${P4MASTERHOST}:${P4PORTNUM} # P4REPLICA is calculated. DO NOT CHANGE P4REPLICA. export P4REPLICA=FALSE # SHAREDDATA defaults FALSE, but is set to TRUE on replica servers that share their /hxdepots # volume with their P4TARGET server - as per lbr.replication configurable # DO NOT CHANGE SHAREDDATA. export SHAREDDATA=FALSE # SERVERID is defined in /p4/N/root/server.id. DO NOT CHANGE SERVERID. if [[ -n "$SERVERID" && "$SERVERID" != "$P4MASTER_ID" ]]; then export P4REPLICA=TRUE TmpP4ROOT= if [[ -f "$P4ROOT/db.config" && ! -f "$P4ROOT/P4ROOT_not_usable.txt" ]]; then TmpP4ROOT=$(mktemp -d "$P4TMP/tmp.p4root.XXXXXX") cp "$P4ROOT/db.config" "$TmpP4ROOT/." # Use egrep because it works where we need it to. Shellcheck suggests # using 'grep -E' instead. # shellcheck disable=SC2196 targetServerPort=$("$P4DBIN" -r "$TmpP4ROOT" -cshow | egrep "^$SERVERID: P4TARGET = ") targetServerPort=${targetServerPort##* = } # Use 'echo' w/o quotes to remove leading/trailing whitespace. # shellcheck disable=SC2116 disable=SC2086 targetServerPort=$(echo $targetServerPort) else targetServerPort= fi if [[ -n "$targetServerPort" ]]; then export P4MASTERPORT="$targetServerPort" if [[ "$targetServerPort" == *":"* ]]; then # strip off any trailing ':.....' element (e.g., ':1666') export P4MASTERHOST="${targetServerPort%:*}" # if P4TARGET includes ssl, also need to strip off ssl: from front export P4MASTERHOST="${P4MASTERHOST#*:}" fi fi repStyle="readonly" if [[ -f "$P4ROOT/db.config" && ! -f "$P4ROOT/P4ROOT_not_usable.txt" ]]; then repStyle=$("$P4DBIN" -r "$P4ROOT" -cshow|"$GREP" "^$SERVERID: lbr.replication = ") repStyle=${repStyle##* = } # Use 'echo' w/o quotes to remove leading/trailing whitespace. # shellcheck disable=SC2116 disable=SC2086 repStyle=$(echo $repStyle) fi if [[ $repStyle == "shared" ]]; then # Used by p4verify.sh among others export SHAREDDATA=TRUE fi # shellcheck disable=SC2115 [[ -n "$TmpP4ROOT" ]] && rm -rf "$TmpP4ROOT/" fi #------------------------------------------------------------------------------ # Proxy settings. These PROXY_* settings can be left as they appear, or changed # as desired. The usage of SSL_PREFIX does not need to be consistent in the # PROXY_TARGET and PROXY_PORT (local listen port). It is possible to have # either, neither, or both PROXY_TARGET and PROXY_PORT using the SSL_PREFIX. # The usage of the SSL for the PROXY_TARGET port value must reflect was is used # by the target p4d server (a commit or edge). export PROXY_TARGET=${SSL_PREFIX}${P4MASTERHOST}:${P4BROKERPORTNUM} export PROXY_PORT=${SSL_PREFIX}$((P4BROKERPORTNUM+1)) # Set Proxy monitoring level to 3 in order to monitor all traffic for all operations export PROXY_MON_LEVEL=3 # Add '-u $P4SERVICEUSER' to PROXY_V_FLAGS if required, i.e. if security >= 5. export PROXY_V_FLAGS="-e 3072 -v proxy.monitor.level=${PROXY_MON_LEVEL:-3} -v net.autotune=1" #------------------------------------------------------------------------------ # Perforce Defect Tracking Gateway (P4DTG) Settings: export P4DTG_CFG=$P4SERVER #------------------------------------------------------------------------------ # If SNAPSHOT_SCRIPT is defined and set to the path of an executable # script, daily_checkpoint.sh will call the referenced scrip after # checkpoints are created. It is expected that the called custom script # will create a snapshot or perform some other backup activity. # The logic below allows for the called SNAPSHOT_SCRIPT to run only # on the master server. # # A sample script to enable snapshots on an AWS master # server is provided as: /p4/common/cloud/aws/bin/snapshot.sh # if [[ -n "$SERVERID" && "$SERVERID" == "$P4MASTER_ID" ]]; then # This snapshot script is exectued only on the master server. export SNAPSHOT_SCRIPT= else # Leave this value empty. export SNAPSHOT_SCRIPT= fi #------------------------------------------------------------------------------ # Set VERIFY_SDP_SKIP_TEST_LIST as an alternative to passing in a lists of tests # using the '-skip' flag to the verify_sdp.sh script. The command line flag to # the script is more appropriate if a test is to be skipped temporarily, while # setting the VERIFY_SDP_SKIP_TEST_LIST variable is better for making a permanent # exception (e.g. skipping crontab checks if some other scheduler is used). # # The value is a comma-delimited list of tests to skip, as defined in the # documentation for verify_sdp.sh. For more info, run: verify_sdp.sh -man # # Valid test names are: # * cron|crontab: Skip crontab check. Use this if you do not expect crontab to # be configured, perhaps if you use a different scheduler. # * excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH. # * init: Skip compare of init scripts w/templates in /p4/common/etc/init.d # * license: Skip license related checks. # * commitid: Skip check ensuring ServerID of commit starts with 'commit' or 'master'. # * masterid: Synonym for commitid. # * offline_db: Skip checks that require a healthy offline_db. # * p4root: Skip checks that require healthy P4ROOT db files. # * p4t_files: Skip checks for existence of P4TICKETS and P4TRUST files. # * passwd|password: Skip SDP password checks. # * version: Skip version checks. export VERIFY_SDP_SKIP_TEST_LIST= #------------------------------------------------------------------------------ # The SDP_ALWAYS_LOGIN should be set to 0 except in certain rare situations # with older p4d servers. With the value set to 0, the p4login script does # a 'p4 login -s' first to see if a ticket is valid, and logs in again only # if required. Generally the P4USER in this script will have a non-expiring # ticket. export SDP_ALWAYS_LOGIN=0 #------------------------------------------------------------------------------ # The SDP_MAX_START_DELAY_* settings are numeric values that define the maximum # number of second to wait for services to start before considering the startup # a failure in various SDP scripts. Successful start is indicated by being # responsive to a 'p4 info' command and the appropriate port for each service. export SDP_MAX_START_DELAY_P4D=120 export SDP_MAX_START_DELAY_P4BROKER=60 export SDP_MAX_START_DELAY_P4P=60 # The SDP_MAX_STOP_DELAY_* settings are numeric values that define the maximum # number of second to wait for services to stop before considering the shutdown # a failure in various SDP scripts. Successful stop is indicated by removal of # the server.pid file for p4d, and lack of active pids for p4broker/p4p. These # values are typically never used. Note that 'p4d' does not have a max delay. export SDP_MAX_STOP_DELAY_P4BROKER=600 export SDP_MAX_STOP_DELAY_P4P=600 #------------------------------------------------------------------------------ # Set SDP_AUTOMATION_USERS to a comma-delimited list of Perforce user accounts # to login when the 'p4login' script is used with the '-automation' flag. This # will normally be a list of users that operate trigger scripts, such as the # 'swarm' user if Helix Swarm is used. Typically such users will have long-term # tickets. For more info, run: p4login -man export SDP_AUTOMATION_USERS="swarm" #------------------------------------------------------------------------------ # Set DO_PARALLEL_CHECKPOINTS=0 to disable parallel checkpoints. # # Set DO_PARALLEL_CHECKPOINTS=N to enable parallel checkpoints, where N # is a positive integer indicating the number of parallel threads to use. # As a special case, setting DO_PARALLEL_CHECKPOINTS=1 is the same as # setting DO_PARALLEL_CHECKPOINTS=4. The N value is passed to the p4d # with the '-N' parameter when doing checkpoint create, dump, and # replay options with 'p4d -jcp', 'p4d -jdp', and 'p4d -jrp', # respectively. # # If parallel checkpoints are enabled: # * The live_checkpoint.sh will create live checkpoints using 'p4d -jcp', and # replay using 'p4d -jrp'. # * The daily_checkpoint.sh will create offline checkpoints using 'p4d -jdp', # and replay using 'p4d -jrp'. # * The recover_offline_db.sh will look for checkpoint directories rather # than singular checkpoint files, and replay with 'p4d -jrp'. # * The refresh_P4ROOT_from_offline_db.sh replays using 'p4d -jrp'. # # The db.checkpoint.threads configurable is ignored with the scripts, as # the '-N' parameter overrides configurable. # # Parallel checkpoints became available in p4d 2022.2. This setting is # ignored if the server version is lower than that. export DO_PARALLEL_CHECKPOINTS=0 #------------------------------------------------------------------------------ # The umask affects default permissions on created files and directories. umask 0026 #------------------------------------------------------------------------------ # Custom Local Changes #------------------------------------------------------------------------------ # Any instance-specific local customizations should appear in the file sourced # below in the /p4/common/site structure, which contains site-specific that # are not part of the SDP package. if [[ -d "/p4/common/site/config/$P4SERVER.vars.local.d" ]]; then # shellcheck disable=SC1090 for f in "/p4/common/site/config/$P4SERVER".vars.local.d/*; do source "$f"; done fi if [[ -r "/p4/common/site/config/$P4SERVER.vars.local" ]]; then # shellcheck disable=SC1090 source "/p4/common/site/config/$P4SERVER.vars.local" fi
# | Change | User | Description | Committed | |
---|---|---|---|---|---|
#60 | 30799 | C. Thomas Tyler |
Added documentation of various settings in instance vars. Addressed some ShellCheck issues without functional change. |
||
#59 | 30517 | C. Thomas Tyler |
Fixed typos in comments. Non-functional change. |
||
#58 | 30509 | C. Thomas Tyler |
Tweaked proxy handling to generate a cleaner command line (as seen in ps -ef), eliminating null or duplicate values for proxy.monitor.level setting. This change makes p4d_base more robust even inf the instance vars settings are slightly misconfigured, e.g. a null value for PROXY_MON_LEVEL. #review-30510 |
||
#57 | 30382 | C. Thomas Tyler | Updated SDP_INSTANCE_VARS_FORMAT version to 1.12. | ||
#56 | 30308 | Mark Zinthefer |
Added comments in instance_vars.template to spell out what the test names are for the skip list. This is meant to make modifying the skip list easier rather than looking in the comments or usage of verify_sdp.sh This means we will have to maintain the comments in this file as verify_sdp.sh. If this proves to cumbersome, we should remove the added comments in instance_vars.template. |
||
#55 | 30223 | C. Thomas Tyler |
Changed so 'p4d -cshow' calls in SDP shell environment mechanism use a copy of db.config to avoid issues with database locking. #review-30224 |
||
#54 | 29923 | C. Thomas Tyler |
Updated HTML hyperlinks to use 'portal.perforce.com'. This replaces currently broken links to 'answers.perforce.com' and currently redirected links to 'community.perforce.com'. #review-29924 |
||
#53 | 29746 | C. Thomas Tyler |
Moved proxy '-e 3072' flag from p4p_base to Instance Vars file so it can be changed on a per-instance basis. It is now part of the def'n of PROXY_V_FLAGS. Moved PROXY_MON_LEVEL from p4_vars to Instance Vars file so it can be changed on a per-instance basis. Updated format settings in sdp_upgrade.sh, and added PROXY_MON_LEVEL to list of user settings to be preserved during SDP upgrades. #review-29747 |
||
#52 | 29609 | C. Thomas Tyler | Updated data format tags per SDP relesae process docs. | ||
#51 | 29563 | Andy Boutte | Adding optional local config directories for both instance and SDP wide configuration. | ||
#50 | 29420 | C. Thomas Tyler |
For parallel checkpoints, provide a way to specify '-N' parallel threads. #review-29421 |
||
#49 | 29396 | C. Thomas Tyler |
Doc improvements to SDP Instance Vars file re: MAILTO/MAILFROM settings. Tweaked Shellcheck SC<NNNN> tags to re-achieve shellcheck compliance. #review-29397 @daniel_ferrara |
||
#48 | 29232 | C. Thomas Tyler |
Added regression test for new parallel checkpoint feature. #review-29220 |
||
#47 | 29116 | C. Thomas Tyler | Fixed value for P4MASTERHOST on broker-only or proxy-only hosts. | ||
#46 | 28641 | C. Thomas Tyler |
Tweaked scripts to support IPv6 SSL prefixes. Added test script to test bash code snippets. First test is the snippet to check if SSL is enabled, and if so get the SSL prefix. |
||
#45 | 28355 | C. Thomas Tyler |
Corrected custom logic extraction issues found during code review/testing. #review-28356 |
||
#44 | 28343 | C. Thomas Tyler |
Added documented-but-missing logic for extracting custom logic from SDP shell environment files. #review-28344 |
||
#43 | 28342 | C. Thomas Tyler |
Added PROXY_V_FLAGS setting to provide a clean way to add proxy settings like "-v net.autotune=1". |
||
#42 | 28247 | C. Thomas Tyler |
Corrected detection of P4MASTERHOST setting. Fixed issue with substitutions occuring in comments, impacting readability of config files. Added step to backup /p4/sdp. Added step to update /p4/sdp. |
||
#41 | 28140 | C. Thomas Tyler |
Refined sdp_upgrade.sh; this is still a non-functional work in progress version, with expanded documentation. Added file format version settings into p4_vars template and instance-specific p4_vars files. |
||
#40 | 27972 | Robert Cowham | Update template comment to be same as the comment in mkdirs.cfg since it is more understandable. | ||
#39 | 27722 | C. Thomas Tyler |
Refinements to @27712: * Resolved one out-of-date file (verify_sdp.sh). * Added missing adoc file for which HTML file had a change (WorkflowEnforcementTriggers.adoc). * Updated revdate/revnumber in *.adoc files. * Additional content updates in Server/Unix/p4/common/etc/cron.d/ReadMe.md. * Bumped version numbers on scripts with Version= def'n. * Generated HTML, PDF, and doc/gen files: - Most HTML and all PDF are generated using Makefiles that call an AsciiDoc utility. - HTML for Perl scripts is generated with pod2html. - doc/gen/*.man.txt files are generated with .../tools/gen_script_man_pages.sh. #review-27712 |
||
#38 | 27512 | C. Thomas Tyler |
Changed so PROXY_TARGET value is consistently: ${SSL_PREFIX}${P4MASTERHOST}:${P4BROKERPORTNUM} Reasons: * The existing sample value always had to be changed, as the replacement value REPL_P4P_TARGET_PORT was never substituted, requiring manual setting when the proxy was added to a topology. * Using the new value is a reasonable default, and promotes the idea that proxies should point to the broker on the master host. (If brokers are not used, it's an easy tweak to change P4BROKERPORTNUM to P4PORTNUM.) |
||
#37 | 27318 | C. Thomas Tyler |
Added comment block introducing '### MAKE LOCAL CHANGES HERE:' comment block to generated SDP files to simplify future automated upgrades of generated SDP files. This is a non-functional change. |
||
#36 | 27194 | C. Thomas Tyler |
Changed default PROXY_PORT value to be whatever numeric portion of the broker port is, plus one. So 1667 by default. The immediate reason for this change is to simplify testing, since the old default of just using P4PORT didn't allow easy testing of the p4p on the same machine as p4d. That made sense insomuch as typically a p4p runs on different machine than the p4d/p4broker stack. But this new value makes as much sense, and is more flexible for testing. #review-27195 |
||
#35 | 27110 | C. Thomas Tyler |
Stop/start robustness enhancements for start/stop_p4*() functions. Added wait loop to start_p4d(), start_p4broker(), and start_p4p() functions and corresponding stop_p4{d/p/broker}() functions. Delays are 1-second intervals up to a configurable maximum number of seconds. Defaults for start are 120 for p4d, 60 each for p4broker and p4p. Defaults for stop are 600 each for p4broker and p4p, but for 'p4d' the wait could cause a hang. This is deemed preferrable over other options. Delays are only incurred as needed. Services that do not start/stop within the alotted time are deemed to have failed to start/stop. Note that while this change is a general robustness enhancement, it is especially important with systemd due to its 'fire and forget' nature when doing a 'sudo systemctl start'. The systemctl command returns immediately and happily even if the p4d startup fails. All 3 start_p4*() and all 3 stop_p4*() functions now have the same "do or die" behavior; they call a 'die' if the service did not stop as requested. When they return, the requested start/stop can be assumed to have completed successfully. Added documentation for new settings in instance_vars.template. Also added doc and example of SDP_AUTOMATION_USERS. #review-27111 |
||
#34 | 27064 | C. Thomas Tyler |
Fixed issue where 'source p4_vars' hangs if load_checkpoint.sh is running. Added new semaphore file, $P4ROOT/P4ROOT_not_usable.txt. This is used in a way similar to 'offline_db_usable.txt' in the offline_db, except that this file only exists when the databases in P4ROOT are not usable. This is the opposite of how offline_db_usable.txt works, because P4ROOT is expected to be usable 99.9% fo the time. p4d_base will refuse to start p4d if this file exists, protecting against possible operator errors (like trying to start p4d when a checkpoint is still loading). Added check_file_dne() function to verify_sdp.sh to confirm a named file does not exist. Added checks in verify_sdp.sh that P4ROOT_not_usable.txt does not exist in P4ROOT or offline_db. Modified switch_db_files() (called by refresh_P4ROOT_from_offline_db.sh) to properly use the new P4ROOT_not_usable.txt safety file. Fixed bugs in p4d_base that could cause p4d_init.log to be overwritten if error output was generated. Removed call to 'backup_functions.sh' in p4d_base, as on balance it added more complexity than needed. #review-27065 |
||
#33 | 26982 | C. Thomas Tyler |
mkdirs.sh v4.1.0: * Accounted for directory structure change of Maintenance to Unsupported. * Added standard command line processing with '-h' and '-man' doc flags, and other flags (all documented). * Added in-code docs and updated AsciiDoc. * Enhanced '-test' mode to simulate /hx* mounts. * Enhanced preflight testing, and fixed '-test' mode installs. * Added support for installing to an alternate root directory. * Added '-s <ServerID>' option to override REPLICA_ID. * Added '-S <TargetServerID>' used for replicas of edge servers. * Added '-t <server_type>' option to override SERVER_TYPE. * Added '-M' option to override mount points. * Added '-f' fast option to skip big chown/chmod commands, and moved those commands near the end as well. verify_sdp.sh v5.9.0: * Added check for /p4/Version file, and checked that other legacy SDP methods of checking version * Added sanity check for crontab. * Added 'test skip' mechanism to skip certain tests: - crontab: Skip crontab check. Use this if you do not expect crontab to be configured, perhaps if a different scheduler is used. - license: Skip license related checks. - version: Skip version checks. - excess: Skip checks for excess copies of p4d/p4p/p4broker in PATH. * Added VERIFY_SDP_SKIP_TEST_LIST setting ton instance_vars.template, to define a standard way to have verify_sdp.sh always skip certain tests for a site. * Extended '-online' checks to check for bogus P4MASTERPORT, a common config error. Update test_SDP.py: * Adjusted test suite to account for various changes in mkdirs.sh. * Added 'dir' parameter to run_cmd() and sudo_cmd(), to run a command from a specified directory (as required to test new mkdirs.sh) * Added check_links() similar to existing check_dirs() function. === Upgrade Process Changes === Made /p4/common/bin/p4d/p4/p4broker/p4p shell script rather than binary. This changes the way SDP new binaries are staged for upgrade. For safety, exes are now staged to a director outside the PATH, the /p4/sdp/exes folder. A new 'get_latest_exes.sh' script simplifies the task of pulling executables from the Perforce FTP server. This can be used 'as is' for environments with outbound internet access, and is useful in any case to describe now to acquire binaries. This addresses an issue where a p4d binary staged for a future upgrade might be called before the actual upgrade is performed. upgrade.sh v4.0.0: * All preflight checks are now done first. Added '-p' to abort after preflight. * Added '-n' to show what would be done before anything is executed. * Minimalist logic to start/stop only servers that are upgrade, and apply upgrades only as needed. * Staging of exes for upgrade is now separate from /p4/common/bin * Improved in-code docs, added '-h' and '-man' options. * Retained pre/post P4D 2019.1 upgrade logic. |
||
#32 | 26718 | Robert Cowham |
Rename P4MASTER to P4MASTERHOST for clarity with comments in: - mkdirs.cfg/mkdirs.sh - p4_<instance>.vars - other files which reference Remove unnecessary sed for p4p.template |
||
#31 | 26652 | Robert Cowham |
This is Tom's change: Introduced new 'Unsupported' directory to clarify that some files in the SDP are not officially supported. These files are samples for illustration, to provide examples, or are deprecated but not yet ready for removal from the package. The Maintenance and many SDP triggers have been moved under here, along with other SDP scripts and triggers. Added comments to p4_vars indicating that it should not be edited directly. Added reference to an optional site_global_vars file that, if it exists, will be sourced to provide global user settings without needing to edit p4_vars. As an exception to the refactoring, the totalusers.py Maintenance script will be moved to indicate that it is supported. Removed settings to support long-sunset P4Web from supported structure. Structure under new .../Unsupported folder is: Samples/bin Sample scripts. Samples/triggers Sample trigger scripts. Samples/triggers/tests Sample trigger script tests. Samples/broker Sample broker filter scripts. Deprecated/triggers Deprecated triggers. To Do in a subsequent change: Make corresponding doc changes. |
||
#30 | 26398 | C. Thomas Tyler |
Added delay on start, to workaround common issues where a fast cycle of p4d process fails because the TCP port (e.g. 1666) is still in use briefly after a clean shutdown, a system resource isn't available immediately on boot, or similar timing issue. The delay on start is skipped if P4ROOT is empty (tested for by checking for existence of db.comain). Added optional new SDP_START_DELAY setting in the p4_vars file (generated from p4_vars.template) to configure the delay. Files in this changelist now pass shellcheck (v0.61) (driving various minor non-functional edits). |
||
#29 | 26394 | C. Thomas Tyler |
Provide snapshot hook for daily_checkpoint.sh. Perhaps the most common SDP customization is a tweak to integrate the SDP checkpoint mechanism with site-local capability of the underlying hardware, so that the snapshot is created at the ideal point in time, right after the metadata checkpoint is created. The intent of this change is to reduce the need for such customization by making a well defined hook in the daily checkpoint process for calling a site-specific custom checkpoint script. Key elements: * Made an illustrative sample change to instance_vars.template, to show setting a SNAPSHOT_SCRIPT variable defined only if running on the master. This logic would be customized to meet local needs, e.g. perhaps checking hostname if snapshots can only be run on certiain machines. The custom logic would live in the /p4/common/config/p4_N.vars file. As a for-example, The sample logic shows a commented-out call to an Amazon Web Services (AWS) Elastic Block Store (EBS) snapshot. (This could just as easily reference a NetApp filer snapshot for an on-prem installation.) * A change to dump_checkpoint() in backup_functions.sh to call the snapshot script defined by the $SNAPSHOT_SCRIPT variable if defined. As coded, a failure of either the checkpoint or the snapshot will cause the script to report a failure. However, a failure of the checkpoint will not prevent the snapshot from being attempted. Possible future tweaks: Consider adding a LIMIT_ONE_DAILY_SNAPSHOT setting (akin to the LIMIT_ONE_DAILY_CHECKPOINT setting in the Windows SDP). |
||
#28 | 25205 | Robert Cowham |
Refactor to reduce repetition and make clearer Add comments |
||
#27 | 23454 | cgeen |
Update to hms script to do two new functions: pull and df. Pull issues a pull -lj on all the edge/replica servers to get the status df issues a diskspace on all the servers. The reason for this is with a hardened edge environment the P4USER does not have permissions to run these commands on the edge servers. Therefore we need to run them centrally as a super. This replaces the replica_status commands on the edge. The updates also includes an update to the p4_vars to change the P4USER on an edge server. The idea is that P4USER only has super privs on the master on the replica edge servers the default user can only have admin privs. This is so the box adminstrator in restricted regions can't change the protection table. Hence the need for pull df centrally as these are the only commands that can't be run on the replicas/edges from the maintance scripts. It also goes without saying that ssh should only go out from the commit and never back. |
||
#26 | 22957 | C. Thomas Tyler |
Merged two changes initiated on main down to: Change @22826 by robert_cowham: Ignore some files on Mac. Change @22950 by awkan/ttyler: Provide an override for difference between proxy listening port and target port |
||
#25 | 22625 | Russell C. Jackson (Rusty) | Approved in Review 22364 | ||
#24 | 22610 | Russell C. Jackson (Rusty) | Added check to make sure db.domain exists before running p4d -cshow to avoid creating an empty database. | ||
#23 | 22333 | Russell C. Jackson (Rusty) |
The service user always needs to be defined since the proxy base script uses it as well. Added note about making sure to leave replica FALSE when setting up a proxy. If you don't the instance vars script will create an empty database by running the cshow commands. |
||
#22 | 22080 | C. Thomas Tyler |
Cosmetic change, doc tweaks and whitespace fix (tabs->spaces). No functional change. |
||
#21 | 20972 | C. Thomas Tyler | Merge down of change made in dev & main. | ||
#20 | 20921 | Russell C. Jackson (Rusty) | Fix that Adrian shelved on Main for ssl: issue. | ||
#19 | 20749 | C. Thomas Tyler |
Approved and committed, but I believe that the shared data setting is always set to false on the master and we should look at fixing that in another change. Enhanced p4login again. Improvements: Default behavior with no arguments gives the desired results. For example, if run on a master, we login on the super user P4USER to P4PORT. If run on a replica/edge and auth.id is set, we login P4USER to the P4TARGET port of the replica. All other login functionality, such as logging in the replication service user on a replica, logging in supplemental automation users, is now accessed via new flags. A usage message is now available via '-h' and '-man' options. The new synopsys is: p4login [<instance>] [-p <port> | -service] [-automation] [-all] The <instance> parameter is the only non-flag positional parameter, and can be ommitted if SDP_INSTANCE is already defined (as is typical when called by scripts). With this change, several other scripts calling either the 'p4login' script or 'p4 login' commands were normalized to call p4login as appropriate given the new usage. Reviewer Note: Review p4login first, then other files. Most changes are in p4login. In other scripts callling p4login, calls similar to: $P4BIN -u $P4USER -p $P4PORT login < /path/to/pwd are replaced with: $P4CBIN/p4login In other scritps calling p4login, calls similar to: $P4BIN -p $P4MASTERPORT login < /path/to/pwd are replaced with: $P4CBIN/p4login -p $P4MASTERPORT Note that, if auth.id is set, calling 'p4login' actually has the same behavior as 'p4login -p $P4MASTERPORT', since p4login called on a replica with auth.id set will just login to the master port anyway. Depending on intent, sometimes $P4BIN/p4login -service is used. == Misc Cleanup == In doing the cleanup: * Fixed a hard-coding-to-instance-1 bug in broker_rotate.sh. * Fixed an inconsistency in recreate_db_sync_replica.sh, where it did just a regular login rather than a login -a as done in other places for (for compatibility with some multi-interface NIC card configs). == p4login Call Normalization == Code cleanup was done to normalize calls to p4login, such that: 1) the call starts with $P4CBIN/p4login (not the hard-coded path), and 2) logic to redirect sdtout/stderr to /dev/null was removed, since it's not necessary with p4login. (And if p4login ever does generate any unwanted output, we only fix it in one place). == Tweak to instance_vars.template == This change includes a tweak to set P4MASTERPORT dynamically on a replica to ensure the value precisely matches P4TARGET for the given replica. This will reduce a source of problems when SSL is used, as it is particularly sensitive to the precise P4PORT values used, and will also help for environments which have not yet set auth.id. If the port cannot be determined dynamically, we fall back to the old logic using the assigned value. == Tweak to SDP_ALWAYS_LOGIN behavior == This used to default to 1, now it defaults to 0. At this point we should no longer need to force logins, and in fact doing so can get into a 'p4 login' hang situation with auth.id set. Best to avoid unnecessary logins if we already have a valid ticket. (I think the need to force a login may have gone away with p4d patches). == Obsolete Script == With this change, svclogin.sh is now obsolete. All it was doing was a few redundant 'p4 login' commands followed by a call to p4login anyway. == Testing == Our test suite doesn't fully cover this change, so additional manual testing was done in the Battle School lab environment. |
||
#18 | 20694 | Russell C. Jackson (Rusty) |
Remove instance specific p4trust and p4ticket files because it is causing you to have to login again when you change the server.id, and that isn't needed when using auth.id and rpl.forward.login. If you are using a shared depotdata volume, just be sure to use the auth.id and rpl.forward.login and you will not have a problem with the ticket files since the login ticket will be the same on all servers. We never had a problem with the trust files in a shared environment since the trust is always based on the master server's IP address. |
||
#17 | 20432 | C. Thomas Tyler |
Improved edge & daisy chained replica support in instance_vars.template. Changed so P4MASTER is set dynamically, based on how/whether the P4TARGET of the current ServerID is set. This also eliminates a possible discrepancey beteween P4MASTER as defined in the p4_N.vars/mkdirs.sh and the master hostname as defined in P4TARGET configurables. The value defined with P4TARGET must also work with SSH keys. (As a best practice P4TARGET should be a host alias so that it doesn't need to be changed in case of failvoer of your P4TARGET server). Changed so SHAREDDATA is set dynamically, based on how/whether the lbr.replication of the current ServerID is set. If it is unset, set to none, ondemand, or cache, then SHAREDDATA is set to TRUE, otherwise FALSE. Dynamic queries use 'p4d -cset' so they work regardless of whether the p4d process is up or not. Some internal refactoring was necessary to ensure all variables are set before they are used. This involved a minor tweak to mkdirs.sh to remove the now-unnecessary 'sed' for SHAREDDATA when generating p4_N.vars from the template. SHAREDDATA must still be configured in mkdirs.sh because it can be run before a replica is fully configured. Goals: * Simplfy SDP configuration for complex topologies by eliminating configuration external to p4d where possible/practical. * Reduce chances for discrepancies and errors as topologies evolve over time. * Allow the p4_1.vars file to identical on all hosts in the topology, an HMS requirement. |
||
#16 | 20376 | C. Thomas Tyler |
Incorporated HMS service user naming standard into the SDP, i.e. "svc_<serverid>." Removed SVCUSER setting from mkdirs.sh accordingly. Fixed mkdirs.sh so /p4/common/etc is created if it doesn't already exist, just as the 'lib' dir is handled. Also a minor structural enhacement in instance_vars.template. Added SDP_ALWAYS_LOGIN setting to instance_vars.template, setting the default to 0 to prevent unnecessary logins. |
||
#15 | 20348 | C. Thomas Tyler |
Use pid to shutdown the p4d process. Goals: 1. Make shutdown more standard using kill/SIGTERM. 2. Make shutdown more reliable; no need to be logged in, insluated from P4AUTH, etc. 3. Use the now-built-in pid mechanism. 4. On shutdown, make it so the script doesn't return until the p4d process is well and truly down. This was implemented with some backward-compatibilty features to simplfy SDP upgrades: 1. Though the p4_N.vars file should add the required '--pid-flag' to P4D_FLAGS, this new p4d_base checks and adds it if it is missing. 2. The old 'p4 admin stop' logic is retained as a new 'admin_stop' function, and is used if the server.pid file does not exist when stop is called. |
||
#14 | 20170 | Russell C. Jackson (Rusty) |
Moved password and users into the config directory to allow for instance specific users and passwords. Ran into a case where two different teams were sharing the same server hardware and needed this type of differentiation. Surprised that we haven't hit this sooner. Also defaulted mkdirs to use the numeric ports since this is the most common installation. |
||
#13 | 19965 | C. Thomas Tyler |
The Ultimate Perforce Login Script. Enhanced p4login v3.1.0: * If on a replica/edge server, logs in replication service users. * Uses 'p4 login -a' for non-service type accounts, and 'p4 login 'for service type accounts. * Accounts for auth.id, and behaves appropriately whether auth.id is set or not, e.g. doing an extra 'p4 login' as needed if auth.id isn't set. * Logs in external automaiton users, e.g. trigger or broker filter script users, if defined by the SDP_AUTOMATION_USERS setting in /p4/common/config/p4_<instance>.vars. * Fixed bug where it broke if p4_vars wasn't sourced. It now sources p4_vars. * Logs whether actual 'p4 login' commands were needed and the number of logins done, along with other cosmetic logging enhancements. * Added Version identifier. |
||
#12 | 19314 | Russell C. Jackson (Rusty) |
Change p4verify.sh to use -S to verify shelves on a replica instead of printing the files on the shelf. Removed the HOST_IP settings from mkdirs and instance_vars since it causes problems in a shared depotdata environment, and it was a temporary fix to work around a bug with auth.id that is being fixed. |
||
#11 | 18969 | C. Thomas Tyler |
Just moved the P4DTG setting from between some P4Web settings. This is a minor code cleanup/readability thing with no function impact. |
||
#10 | 18925 | C. Thomas Tyler |
Enhanced p4_vars.template to support operating on an SDP-managed host where no p4d process runs, such as a p4p (proxy) host, where no /p4/n/root/server.id file exists. The $SERVERID value will be empty (but defined) in this case. Enhanced to better support operating with replicas that share /depotdata with their master servers, by making P4TRUST and P4TICKETS values contain $SERVERID. Moved SHAREDDATA from p4_vars to instance_vars, since it is not inherently a global setting. In sophisticated enterprise environments, it can vary on a per-replica basis. Adjusted mkdirs.sh accordingly. |
||
#9 | 18852 | C. Thomas Tyler | Enhanced P4DTG config to be more SDP-ified. | ||
#8 | 16335 | C. Thomas Tyler |
Routine Merge Down to dev from main using: p4 merge -b perforce_software-sdp-dev |
||
#7 | 14136 | C. Thomas Tyler |
Routine merge down to dev from main for SDP using perforce_software-sdp-dev. |
||
#6 | 13906 | C. Thomas Tyler |
Normalized P4INSTANCE to SDP_INSTANCE to get Unix/Windows implementations in sync. Reasons: 1. Things that interact with SDP in both Unix and Windows environments shoudn't have to account for this obscure SDP difference between Unix and Windows. (I came across this doing CBD work). 2. The Windows and Unix scripts have different variable names for defining the same concept, the SDP instance. Unix uses P4INSTANCE, while Windows uses SDP_INSTANCE. 3. This instance tag, a data set identifier, is an SDP concept. I prefer the SDP_INSTANCE name over P4INSTANCE, so I prpose to normalize to SDP_INSTANCE. 4. The P4INSTANCE name makes it look like a setting that might be recognized by the p4d itself, which it is not. (There are other such things such as P4SERVER that could perhaps be renamed as a separate task; but I'm not sure we want to totally disallow the P4 prefix for variable names. It looks too right to be wrong in same cases, like P4BIN and P4DBIN. That's a discussion for another day, outside the scope of this task). Meanwhile: * Fixed a bug in the Windows 2013.3 upgrade script that was referencing undefined P4INSTANCE, as the Windows environment defined only SDP_INSTANCE. * Had P4INSTANCE been removed completely, this change would likely cause trouble for users doing updates for existing SDP installations. So, though it involves slight technical debt, I opted to keep a redundant definition of P4INSTANCE in p4_vars.template, with comments indicating SDP_INSTANCE should be used in favor of P4INSTANCE, with a warning that P4INSTANCE may go away in a future release. This should avoid unnecessary upgrade pain. * In mkdirs.sh, the varialbe name was INSTANCE rather than SDP_INSTANCE. I changed that as well. That required manual change rather than sub/replace to avoid corrupting other similar varialbe names (e.g. MASTERINSTANCE). This is a trivial change technically (a substitute/replace, plus tweaks in p4_vars.template), but impacts many files. |
||
#5 | 12923 | C. Thomas Tyler |
Routine merge down from main to dev. Resolved with 'p4 resolve -as', no merges or conflicts. |
||
#4 | 12028 | C. Thomas Tyler | Refreshed SDP dev branch, merging down from main. | ||
#3 | 11523 | Russell C. Jackson (Rusty) |
Modified P4BROKERPORTNUM to just be the port number and added P4BROKERPORT to instance_vars to be consistent with P4PORT. Also makes it easier to modify p4review.py to use P4BROKERPORT rather than P4PORT for the subject line when needed. |
||
#2 | 11490 | Russell C. Jackson (Rusty) |
Added SSL_PREFIX back and P4MASTERPORTNUM in order to support the sync_replica.sh and weekly_sync_replica.sh scripts. |
||
#1 | 11477 | Russell C. Jackson (Rusty) |
Updated to use /usr/bin/env python Added workshop header. Changed cfg to config. |
||
//guest/perforce_software/sdp/dev/Server/Unix/p4/common/cfg/instance_vars.template | |||||
#1 | 11466 | Russell C. Jackson (Rusty) |
Initial work to simplify p4_vars and remove cluster stuff. Testing of named instances surfaced some bugs that are in prod sdp, now fixed in dev. Added three triggers from RCJ SDP Moved p4review.cfg into the new /p4/common/cfg to go along with the instance_vars files. mkdirs.sh now generates an instance_p4review.cfg as well. Removed incremental p4verify to clean up a bit. It didn't support replicas and was really never used. All port settings now live in <instance>_vars file. You set what you want the ports to be in mkdirs.sh. There is no more fancy logic to try to guess what the port should be. You set it, and that is what it is. Remaining to do is to updated scripts to not need p4master_run. Saved that work for later since this is tested and works. |