= Server Deployment Package (SDP) for Perforce Helix: Unsupported Scripts and Triggers Perforce Professional Services :revnumber: v2019.3 :revdate: 2020-08-21 :doctype: book :icons: font :toc: :toclevels: 3 :xrefstyle: full == Preface This page contains documentation for the SDP Unsupported folder. This folder contains sample scripts that have proven useful in the past. However, they are not maintained nor included in the automated test suite. Unlike the rest of the SDP, these scripts here in this Unsupported folder are NOT officially supported. They are NOT fully tested and NOT guaranteed to work. Treat files in here with some caution, and be sure to test before using in your own environment. All triggers and scripts in the `Unsupported` folder are provided as examples. They are Community supported only. Most have worked in customer environments at some point, but they are not maintained and tested at the same quality levels as the main SDP scripts. *Please Give Us Feedback* Perforce welcomes feedback from our users. Please send any suggestions for improving this document or the SDP to consulting@perforce.com. :sectnums: == Samples Scripts and utilities in this folder are examples which were part of the SDP in the past. === bin/htd_move_logs.sh Script to compress and move Helix Server structured audit logs Implementation assumptions and suggestions: * Assumes the rotated log files are named audit-nnn.csv * Do NOT configure your log files to be placed in $P4ROOT * Set TARGETDIR in script === bin/p4web_base P4Web base init script for running a p4web instance. Similar function for `/p4/common/bin/p4d_base`. Please refer to SDP_Guide.Unix for more details. Can be used from service files or even systemd service definitions. This is located here because *P4Web* is a deprecated and unsupported product. === broker/one_per_user.sh This broker filter script limits commands of a given type to one process running at a time for the same user. It relies on `p4 monitor` having been enabled with `p4 configure set monitor=1` (or higher than 1). This is called by the `p4broker` process. The p4broker provides an array of fields on STDIN, e.g. "command: populate" and "user: joe", which get parsed to inform the business logic of this script. This is enabled by adding a block like the following to a broker config file, with this example limiting the `p4 populate` command: command: ^populate$ { action = filter; execute = /p4/common/bin/broker/one_per_user.sh; } === Triggers ==== Workflow Enforcement Triggers These triggers are documented in link:WorkflowEnforcementTriggers.html[HTML doc] or link:WorkflowEnforcementTriggers.pdf[PDF doc] Where appropriate below reference will be made to this section. ==== CheckCaseTrigger.py This trigger checks for attempts to add new files (including via branching or renaming) which only differ in case in *some part of their path* from existing files. This avoids issues in these scenarios: * for a case insensitive server: ** Windows clients (case insensitive) can accidentally create new directories or files which sync into different directoroes on Unix (case sensitive) * for a case sensitive server: ** Unix clients can create 2 (or more) files which only differ in case, and when synced to a Windows client, only one of them appears [source] .Usage ---- include::../Samples/triggers/CheckCaseTrigger.py[tags=includeManual] ---- ==== CheckChangeDesc.py This trigger parses change descriptions to ensure that they only have the required format. See source for an example of Python regexes. [source] .Usage ---- include::../Samples/triggers/CheckChangeDesc.py[tags=includeManual] ---- ==== CheckFixes.py Part of <<_workflow_enforcement_triggers>> This trigger is intended for use with P4DTG (Defect Tracking Replication) installations. It will allows fixes to be added or deleted depending on the values of job fields. Thus you can control workflow and disallow fixes if jobs are in a particular state. So if the field JiraStatus is closed, then you are not allowed to add or delete a fix. [source] .Usage ---- include::../Samples/triggers/CheckFixes.py[tags=includeManual] ---- ==== CheckFixes.yaml Sample YAML config file for <<_checkfixes_py>> ==== CheckFolderStructure.py Part of <<_workflow_enforcement_triggers>> This trigger will ensure that any attempt to add (or branch) a file does not create a new folder name at specific levels. With new streams protections options in p4d 2020.1 this can be removed. [source] .Usage ---- include::../Samples/triggers/CheckFolderStructure.py[tags=includeManual] ---- ==== CheckJobEditTrigger.py For enforcment of job editing with P4DTG usage (e.g. JIRA). [source] .Usage ---- include::../Samples/triggers/CheckJobEditTrigger.py[tags=includeManual] ---- ==== CheckStreamNameFormat.py [source] .Usage ---- include::../Samples/triggers/CheckStreamNameFormat.py[tags=includeManual] ---- ==== CheckSubmitHasReview.py Part of <<_workflow_enforcement_triggers>> Since Swarm 2019.1 this trigger is no longer required since it can be replaced with Swarm's Workflow functionality. [source] .Usage ---- include::../Samples/triggers/CheckSubmitHasReview.py[tags=includeManual] ---- ==== ControlStreamCreation.py [source] .Usage ---- include::../Samples/triggers/ControlStreamCreation.py[tags=includeManual] ---- ==== CreateSwarmReview.py Part of <<_workflow_enforcement_triggers>> This trigger creates Swarm reviews automatically for committed changes. It is deprecated since Swarm now supports this as part of its workflow. [source] .Usage ---- include::../Samples/triggers/CreateSwarmReview.py[tags=includeManual] ---- ==== DefaultChangeDesc.py Creates a default changelist description. [source] .Usage ---- include::../Samples/triggers/DefaultChangeDesc.py[tags=includeManual] ---- ==== DefaultSwarmReviewDesc.py Updates Swarm review changelist with suitable template text. This can encourage users to record information against the review, or use a checklist. Part of <<_workflow_enforcement_triggers>> [source] .Usage ---- include::../Samples/triggers/DefaultSwarmReviewDesc.py[tags=includeManual] ---- ==== DefaultSwarmReviewDesc.yaml Example YAML config file for <<_defaultswarmreviewdesc_py>> ==== JobIncrement.pl Trigger to increment jobs with custom names. Enable this as a form-in trigger on the job form, for example: Triggers: JobIncrement form-in job "/p4/common/bin/triggers/JobIncrement.pl %formfile%" The default job naming convention with Perforce Jobs has an auto-increment feature, so that if you create job with a `Job:` field value of `new`, it will be changed to jobNNNNNN, with the six-digit value being automatically incremented. Perforce jobs also support custom job naming, e.g. to name jobs PROJX-123, where the name itself is more meaningful. But if you use custom job names, you forego the convenience of automatic generation of a new job number. Now typically, if the default job naming feature isn't used, it's because new issues originate jn an external issue tracking, so there's no need for incrementing by Perforce; the custom job names just mirror the value in the external system. This script is aims to make it easier to use custom job names with Perforce even when there is no external issue tracker integration, by providing the ability to generate new job names automatically. The `Project:` field in the Jobspec has a `select` field with pre-defined values for project names. Projects desiring to use custom jobs names will define a counter named JobPrefix-, with the value being a tag name, a short form of the project name, to be used as a prefix for job names. For example, a project named joe_schmoe-wicked-good-thing might have a prefix of WGT. Jobs will be named WGT-1, WGT-2, etc. By convention, job prefixes are comprised only of uppercase letters, matching the pattern ^[A-Z]$. No spaces, commas, dashes, underbars, etc. allowed. (There is no mechanicm for mechanical enforcement of this convention, nor none needed, as tags are defined and created by Perforce Admins). To define a prefix for a project, an admin define a value for the appropriate counter, e.g. by doing: p4 counter JobPrefix-some-cool-project SCT *High Number Counter* For projects with defined tags, there will also be a high number counter tracking the highest numbered job with a give prefix. This counter is created automatically and maintained by this script. This trigger script fires as a `form-in` trigger on job forms, i.e. it fires on jobs that are on their way into the server. If `Job:`` field value is `new` and the `Project:` field value has an associated JobPrefix counter, then the name of the job is determined and set by incrementing the High Number counter, ultimately replacing the value `new` with something like SCT-201 before it ever gets to the server. If no High Number counter exists for the project, it gets set to `1`. Usage: JobIncrement.pl -h|-man Display a usage message. The `-h` display a short synopsis only, while `-man` displays this message. Return status: Zero indicates normal completion, Non-Zero indicates an error. ==== JobsCmdFilter.py This script is designed to run as a jobs command filter trigger on the server. It ensures that multiple wildcards are not specified in any search string (which can cause the server to perform excessive processing and impact performance) Usage: jobs-cmd-filter command pre-user-jobs "/p4/common/bin/triggers/JobsCmdFilter.py %args%" ==== P4Triggers.py Base class for many of the Python triggers. ==== PreventWsNonAscii.py This script is designed to run as a form save trigger on the server. It will cause a workspace save to fail if any non-ascii characters are present in the workspace spec. It also will block odd characters from the workspace name. Usage: PreventWSNonASCII form-save client "/p4/common/bin/triggers/PreventWsNonAscii.py %formfile%" ==== RequireJob.py Allows admins to require jobs to be associated with all submits in particular areas of repository. Part of <<_workflow_enforcement_triggers>> [source] .Usage ---- include::../Samples/triggers/RequireJob.py[tags=includeManual] ---- ==== SetLabelOptions.py This script is designed to run as a form-in and form-out trigger on the server. It sets the autoreload option for static labels and doesn't allow the user to change it. Usage: setlabelopts form-out label "/p4/common/bin/triggers/SetLabelOptions.py %formfile%" setlabelopts form-in label "/p4/common/bin/triggers/SetLabelOptions.py %formfile%" ==== SwarmReviewTemplate.py Part of <<_workflow_enforcement_triggers>> [source] .Usage ---- include::../Samples/triggers/SwarmReviewTemplate.py[tags=includeManual] ---- ==== TFSJobCheck.py Simple trigger to ensure jobs always contain a particular string. Note that <<_checkchangedesc_py>> is a more general form of this trigger. Trigger table entry: TFSJobCheck change-submit //depot/path/... "/p4/common/bin/triggers/TFSJobCheck.py %changelist%" ==== ValidateContentFormat.py This trigger is intended for file content validation as part of change-content trigger Works for YAML, XML - only checks files matching required file extensions - see below. Particularly useful to ensure that `Workflow.yaml` file itself doesn't get accidentally corrupted! Part of <<_workflow_enforcement_triggers>> [source] .Usage ---- include::../Samples/triggers/ValidateContentFormat.py[tags=includeManual] ---- ==== Workflow.yaml Sample generic configuration file for use with <<_workflow_enforcement_triggers>> ==== WorkflowTriggers.py Base class for use by various Workflow triggers as per <<_workflow_enforcement_triggers>> Imports <<_p4triggers_py>> ==== archive_long_name_trigger.pl This script unconverts #@*% from ascii in archive files. The reason for doing this is when they are expanded they can overflow the unix file path length. Usage: $file_prefix -op -rev -lbr < stdin Trigger: arch archive //... "/p4/common/bin/triggers/archive_long_name_trigger.pl -op %op% -lbr %file% -rev %rev%" ==== command_block.py This is command trigger to allow you to block commands from all but listed users. Trigger table entry examples: command-block command pre-user-obliterate "/p4/common/bin/triggers/command_block.py %user%" command-block command pre-user-(obliterate|protect$) "/p4/common/bin/triggers/command_block.py %user%" ==== dictionary This file contains dictionary translations for parsing requests and generating responses. Used by <<_rad_authcheck_py>> ==== externalcopy.txt Documents how to use externalcopy programs to transfer data between commit and edge. ==== otpauthcheck.py This trigger will check to see if the userid is in a the LOCL_PASSWD_FILE first and authenticate with that if found. If the users isn't in the local file, it checks to see if the user is a service user, and if so, it will authenticate against just the auth server. Finally, it will check the user's password and authenticator token if the other two conditions don't match. The trigger table entry is: authtrigger auth-check auth "/p4/common/bin/triggers/vip_authcheckTrigger.py %user% %serverport%" ==== otpkeygen.py This script generates a key for a user and stores the key in the Perforce server. Used together with <<_otpauthcheck_py>> ==== rad_authcheck.py This trigger will check to see if the userid is in a the LOCL_PASSWD_FILE first and authenticate with that if found. If the users isn't in the local file, it checks to see if the user is a service user, and if so, it will authenticate against LDAP. Finally, it will check the user against the Radius server if the other two conditions don't match. You need to install the python-pyrad package and python-six package for it to work. It also needs the file named dictionary in the same directory as the script. Set the Radius servers in RAD_SERVERS below Set the shared secret Pass in the user name as a parameter and the password from stdin. The trigger table entry is: authtrigger auth-check auth "/p4/common/bin/triggers/rad_authcheck.py %user% %serverport% %clientip%" NOTE: The script current is set such that the Perforce user names should match the RSA ID's. In the case of one customer, the RSA ID's were all numeric, so we made the Perforce usernames be realname_RSAID and had this script just strip off the realname_ part. Example commented out in the main function. ==== radtest.py This is a radius test script. You need to install the python-pyrad package and python-six package for it to work. It also needs the file named dictionary in the same directory as the script. Set the Radius servers in radsvrs below Set the shared secret Pass in the user name as a parameter and the password from stdin. ==== submit_form_1.py This script modifies the description of the change form for the Perforce users listed in the submit_form_1_users group. Trigger table entry: submitform1 form-out change "/p4/common/bin/triggers/submit_form_1.py %formfile% %user%" ==== submit_form_1_in.py This script checks the input of the description form for the path specified in the triggers table. Trigger table entry: submitform1_in change-submit //depot/somepath/... "/p4/common/bin/triggers/submit_form_1_in.py %changelist% %user%" === triggers / tests This directory contains test harnesses for various triggers document in the section above. They import a common base class `p4testutils.py` and then run various unit tests. The tests are one of two types: * simple unit tests * integration tests using a real `p4d` instance (running in DVCS-style mode). The tests make it straight forward to ensure that you haven't broken any tests. You need to ensure you have a `p4d` executable in your PATH. Run any individual test: python3 TestRequireJob.py == Maintenance These are example scripts which have proven useful in the past. They are NOT fully tested and NOT guaranteed to work, although in most cases they are fairly simple and reliable. Treat with some caution!!! === accessdates.py This script is normally called by another script, such as <<_unload_clients_py>> However, if run standalone, it will generate 4 files with a list of specs that should be archived based on the number of weeks in <<_maintenance_cfg>> The file generated are: branches.txt clients.txt labels.txt users.txt === add_users.sh This script adds a bunch of users from a users_to_add.csv file of the form: ,,[,&1 | tee add_users.$(date +'%Y%m%d-%H%M').log === addusertogroup.py This script adds a user or users to the specified group. Usage: python addusertogroup.py [instance] user group * instance defaults to 1 if not given. * user = user_name or a file containing a list of user names, one per line. * group = name of Perforce group to add the user(s) to. === checkusers.py This script will generate a list of all user names that are listed in any group, but do not have a user account on the server. The results are written to `removeusersfromgroups.txt`. Usage: python checkusers.py You can pass that to `removeuserfromgroups.py` to do the cleanup. === checkusers_not_in_group.py This script will generate a list of all standard Perforce user names that are not listed in any group. It prints the results to the screen, so you may want to redirect it to a file. Usage: python checkusers_not_in_group.py === clean_protect.py This script will drop all lines in the protect tabel that have a group referenced from the file `groups.txt` passed into the script. The list of groups to drop is passed in as the first parameter and the protect table is passed in as the 2nd parameter. Usage: python protect_groups.py remove_groups.txt p4.protect `remove_groups.txt` is generated using <<_protect_groups_py>> - See that script for details. Run `p4 protect -o > p4.protect` to generate the protections table. You can redirect the output of this script to a file called `new.p4.protect` and then you can compare the original `p4.protect` and the `new.p4.protect`. If everything looks okay, you can update the protections table by running: p4 protect -i < new.p4.protect === convert_label_to_autoreload.py Converts label and sets `autoreload` option https://www.perforce.com/manuals/cmdref/Content/CmdRef/p4_label.html#Form_Fields_..503[see Command Reference] Usage: convert_label_to_autoreload.py