= P4 Helm Management System (HMS) Tight Ship Management Perforce Professional Services :revnumber: v2023.1 :revdate: 2023-09-15 :doctype: book :icons: font :toc: :toclevels: 5 :sectnumlevels: 4 :xrefstyle: full // Keep the :sectnums: placement here in case needed. Section numbering isn't yet needed due to // the small size of this document. // :sectnums: == Introduction The P4 Helm Management System (HMS) has many potential features if fully deployed. The following discusses some aspects of HMS: * Bastion Host to Access Perforce P4 Servers * Tight Ship Management * Monitoring and Alerting * Host Configuration Management Each is discussed in this document. === Bastion Host to Access Perforce P4 Servers The HMS server machine, typically `hms.YourDevDomain.net` (where `YourDevDomain.net` is a domain reserved for internal services), is effectively a bastion host for Perforce P4 Servers, including the primary P4 commit server, any P4 replicas or edge servers, and P4 Broker and Proxy server machines. When logged in as the OS user perforce to the `hms`, all other hosts can be accessed without further authentication, as secure and trusted paths have been established with SSH keys. === Tight Ship Management ==== What is Tight Ship Management? Tight Ship management entails using version control so that changes to certain files that affect the operation of Perforce P4 Core software are versioned. A tiny p4d service, `p4d_hms`, separate from your other p4d services, is used for Tight Ship Management. The Server Deployment Package (SDP) scripts are versioned on various server machines with HMS, and thus Tight Ship Management is sometimes referred to as SDP Fleet Management. On each managed server machine, following are versioned: * The `/p4/common` directory tree * Any `/p4/N/bin` directory trees (where `N` is any SDP instance; there can be more than one on any given host) * The crontabs for the perforce OS user All Perforce P4 Server machines, including those running P4 Server (p4d), P4 Broker (p4broker) and/or P4 Proxy (p4p) services, are managed using Tight Ship Management. This means that changes to any files under the noted directories require interaction with version control, i.e. submitting changes made. A script running in a crontab checks for Tight Ship compliance. The script is: `sdp_sync`. It is governed by a config file that determines what hosts are managed, `sdp_hosts.cfg`. The script should not require local updates. The config file should be updated if the Perforce P4 topology changes, e.g. if a proxy is added or decommissioned. A goal with Tight Ship management is to achieve and maintain Tight Ship compliance. That essentially means that the `sdp_sync` script reports all is well, which means: * Each managed directory on all machines is in sync with what's in version control, with any discrepancies reported. * No files are checked out, with any discrepancies reported. * The live crontabs on each machine match the related versioned files. * The versioned crontabs are stored in `/p4/common/etc/cron.d`, and are named `crontab.perforce.short_hostname`, e.g. `crontab.perforce.p4d-usw2-01`. The `sdp_sync` script will email a compliance report daily. ==== How to work in a Tight Ship environment **Setup Personal P4CONFIG files** There are two ways to work in the Tight Ship environment: You can use the default perforce P4USER user account, or your own personal P4USER account. For accountability, you should use your personal account whenever possible. On each server machine A P4CONFIG file (see link:https://www.perforce.com/manuals/p4guide/Content/P4Guide/configuration.process.configfiles.html[docs for P4CONFIG]) exists for using HMS. By default, when you login as perforce, the default shell environment is not setup for managing HMS, but instead for working with other p4d instances. To switch any given terminal shell session into HMS management mode, simply set the P4CONFIG variable. To act as the perforce OS user (recommended only if you are doing read-only actions), do: export P4CONFIG=/p4/.p4config.SDP Before you act as yourself, make sure you have your own P4CONFIG file, and reference it, which will use a lowercase **sdp** (where the one perforce users uppercase **SDP**), and look like this sample: export P4CONFIG=/p4/.p4config.sdp.jsmith The file should look like this sample: P4PORT=ssl:hms.YourDevDomain.net:7467 P4USER=jsmith P4CLIENT=sdp.p4d-usw2-01 P4TICKETS=/p4/hms/.p4tickets P4TRUST=/p4/hms/.p4trust P4IGNORE=.p4ignore Once this file exists and the `export P4CONFIG` command is done to reference it, you will need to do a `p4 login` command. Then you can use normal Perforce P4 user commands to open files for edit and submit. For example, you might need to update the file `/p4/common/site/hms/config/sdp_hosts.cfg`, and could do so like this sample: export P4CONFIG=/p4/.p4config.sdp.jsmith cd /p4/common/site/hms/config p4 login p4 edit sdp_hosts.cfg vi sdp_hosts.cfg ## Make your changes. p4 submit -d "Added new proxy." sdp_host.cfg ===== Example: Updated crontab In this example, we'll make an update to the crontab on the p4d-usw2-01 server machine. Let's say you have a specific task mind, such as changing the schedule of a cron job. Generally, you'll start everything by first logging in as perforce@hms.YourCorpDomain.net, and then from there ssh to any other server in the environment. First, SSH to the machine and get setup to use HMS: ssh p4d-usw2-01 export P4CONFIG=/p4/.p4config.sdp.jsmith Then get into the directory where crontabs are, and set a shell variable C for convenience, and check out the file. cd /p4/common/etc/cron.d C=crontab.$USER.$(hostname -s) p4 edit $C Next, get the crontab from the machine into the local file, and be sure there are no diffs reported; there shouldn't be at this point. crontab -l > $C p4 diff Then edit the file, make your changes, diff them, upload the local file back into cron, and submit the versioned file: vi $C # Make your changes p4 diff crontab $C p4 submit -d "Adjusted timing of proxy cache cleaning." $C That's it. This will get the job done and maintain Tight Ship compliance. === Host Configuration Management Host Configuration Management (HCM) is essentially how to work with HMS when operating as the OS root user rather than perforce. This is used to version specific changes to key system files owned by the root user, and does not apply to files in or under the `/p4` directory tree. When logged in as the root user on the hms server machine (and only on this machine), sparse version management is used. That is, only a hand-picked set of individual files are versioned, not entire directory trees. To see the set of files versioned in this way, you can run the command: p4 files //... This will show files under the path `//streams/main/...`, which corresponds to the `/` directory, the root of the filesystem. For example, if the file `//streams/main/etc/hosts` is shown, that indicates that the `/etc/hosts` file is versioned. When working as root, there is no live running p4d service to connect to as there is with HMS. Instead, the P4 native DVCS features are used. This behaves much like operating with Git on a personal project – you are working on a private set of files, with no live running service. As far as interactions go and the p4 commands you use to edit and submit, nothing is different. There are operational differences of Host CM from regular HMS management: * You don't need to change the P4CONFIG file, as it is already set in the root user's .bashrc and need never change. * After you make and submit changes to any files, you need to run the command `p4 push` to get them into the HMS server. * Because there is no live running p4d service when using P4 native DVCS features, you can do versioning as root even if the service is not online. However, the `p4d_hms` service must be running in order to do a `p4 push` from the Host CM server to the HMS server. Using Host CM has some significant benefits, which are generally the benefits of version control, but applied to specific files on the `hms` server machine. Mainly, it is used for: * Managing P4Prometheus and Alert Manager files, e.g. to make changes to alerting rules. * Changing other files like `/etc/hosts`, /etc/grafana/grafana.ini`, and `/etc/alertmanager/alertmanager.yml`. * With versioning, you have the knowledge of knowing what things in key files changed, when, and why (in the form of a change description). This is particularly useful for complex configuration files like `grafana.ini`. And you can easily rollback files to earlier versions if needed. Host Versioning can be augmented by adding a P4 Code Review server to the HMS server, to add notification and optionally code review for changes to any of the files managed with Host CM. See the file `/root/HostCM/ReadMe.md` on the `hms` server machine for additional detail on using Host CM. === Monitoring and Alerting Prometheus is an open source project used for general purposes systems monitoring. (We actually use Victoria Metrics, a fork of Prometheus). P4Prometheus is the extension to Prometheus provided by Perforce Software that adds knowledge of additional things to monitor on P4 server machines. Grafana is a software tool that adds useful visualizations via a web interface to the Prometheus data. The Prometheus, Prometheus AlertManager, and Grafana services all run on the hms server machine, and are managed with `systemctl`, with these service names (illustrated with status commands): * `systemctl status prometheus` * `systemctl status grafana-server` * `systemctl status alertmanager` The P4Prometheus software is installed and operating on various managed p4d server machines. On each, there is a `p4prometheus` service and a `node_exporter` service, managed with `systemctl`. ==== Alerting Notification Receivers and Alerting Rules Alerting rules may need to be adjusted from time to time. The alerting rules are defined in 2 files: * Notification receivers, i.e. who gets emails, are defined in /etc/alertmanager/alertmanager.yml * Alerting rules are defined in /etc/prometheus/perforce_rules.yml ==== Changing Alerting Notification Receivers To change the receivers, use the following procedure. You will need to be familiar with the format of the `alertmanager.yml` file; see links:https://github.com/prometheus/alertmanager/blob/main/docs/alertmanager.md[docs here]. Its format is fairly intuitive and you may decide you don't need additional documentation once you see it. To edit the rules, do like so: Login as root@hms.YourCorpDomain.net cd /etc/alertmanager p4 edit alertmanager.yml vi alertmanager.yml # Make your changes make The `make` command will perform a validation of your changes. If that indicates success, then restart the service for your changes to take effect, like so: make restart TIP: As a shortcut, you can do only the `make restart`, and skip the separate call to `make` to do only validation. The `make restart` will do the validation first and, if that is successful, restart the service. Then version the changes in the HostCM server, and push to HMS: p4 submit -d "Your change description" alertmanager.yml p4 push ==== Changing Alerting Rules To change the alerting rules, use the following procedure. You will need to be familiar with the format rules file; see link:https://github.com/prometheus/alertmanager/blob/main/docs/configuration.md[docs here]. Its format is fairly intuitive and you may decide you don't need additional documentation once you see it. To edit the rules, do like so: Login as root@hms.YourCorpDomain.net cd /etc/prometheus p4 edit perforce_rules.yml vi perforce_rules.yml # Make your changes make The make command will perform a validation of your changes. If that indicates success, then restart the service for your changes to take effect, like so: make restart TIP: As a shortcut, you can do only the `make restart`, and skip the separate call to `make` to do only validation. The `make restart` will do the validation first and, if that is successful, restart the service. Then version the changes in the HostCM server, and push to hms: p4 submit -d "Your change description" perforce_rules.yml p4 push