# Ansible SDP

Ansible automation for deploying and managing Perforce Helix Core servers using the [Server Deployment Package (SDP)](https://swarm.workshop.perforce.com/projects/perforce-software-sdp) framework.

## Table of Contents

- [Overview](#overview)
- [What Gets Installed](#what-gets-installed)
- [Prerequisites](#prerequisites)
- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Files You Must Replace](#files-you-must-replace)
- [Passwords and Secrets](#passwords-and-secrets)
- [Inventory Configuration](#inventory-configuration)
- [Helper Scripts](#helper-scripts)
- [Flow Control Variables](#flow-control-variables)
- [Server Types](#server-types)
- [Monitoring](#monitoring)
- [Cron Jobs](#cron-jobs)

## Overview

This project automates the full lifecycle of Perforce Helix Core deployments:

- **Install** -- Fresh SDP installation with p4d, p4broker, pyenv/Python, and p4python
- **Update** -- Upgrade p4/p4d binaries and SDP scripts
- **Monitor** -- Deploy p4prometheus and node_exporter metrics
- **Maintain** -- Manage cron jobs, SSL certificates, sudoers, and system dependencies

Supported operating systems: **Ubuntu/Debian** and **RHEL/CentOS**.

## What Gets Installed

### System Packages

Common packages installed on all servers:

`atop`, `python3`, `python3-setuptools`, `screen`, `ca-certificates`, `curl`, `htop`, `rsync`, `util-linux`, `jq`, `iperf3`, `openssl`, `acl`, `vim`, `net-tools`, `make`, `wget`, `git`

Additional Debian packages include build dependencies for pyenv, SSH server, PHP, `rdiff-backup`, `mailutils`, and more. See `roles/perforce-sdp-install/defaults/main.yml` for the full list.

Apache2 is **removed** as part of the installation.

### Perforce Components

| Component | Description |
|-----------|-------------|
| **SDP** | Server Deployment Package framework (downloaded from Perforce Workshop) |
| **p4d** | Perforce server daemon |
| **p4** | Perforce command-line client |
| **p4broker** | Perforce broker (optional, controlled by `install_broker`) |
| **p4python** | Python bindings for the Perforce API |

Binary versions are controlled by `perforce_version` and `perforce_broker_version` variables.

### Python Environment

- **pyenv** is installed under the `perforce` OS user (`/p4/.pyenv`)
- The latest stable Python 3.x is installed automatically via pyenv
- **p4python** is installed via pip in the pyenv environment

### System Configuration

- **Perforce OS user/group** created with configurable UID/GID (default: 10666)
- **ansibleuser** created for automation with passwordless sudo
- **Dummy network interface** for multi-homed licensing (optional, controlled by `dummy_interface`)
- **Transparent Huge Pages** disabled (recommended for database workloads)
- **sysctl tuning** for network performance (TCP buffer sizes, BBR congestion control)
- **systemd service** for p4d with automatic startup

## Prerequisites

- Ansible 2.4+ (2.9+ for the monitoring role)
- SSH access to target servers with a user that has sudo privileges
- Target servers must have the required storage volumes pre-mounted
- If using NFS for depots, NFS mounts must be pre-configured
- DNS names must resolve to valid IPs before running the playbook

## Project Structure

```
ansible-sdp/
├── main-playbook.yml              # Primary playbook
├── ansible.cfg                    # Ansible configuration
├── password.txt                   # Vault password (REPLACE - see Secrets section)
├── inventories/
│   ├── p4-sdp-install.yml         # Inventory file
│   ├── group_vars/central/        # Group-level variables
│   └── host_vars/                 # Per-host variables
├── roles/
│   ├── perforce-sdp-install/      # Installation and configuration role
│   │   ├── defaults/main.yml      # Default variable values
│   │   ├── tasks/                 # Task files (see below)
│   │   ├── templates/             # Jinja2 templates
│   │   ├── files/                 # Static files copied to targets
│   │   └── handlers/main.yml     # Service restart handlers
│   └── perforce-sdp-monitoring/   # Monitoring role (p4prometheus)
├── swarm/                         # Helix Swarm config files (manual setup)
└── *.sh                           # Helper scripts
```

### Task Files

| Task File | Triggered By | Purpose |
|-----------|-------------|---------|
| `dependencies.yml` | `install_perforce` | System packages, OS user, pyenv, p4python |
| `install.yml` | `install_perforce` (when `new_sdp: true`) | SDP download, mkdirs.sh, p4d service, broker |
| `update.yml` | `update_perforce` | Binary upgrades, SDP update, DB upgrade |
| `cron.yml` | `install_perforce` or `update_cron` | Scheduled maintenance jobs |
| `sudo.yml` | `install_perforce` or `update_sudo` | Sudoers configuration |
| `cert.yml` | `update_cert` | SSL certificate deployment |
| `install_broker.yml` | `install_perforce` (when `install_broker: true`) | P4Broker setup |

## Quick Start

1. Clone this repository
2. Replace placeholder files (see [Files You Must Replace](#files-you-must-replace))
3. Configure secrets (see [Passwords and Secrets](#passwords-and-secrets))
4. Create or modify inventory and host variables for your environment
5. Run the installation:

```bash
# Test connectivity first
./ping.sh <server-or-group>

# Install
./install.sh <server-or-group>
```

## Files You Must Replace

The following files in `roles/perforce-sdp-install/files/` are **placeholders** that must be replaced with real files before a production deployment:

### SSL Certificate and Private Key

| File | Destination on Target | Description |
|------|----------------------|-------------|
| `certificate.txt` | `/p4/ssl/certificate.txt` | SSL/TLS certificate for p4broker |
| `privatekey.txt` | `/p4/ssl/privatekey.txt` | Corresponding SSL/TLS private key |

Replace these with your organization's valid SSL certificate and key. The broker will fail to start with SSL enabled if these are not valid. These files are deployed by the `cert.yml` task.

### Perforce License

| File | Destination on Target | Description |
|------|----------------------|-------------|
| `perforce-license` | `/p4/1/root/license` | Perforce server license file |

Replace with a valid license obtained from Perforce. The license copy is controlled by the `copy_license` host variable. Without a valid license, the server will run in limited mode.

### SSH Private Key

| File | Destination on Target | Description |
|------|----------------------|-------------|
| `id_rsa` | `/p4/.ssh/id_rsa` | SSH private key for the perforce user |

This key is used for replication (SDP scripts use SSH to communicate between master and replicas). The file is encrypted with Ansible Vault. To replace it:

```bash
# Encrypt your SSH private key with Ansible Vault
ansible-vault encrypt /path/to/your/id_rsa --vault-password-file password.txt

# Copy the encrypted file to roles/perforce-sdp-install/files/id_rsa
```

The corresponding public key must be added to `authorized_keys` on all replica/standby servers.

## Passwords and Secrets

### Passwords That Must Be Changed

The following passwords are defined in `inventories/group_vars/central/main.yml` with **example values that must be changed for production**:

| Variable | Current Value | Purpose | How to Change |
|----------|--------------|---------|---------------|
| `perforce_admin_user_pass` | `F@stSCM!` | Perforce admin (`p4admin`) password set during initial server setup | Use Ansible Vault (see below) |
| `perforce_user_password` | `$6$PerforceRules$...` | Hashed password for the `perforce` OS user | Generate with `openssl passwd -6 -salt <salt>` and vault-encrypt |

### Using Ansible Vault for Secrets

The recommended approach is to vault-encrypt sensitive variables rather than storing them in plaintext. Create a vault-encrypted variable file:

```bash
# Create an encrypted vars file for your secrets
ansible-vault create inventories/group_vars/central/vault.yml --vault-password-file password.txt
```

Add your secret variables to the vault file:

```yaml
perforce_admin_user_pass: "YourActualSecurePassword"
perforce_user_password: "$6$yoursalt$yourhash"
```

### Vault Password File

The file `password.txt` in the project root contains the Ansible Vault decryption password. For production:

- Replace the contents with a strong password
- Ensure the file is **not** committed to version control (add to `.gitignore`)
- Alternatively, remove it and use `--ask-vault-pass` when running playbooks

### Encrypted Files

The SSH key `roles/perforce-sdp-install/files/id_rsa` is encrypted with Ansible Vault. It is automatically decrypted during playbook execution when the vault password is provided.

## Inventory Configuration

### Inventory File

Inventory files live in `inventories/` and follow the naming convention:

```
inventories/perforce_sdp-inv-<function|team>-<region>.yml
```

Servers are organized into groups that determine their role:

```yaml
all:
  children:
    central:
      children:
        edgeservers:
          hosts:
            my-edge-server:
        standbys:
          hosts:
            my-standby-server:
        commit:
          hosts:
            my-commit-server:
```

### Host Variables

Each host needs a variables file at `inventories/host_vars/<hostname>.yml`. Required variables:

```yaml
# Server identity
perforce_dnsname: "my-server"              # DNS name for this server
perforce_server_type: "p4d_master"         # See Server Types section

# Volume paths (without leading slash)
perforce_online_metadata_volume: "mnt/p4meta"
perforce_offline_metadata_volume: "mnt/p4meta"
perforce_data_volume: "mnt/p4data"
perforce_common_volume: "mnt/p4data"
perforce_log_volume: "mnt/p4meta"

# Feature flags
perforce_clean: false          # Set true to wipe and reinstall (DESTRUCTIVE)
dummy_interface: true          # Create dummy network interface for licensing
copy_license: true             # Copy license file to server
perforce_depots_use_nfs: false # Set true if depots are on NFS

# Replication (for master servers)
perforce_standby_dnsname: "my-standby"
perforce_master_id: "master"
perforce_replica_id: "standby2"
p4serviceuser: "svc_master"
```

### Group Variables

Group-level variables in `inventories/group_vars/central/main.yml` apply to all servers and include network settings, Perforce version, user/group configuration, and credentials.

## Helper Scripts

All scripts take a server name or group name as the first argument:

| Script | Purpose | Example |
|--------|---------|---------|
| `install.sh` | Full SDP installation | `./install.sh perforce-commit` |
| `upgrade.sh` | Update p4/p4d binaries and SDP | `./upgrade.sh edgeservers` |
| `depend.sh` | Update system dependencies only | `./depend.sh perforce-commit` |
| `monitor.sh` | Install monitoring stack | `./monitor.sh central` |
| `cron.sh` | Update cron jobs | `./cron.sh edgeservers` |
| `cert.sh` | Deploy new SSL certificates | `./cert.sh central` |
| `sudo.sh` | Update sudoers configuration | `./sudo.sh perforce-commit` |
| `ping.sh` | Test Ansible connectivity | `./ping.sh central` |

## Flow Control Variables

These variables control which tasks run and are passed via `--extra-vars` at runtime. They all default to `false`.

| Variable | Description |
|----------|-------------|
| `install_perforce` | Full installation (dependencies, SDP, broker, cron, sudo) |
| `update_perforce` | Update p4/p4d binaries and SDP scripts |
| `monitor_perforce` | Install p4prometheus monitoring |
| `update_depend` | Update system dependencies only |
| `update_cron` | Update cron jobs only |
| `update_sudo` | Update sudoers configuration only |
| `update_cert` | Update SSL certificates only |

Example:

```bash
ansible-playbook main-playbook.yml \
  --extra-vars '{"install_perforce":true}' \
  --vault-password-file password.txt \
  --limit perforce-commit
```

## Server Types

Set `perforce_server_type` in host variables to one of:

| Type | Description | Cron Profile |
|------|-------------|-------------|
| `p4d_master` | Commit/master server | Journal rotation, checkpoints, shelf verify, maintenance |
| `p4d_edge` | Edge server | Checkpoints, cache clean, replica status |
| `p4d_replica` | Read-only replica | Sync replica, DB recreate, replica status |
| `p4d_standby` | Standby for failover | Same as replica |
| `p4d_edgerep` | Edge replica | Same as replica |
| `p4broker` | Standalone broker | Log cleanup |
| `p4proxy` | Proxy server | -- |

## Monitoring

The `perforce-sdp-monitoring` role installs:

- **p4prometheus** -- Scrapes Perforce structured logs and exposes metrics
- **Network monitoring** -- Cron job that pings the commit server and records latency
- **Location metrics** -- Prometheus labels for server location (city, country, coordinates)

Prerequisites: `node_exporter` must already be installed on the target.

Run with:

```bash
./monitor.sh <server-or-group>
```

See `roles/perforce-sdp-monitoring/README.md` for variable reference.

## Cron Jobs

Cron jobs are configured per server type. Key schedules:

### Master Servers
- Every 2 hours: Journal rotation
- Every 10 minutes: Update limits
- Every 15 minutes: Shelf verification
- Weekly (Saturday 5:05 AM): Checkpoint
- Weekly (Saturday 6:00 AM): Maintenance

### Edge Servers
- Daily (6:05 AM): Checkpoint
- Weekly (Saturday 6:00 AM): Maintenance
- Every 6 minutes: Replica status check

### Replica/Standby Servers
- Daily (12:00 PM): Sync replica
- Monthly (first Saturday 10:05 PM): Recreate DB sync
- Every 6 minutes: Replica status check

### All Servers
- Daily (1:00 AM): Prometheus metrics cleanup