# P4 Health Check Examples and Process

This is a practical set of examples for a P4 health check. 

The appropriate design depends on recovery objectives, user locations,
latency, bandwidth, data volume, and the ability to operate the system during
an incident. These checks are prompts for investigation, not requirements that
every installation must implement.

This activity is performed together with the customer to achieve their stated goals. The results are recommendations.

## Service and Operations

- Confirm the commit or master server is reachable, correctly licensed, and
  operated by appropriately privileged administrators.
- Review server version, patch level, operating system support, and the P4
  Server Deployment Package layout where it is in use.
- Check CPU, memory, disk capacity, I/O latency, and network saturation during
  normal and peak activity.
- Review command failures, long-running commands, lock contention, and the
  workload from interactive users, automation, and build farms.
- Check that monitoring alerts cover P4 availability, disk space, replication,
  backup, checkpoint, journal, and verification failures.

## Backup and Recovery

- Verify that checkpoints, journals, and versioned archives are included in
  the backup and retention policy.
- Confirm that backups are copied to offline or remote storage and that their
  results are checked daily.
- Review recent `p4verify` results and where it runs. Frequent verification is
  CPU-intensive, so a redundant DR system may be a better location than the
  production server.
- Test the documented recovery procedure: restore a checkpoint, replay the
  required journal, and verify that the resulting metadata is usable.
- Confirm the defined recovery point objective (RPO) and recovery time
  objective (RTO) can be met with the actual replication and backup intervals.
- Check whether primary and recovery systems have documented differences in
  hardware, operating system, filesystem layout, capacity, or cloud region.

## Topology Inventory

For every server, identify its server ID, role, location, P4PORT, P4TARGET,
replication mode, archive policy, storage, and expected workload. Compare this
inventory with `p4 servers -J`, `p4 -ztag servers`, `p4 -ztag topology`, and
the generated topology JSON.

| Role | Typical purpose | Health-check focus |
| --- | --- | --- |
| Commit or standard server | Authoritative metadata and submits | Capacity, backups, journals, and central availability |
| Proxy | Caches versioned files on demand | Cache effectiveness, disk capacity, and connectivity to commit |
| Forwarding replica | Read/reporting offload and file cache | Replication lag, filtering, monitoring, and command routing |
| Read-only replica | HA/DR or read-only workload | Journal/copy health and recovery readiness |
| Standby replica | Manual failover candidate for commit or edge | Parity with primary, replication freshness, and tested failover |
| Edge server | Local workspace and work-in-progress operations | User latency, workspace-state protection, locks, and submit path |
| Build edge | Isolated build-farm activity | Automation load, reseed time, and confirmation that humans do not rely on it |

## Example Deployment Patterns

### Single Commit or Master Server

Suitable for a smaller installation or a well-specified server supporting a
large but centrally located user base. Review whether backups, restore testing,
and capacity planning provide an acceptable outage and data-loss profile.

### Remote Office Proxy

Use a proxy where users primarily need faster file transfers and can tolerate
dependence on the commit server. It has no metadata copy, so it does not speed
reporting and cannot serve users if the commit server connection fails. It is
typically more effective for larger files than many small files.

### Forwarding Replica for Read Offload

Use a forwarding replica for reporting and file caching near a user population
or workload. Confirm its metadata and archive filtering meet the intended use,
replication is current, and write/update actions correctly return to the commit
server.

### Edge Server for Distributed Users

Use an edge where local users need low-latency workspace operations. It handles
most normal workspace activity locally while submits go to the commit server.
Review distributed-lock behavior, submit latency, and protection for the edge's
workspace state. An edge is not a commit-server failover target; protect it
with its own standby if that state is important.

### Basic Disaster Recovery

Use a primary system and a separately located, preferably comparable, recovery
system with replication plus independent backup media. Review the last known
replication point, expected data loss after a site failure, DNS/failover steps,
and the ability to start the replacement server using known-good metadata.

### Warm Standby and Disaster Recovery

Use a primary, a local warm standby, and a remote DR system when a shorter
local recovery is needed alongside site-level recovery. Confirm that the team
understands the possible synchronization delay and does not assume this is
automated high availability.

### Clustered High Availability with DR

Use clustered local nodes with shared SAN/NAS storage for a short local outage,
then add a remote replicated DR system for site loss. Review storage as a
potential shared failure domain, the manual P4 restart procedure, and the
remote replication interval. NAS is not recommended for P4 metadata.

## Replication and Routing Checks

- A replica's P4TARGET should normally be the target server's P4PORT, not a
  `p4broker` P4PORT.
- Edge servers should target a forwarding replica or the commit server.
- Forwarding replicas should target a forwarding replica or a master/commit
  server. They can target an edge in supported configurations when configured
  specifically for read-only offload.
- Proxies cannot be chained and cannot target forwarding replicas.
- Do not put a non-forwarding replica between the commit server and a server
  that forwards requests, including an edge server.
- Check that archive-sharing configurations use the supported relationship:
  replicas may share archives with a master/commit server, not another replica.

Examples of supported arrangements include:

```text
master -> forwarding replica -> forwarding replica
commit -> edge -> edge -> proxy
commit -> edge -> standby
commit -> edge -> forwarding replica (read-only offload)
```

Examples to investigate or avoid include:

```text
master -> forwarding replica -> proxy
master -> p4broker -> forwarding replica
commit -> edge -> proxy -> proxy
commit -> p4broker -> edge
```

## Failover Readiness

- Confirm a trained P4 administrator owns the failover decision and procedure.
  Automatic failover can mistake a temporary network failure for a server
  failure and create data-integrity risks.
- Ensure a current, manually initiated failover script or runbook covers DNS
  redirection, role changes, metadata recovery, service startup, and stopping
  the inactive server.
- Exercise the runbook regularly and record elapsed recovery time, data loss,
  exceptions, and follow-up actions.