# Welcome to the Perforce P4 Log Feeder, P4LF!

## Introduction

P4LF is the Perforce Log Feeder — a lightweight Go service that continuously
tails a Perforce server log (P4LOG) and writes compressed chunk files to a
configurable directory. External automation (outside the scope of this project)
is expected to pick up files matching `log.*.gz`, ship them elsewhere (e.g. for
ingestion into Splunk), and then delete them. P4LF's job is to populate the
directory; external software does the rest.

This near-real-time feeding of log data enables analytics and diagnostics in
tools like Splunk while the Perforce server continues to run normally.

This works with servers deployed with the P4 Server Deployment Package (SDP).
Executables, binaries, and config files are deployed in the SDP structure in
the `/p4/common/site/log_feeder` folder.

## How It Works

P4LF tails the P4LOG directly by reading bytes from the file, using an
inode-based rotation detection scheme:

* On startup (or resume after restart), P4LF reads its state file to recover
  the last known inode and byte offset of the P4LOG.
* At each `LogTailDelay` interval, P4LF reads all new bytes from the P4LOG
  since the last read and compresses them into a chunk file.
* Chunk files are written atomically (temp file + rename) to `LogChunksDir`
  with the naming format: `log.<YYYY-MM-DD>.<startOffset>.<endOffset>.gz`
* The date in the filename prevents naming collisions after a log rotation
  and makes files easy to identify chronologically.
* Log rotation is detected by comparing the current file inode against the
  saved inode. On rotation, P4LF resets to offset 0 (subject to
  `MaxRotationRecoverySize`) and logs the event.
* State (inode + offset) is saved atomically to `StateFile` after each flush,
  so P4LF resumes correctly after a restart with no gaps or duplicates.

## Chunk File Management

When `MaxLogChunks > 0` (default: 5000), P4LF enforces the limit by deleting
the oldest chunk files before writing a new one. This ensures p4lf keeps
running and capturing current P4LOG data even when the downstream consumer
(e.g. Splunk) is not processing files. A warning is logged whenever files are
deleted, noting that log data will be lost for those chunks.

`MinLogSpace` provides an independent disk-space guard: when free space in
`LogChunksDir` falls below the configured threshold, chunk writes are paused
until space is available.

## Config File Settings

The config file uses `KEY = VALUE` format. Lines beginning with `#` are
comments. The service reloads the config on SIGHUP or when it detects a
modification time change on the config file.

* **P4LogFile** — Path to the Perforce server log to tail.
  Default: `$P4LOG` (set by SDP `p4_vars`).

* **LogTailDelay** — How often to flush a log chunk.
  Format: `<integer>[s|m|h]` (e.g. `60s`, `5m`, `1h`). Default: `60s`.

* **LogChunksDir** — Directory where compressed chunk files are written.
  Default: `$LOGS/logchunks`.

* **MaxLogChunks** — Maximum number of `log.*.gz` files in `LogChunksDir`.
  When exceeded, the oldest files are deleted (oldest-first) to make room,
  and a warning is logged that log data has been lost. Set to `0` for no
  limit. Default: `5000`.

* **MinLogSpace** — Minimum free space in the `LogChunksDir` volume before
  chunk writes are paused. Accepts a percentage (e.g. `3%`) or a size
  (e.g. `500M`, `3G`). `None` or `0` = no check. Default: `None`.

* **MaxLogSize** — Maximum size of p4lf's own log (`$LOGS/p4lf.log`) before
  it is rotated and gzipped. `0` = no rotation. Default: `100M`.

* **MaxRotationRecoverySize** — When a P4LOG rotation is detected, p4lf
  reads from the beginning of the new log if its size is ≤ this value.
  If the new log is already larger (e.g. the service was down for a long
  time), p4lf starts from EOF and logs a warning to avoid ingesting a huge
  backlog. `0` = always read from the beginning. Default: `500M`.

* **StateFile** — Path for the persistent state file (inode + byte offset).
  Default: `$LOGS/p4lf.state`.

* **ReadFromStart** — On first run (no state file), whether to read the
  P4LOG from the beginning (`true`) or from the current end (`false`).
  Default: `true`.

* **Debug** — Verbosity: `0` = off, `1` = debug, `2` = pedantic. Default: `0`.

## Basic Flow

1. On startup, the config file is loaded and the state file is read to resume
   from the last known position.
2. A `LogTailDelay` ticker fires periodically. On each tick, p4lf:
   a. Checks for P4LOG rotation (inode change or truncation).
   b. Reads all new bytes from the P4LOG.
   c. If `MaxLogChunks > 0` and the limit is reached, deletes the oldest files.
   d. Compresses the bytes and atomically writes `log.<date>.<start>.<end>.gz`.
   e. Saves updated state (inode + offset) to `StateFile`.
3. On SIGHUP (or config file modtime change), the config is reloaded and
   p4lf's own log is rotated.
4. On SIGTERM/SIGINT, p4lf performs a final flush, saves state, and exits.

## Design Concepts

* Written in Go for reliability and easy cross-platform binary distribution.
* Runs as a systemd service named `p4lf`, operating as `User=perforce`.
* Designed for high-frequency operation — suitable for crontab at 1-minute
  intervals, generating ~1440 files/day.
* Avoids undue impact on p4d performance: if the service is offline, it
  is better to lose chunks of historical log data than to ingest a huge
  backlog at once. `MaxRotationRecoverySize` controls this trade-off.
* Log rotation is detected by inode change (SDP rotate_log_file style) or
  file truncation (copytruncate style). No `p4 logtail` command is used.
* All chunk files are written atomically: data is written to a `.tmp` file
  first, then renamed into place.
* Honors the SDP environment: `$P4LOG` and `$LOGS` environment variables
  are used as defaults; the systemd unit sources `/p4/common/bin/p4_vars`.