About

  • 1
    Member
  • 0
    Followers
  • 2
    Branches
Members
tom_tyler (C. Thomas Tyler)
Branches

Welcome to the Perforce P4 Log Feeder, P4LF!

Introduction

P4LF is the Perforce Log Feeder — a lightweight Go service that continuously tails a Perforce server log (P4LOG) and writes compressed chunk files to a configurable directory. External automation (outside the scope of this project) is expected to pick up files matching log.*.gz, ship them elsewhere (e.g. for ingestion into Splunk), and then delete them. P4LF's job is to populate the directory; external software does the rest.

This near-real-time feeding of log data enables analytics and diagnostics in tools like Splunk while the Perforce server continues to run normally.

This works with servers deployed with the P4 Server Deployment Package (SDP). Executables, binaries, and config files are deployed in the SDP structure in the /p4/common/site/log_feeder folder.

Installation

The quickest way to install p4lf on a Linux SDP server is the included install.sh script, run as root (it downloads the platform binary, example config, and systemd unit from workshop.perforce.com):

curl -fsSL https://workshop.perforce.com/download/p4lf/main/install.sh -o install.sh
sudo bash install.sh

What it does:

  1. Detects the current platform (Linux amd64/arm64; macOS builds can also be installed, but are intended for development/testing only, not production).
  2. Downloads the p4lf binary, p4lf.cfg.example, and p4lf.service for the selected stream (default: main) from workshop.perforce.com.
  3. Installs the binary to <install_dir>/p4lf (default install dir: /p4/common/site/log_feeder).
  4. Installs p4lf.cfg.example, and — on first install only — copies it to p4lf.cfg as a starting point. An existing p4lf.cfg is never overwritten.
  5. Installs or updates /etc/systemd/system/p4lf.service (only if it differs from what's already on disk) and runs systemctl daemon-reload when it does.
  6. Prints next-step instructions.

Useful options:

  • -s <stream> — P4 stream to install from (default: main; e.g. dev, r1.0).
  • -d <dir> — Installation directory (default: /p4/common/site/log_feeder).
  • -n — Dry run: show what would be done without making any changes.
  • -h — Show help.

After installing, edit the generated p4lf.cfg (set P4LogFile at minimum if $P4LOG isn't available in the service environment — see Config File Settings below), then:

sudo systemctl enable p4lf
sudo systemctl start p4lf
sudo systemctl status p4lf
journalctl -u p4lf -f

See the comments at the top of install.sh for the full option reference.

How It Works

P4LF tails the P4LOG directly by reading bytes from the file, using an inode-based rotation detection scheme:

  • On startup (or resume after restart), P4LF reads its state file to recover the last known inode and byte offset of the P4LOG.
  • At each LogTailDelay interval, P4LF reads all new bytes from the P4LOG since the last read and compresses them into a chunk file.
  • Chunk files are written atomically (temp file + rename) to LogChunksDir with the naming format: log.<YYYY-MM-DD>.<startOffset>.<endOffset>.gz
  • The date in the filename prevents naming collisions after a log rotation and makes files easy to identify chronologically.
  • Log rotation is detected by comparing the current file inode against the saved inode. On rotation, P4LF resets to offset 0 (subject to MaxRotationRecoverySize) and logs the event.
  • State (inode + offset) is saved atomically to StateFile after each flush, so P4LF resumes correctly after a restart with no gaps or duplicates.

Chunk File Management

When MaxLogChunks > 0 (default: 5000), P4LF enforces the limit by deleting the oldest chunk files before writing a new one. This ensures p4lf keeps running and capturing current P4LOG data even when the downstream consumer (e.g. Splunk) is not processing files. A warning is logged whenever files are deleted, noting that log data will be lost for those chunks.

MinLogSpace provides an independent disk-space guard: when free space in LogChunksDir falls below the configured threshold, chunk writes are paused until space is available.

Config File Settings

The config file uses KEY = VALUE format. Lines beginning with # are comments. The service reloads the config on SIGHUP or when it detects a modification time change on the config file.

  • P4LogFile — Path to the Perforce server log to tail. Default: $P4LOG (set by SDP p4_vars).

  • LogTailDelay — How often to flush a log chunk. Format: <integer>[s|m|h] (e.g. 60s, 5m, 1h). Default: 60s.

  • LogChunksDir — Directory where compressed chunk files are written. Default: $LOGS/logchunks.

  • MaxLogChunks — Maximum number of log.*.gz files in LogChunksDir. When exceeded, the oldest files are deleted (oldest-first) to make room, and a warning is logged that log data has been lost. Set to 0 for no limit. Default: 5000.

  • MinLogSpace — Minimum free space in the LogChunksDir volume before chunk writes are paused. Accepts a percentage (e.g. 3%) or a size (e.g. 500M, 3G). None or 0 = no check. Default: None.

  • MaxLogSize — Maximum size of p4lf's own log ($LOGS/p4lf.log) before it is rotated and gzipped. 0 = no rotation. Default: 100M.

  • MaxRotationRecoverySize — When a P4LOG rotation is detected, p4lf reads from the beginning of the new log if its size is ≤ this value. If the new log is already larger (e.g. the service was down for a long time), p4lf starts from EOF and logs a warning to avoid ingesting a huge backlog. 0 = always read from the beginning. Default: 500M.

  • StateFile — Path for the persistent state file (inode + byte offset). Default: $LOGS/p4lf.state.

  • ReadFromStart — On first run (no state file), whether to read the P4LOG from the beginning (true) or from the current end (false). Default: true.

  • Debug — Verbosity: 0 = off, 1 = debug, 2 = pedantic. Default: 0.

Basic Flow

  1. On startup, the config file is loaded and the state file is read to resume from the last known position.
  2. A LogTailDelay ticker fires periodically. On each tick, p4lf: a. Checks for P4LOG rotation (inode change or truncation). b. Reads all new bytes from the P4LOG. c. If MaxLogChunks > 0 and the limit is reached, deletes the oldest files. d. Compresses the bytes and atomically writes log.<date>.<start>.<end>.gz. e. Saves updated state (inode + offset) to StateFile.
  3. On SIGHUP (or config file modtime change), the config is reloaded and p4lf's own log is rotated.
  4. On SIGTERM/SIGINT, p4lf performs a final flush, saves state, and exits.

Design Concepts

  • Written in Go for reliability and easy cross-platform binary distribution.
  • Runs as a systemd service named p4lf, operating as User=perforce.
  • Designed for high-frequency operation — suitable for crontab at 1-minute intervals, generating ~1440 files/day.
  • Avoids undue impact on p4d performance: if the service is offline, it is better to lose chunks of historical log data than to ingest a huge backlog at once. MaxRotationRecoverySize controls this trade-off.
  • Log rotation is detected by inode change (SDP rotate_log_file style) or file truncation (copytruncate style). No p4 logtail command is used.
  • All chunk files are written atomically: data is written to a .tmp file first, then renamed into place.
  • Honors the SDP environment: $P4LOG and $LOGS environment variables are used as defaults; the systemd unit sources /p4/common/bin/p4_vars.

Developer Guide

Working on p4lf itself (building, testing, versioning, or the Perforce dev → main → release promotion workflow)? See docs/DeveloperGuide.md.