= [DRAFT] Perforce P4D Sample Storage Setup - LVM
Perforce Professional Services <consulting-helix-core@perforce.com>
// Metadata
:revnumber: v2025.1
:revdate: 2025-07-02
:description: Instructions for setting up initial storage using LVM
// Settings
:doctype: book
:icons: font
:toc:
:toclevels: 5
:sectnumlevels: 4
:xrefstyle: full
:source-highlighter: highlight.js
// Attribute for ifdef usage
:unix_doc: true
[discrete]
=== Todo Items
* Test script operation
// Horizontal Rule
// '''
== Preface
This document illustrates a basic set of commands to setup data storage using Linux Volume Manager (LVM).
[NOTE]
====
LVM is the preferred method of setup for Perforce volumes.
Both Physical servers and Cloud/VM servers benefit from features of LVM covered in this document.
====
The goal in these examples is to configure three LVM storage volumes separate from the OS volume on a server destined to become a Helix Core server.
At the start of this procedure, empty volumes with no data are formatted.
[discrete]
=== Console sessions
When console sessions are shown in the text:
[source, console]
.Normal user shown with `$` prompt
----
$ ls -lR
----
[source, console]
.Root user shown with `#` prompt
----
# systemctl daemon-reload
----
*Please Give Us Feedback*
Perforce welcomes feedback from our users.
Please send any suggestions for improving this document or the SDP to consulting-helix-core@perforce.com.
:sectnums:
== Why use LVM ?
Some of the benefits of using LVM over standard disk partitions are:
* Expansion of physical volumes is not limited to the size remaining on the disk. Additional disks can be added to the Volume Group as needed for expansion.
* Volume name/UUID are embedded into the media. This helps identify snapshots and cloned volumes used for migrations.
* LVM avoids need for UUID's in `/etc/fstab`, as volumes are always assigned unique device names.
WARNING: LVM also supports a volume snapshot and revert feature. Howver, LVM snapshots incur a significant performance penalty due to the Copy-On-Write (COW) method utilized. For this reason LVM snapshots are not recommended for use on Perforce volumes.
== Sample Storage Setup for SDP - Mounts and Storage
=== Starting State
This procedure assumes the following start state:
* Destination Server has a basic install of Ubuntu 24.04. ( RHEL/Rocky9 TBD ) with following installed:
** LVM2, if missing run `apt install lvm2`
** XFS, if missing run `apt install xfsprogs`
* Three separate storage volumes are attached Server / VM. (in addition to the OS root volume), intended to become `/p4depots`, `/p4logs`, and `/p4metadta`:
- `/p4depots` - Give this as much space as you think you'll need. This is highly variable. You may want 30G source code projects, or 1T or more for virtual production or game development. Size can be increased easily and non-disruptively later, so you don't need to overprovision (and overpay) for storage.
- `/p4metadata` - Use 25G to start. This needs to hold (2) copies of the database.
- `/p4logs` - Use 20G to start. Typically low usage, but large enough to contain large journals during any occasional purge/obliterate operations.
* Volumes may be Physical disks, Direct-Attached-Storage (DAS), EBS volumes on AWS, VMFS disks on a VM or other block storage.
[NOTE]
--
There is no easy method of matching the device being attached to the assigned kernel name such as: `nvme1n1` or `nvme1n2`. However if we choose disks of different sizes they can easliy be matched up with kernal assigned names.
For instance, in the example above, the 1000G volume `nvme2n1` is for `/p4depots`, the 25G volume `nvme1n1` is for `/p4metadata` and the 20G volume is for `/p4logs`
--
* See Also: https://www.reddit.com/r/linuxadmin/comments/8cg1t4/benefits_of_lvm/
=== Storage Formatting and Mounting Procedure
First, become root with the `sudo su -` command, and then list the mounted storage before we mount the new volumes:
[source, console]
----
$ sudo su -
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/nvme0n1p1 10G 1.6G 8.5G 16% /
tmpfs 356M 0 356M 0% /run/user/0
----
You don't yet see the newly attached volumes, as they're not yet mounted. But you can list them with `lsblk`:
[source, bash]
----
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 10G 0 disk
└─nvme0n1p1 259:1 0 10G 0 part /
nvme1n1 259:2 0 25G 0 disk
nvme2n1 259:3 0 20G 0 disk
nvme3n1 259:4 0 100G 0 disk
----
==== LVM
The Linux Volume Manager (LVM) is used in place of old style disk partitions, and is much more flexible. Volumes may be expanded while the system is running, and volumes may span more than one disk.
Ubuntu 24.04 installer includes LVM.
* see: https://documentation.ubuntu.com/server/explanation/storage/about-lvm/#
This document provides a summary of only the basic commands to setup LVM following best practices.
.LVM components
****
PV's:: Physical Volumes. `PV` volumes provide storage to their assigned `VG` Marking a disk as a `PV` makes it available to be assigned to a `VG`
VG's:: Volume Groups. `VG` are comprised of one or more `PVs` that supply storage to the `VG` The VG storage is comprised of all disks attached to it and is not limited to the size of any single disk.
LV's:: Logical Volumes `LVs` are composed by allocating space from a `VG` Disks with different performance characteristics should be separated into different VG's so they may be assigned to appropriate `LVs`
****
NOTE: LVM marks PV's, VG's and LV's with unique `UUIDs` and includes the name of the `VG` or `LV` on the media.
==== Physical Volumes(PVs)
Normally the entire disk is marked as a `PV`, however in the past folks have also marked disk *partitions* as `PVs`.
NOTE: The reason full disks are used without parititions , is that any later disk expansion can then avoid expanding that partition.
First mark each of the 3 new blank volumes as an `PV`.
[source, console]
----
[root@~]# pvcreate /dev/nvme1n1
Writing physical volume data to disk "/dev/nvme1n1"
Physical volume "/dev/nvme1n1" successfully created.
[root@~]# pvcreate /dev/nvme2n1
Writing physical volume data to disk "/dev/nvme2n1"
Physical volume "/dev/nvme2n1" successfully created.
[root@~]# pvcreate /dev/nvme3n1
Writing physical volume data to disk "/dev/nvme3n1"
Physical volume "/dev/nvme3n1" successfully created.
----
[discrete]
===== Display PV's
You can display the new PV's in short or long mode.
[source, console]
.Short
----
# pvs
PV VG Fmt Attr PSize PFree
/dev/nvme1n1 lvm2 --- <25G <25G
/dev/nvme2n1 lvm2 --- <20G <20G
/dev/nvme3n1 lvm2 --- <1000G <1000G
----
[source, console]
.Long
----
# pvdisplay
... < many lines showing Name, size, UUID >
----
==== Volume Groups (VGs)
Next create `VGs` for each of the three types of storage.
We will name the 3 `VGs` as:
* vg_p4metadata
* vg_p4logs
* vg_p4depots
Create the new 'VGs' and assign the appropriate `PV` with:
[source, console]
----
# vgcreate vg_p4metadata /dev/nvme1n1 #<.>
Volume group "vg-p4metadata" successfully created
# vgcreate vg_p4logs /dev/nvme2n1 #<.>
Volume group "vg-p4logs" successfully created
# vgcreate vg_p4depots /dev/nvme3n1 #<.>
Volume group "vg-p4depots" successfully created
----
<.> p4metdata was determined to use `nvme1n1` by matching disk size of 25GB
<.> p4logs was determined to use `nvme2n1` by matching disk size of 20GB
<.> p4depots was determined to use `nvme3n1` by matching disk size of 1000GB
[discrete]
===== Display VG's
You can display the new VG's in short or long mode.
[source, console]
.Short
----
# vgs
VG #PV #LV #SN Attr VSize VFree
vg_p4metadata x 0 0 wz--n- <250G
vg_p4logs x 0 0 wz--n- <200G
vg_p4depots x 0 0 wz--n- <1000G
----
[source, console]
.Long
----
# vgdisplay
... < many lines showing Name, Size, PV UUID >
----
==== Logical Volumes (LVs)
Finally create logical volumes `LVs` inside each `VG`, using all of the available storage.
[source, console]
----
# lvcreate -n lv_p4metadata -L 100%FREE vg_p4metadata
Logical volume "lv_p4metadata" created.
# lvcreate -n lv_p4logs -L 100%FREE vg_p4logs
Logical volume "lv_p4logs" created.
# lvcreate -n lv_p4depots -L 100%FREE vg_p4depots
Logical volume "lv_p4depots" created.
----
[discrete]
===== Display LV's
You can display the new LV's in short or long mode.
[source, console]
.Short
----
# lvs
LV VG Attr LSize Pool Origin Data% Meta%
lv_p4metadata vg_p4metadata -wi-a- 25G
lv_p4logs vg_p4logs -wi-a- 0 20G
lv_p4depots vg_p4depots -wi-a- 0 1000G
----
[source, console]
.Long
----
# lvdisplay
... < many lines showing Name, Size, Status, LV UUID >
----
==== Formating LVM volumes
`LVM` *physical block devices* are mapped to Linux *virtual block devices* by the Device Mapper.
The device mapper will automatically create devices for the above LVM disks as follows:
* /dev/mapper/vg_p4metadata-lv_p4metadata
* /dev/mapper/vg_p4logs-lv_p4logs
* /dev/mapper/vg_p4depots-lv_p4depots
NOTE: The explicit `LVM` device names assigned makes formatting these volumes much safer.
Next, Format these new `LVs` as XFS.
[source, console]
----
# mkfs.xfs /dev/mapper/vg_p4metadata-lv_p4metadata
meta-data=/dev/mapper/vg_p4metadata-lv_p4metadata isize=512 agcount=16, agsize=1638400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=26214400, imaxpct=25
= sunit=1 swidth=1 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=12800, version=2
= sectsz=512 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# mkfs.xfs /dev/mapper/vg_p4logs-lv_p4logs
meta-data=/dev/mapper/vg_p4logs-lv_p4logs isize=512 agcount=16, agsize=1638400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=26214400, imaxpct=25
= sunit=1 swidth=1 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=12800, version=2
= sectsz=512 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
# mkfs.xfs /dev/mapper/vg_p4depots-lv_p4depots
meta-data=/dev/mapper/vg_p4depots-lv_p4depots isize=512 agcount=16, agsize=1638400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=26214400, imaxpct=25
= sunit=1 swidth=1 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=12800, version=2
= sectsz=512 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
----
WARNING: Formatting the wrong device may destroy data !
==== Update /etc/fstab mount table
Make a backup copy of the `/etc/fstab` file, and then modify that file to create new volumes.
[source, console]
----
# cd /etc
# ls -l fstab*
-rw-r--r--. 1 root root 394 Nov 15 04:43 fstab
# cp -p fstab fstab.bak.2022-03-08
----
[discrete]
==== Edit fstab
Next add entries to `/etc/fstab` for the mount points.
[source, console]
----
# echo "/dev/mapper/vg_p4metadata-lv_p4metadata /p4metadata xfs defaults 0 0" >> /etc/fstab
# echo "/dev/mapper/vg_p4logs-lv_p4logs /p4logs xfs defaults 0 0" >> /etc/fstab
# echo "/dev/mapper/vg_p4depots-lv_p4depots /p4depots xfs defaults 0 0" >> /etc/fstab
----
NOTE: The previous method of mounting Volumes in `/etc/fstab`` by their `UUID` values is not used with LVM. LVM uses UUID's internally, and the Linux kernel Device mapper always maps UUID's to the same device.
[discrete]
==== Update mounts
After changing `/etc/fstab` run this command up update mount info.
[source, console]
----
# systemctl daemon-reload
----
CAUTION: Run the `systemctl daemon-reload` command after each change to `/etc/fstab` or bad things might happen. It is also a good idea to mount new filesystems the first time with `mount -a` which should report any errors in `/etc/fstab` before rebooting the machine.
Proceed with creating empty directories that will be the "mount points" for the volume to be mounted.
[source, console]
----
# mkdir /p4depots /p4logs /p4metadata
----
Next, use the `mount -a` command. This will now assocate the mount points you just created with the storage device information that is now in that `/etc/fstab` file, and mount the volumes.
[source, console]
----
# mount -a
----
Then see if they are mounted. This is what victory looks like, with the `/p4*` volumes all mounted with desired sizes:
[source, console]
----
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 17M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/nvme0n1p1 10G 1.6G 8.5G 16% /
tmpfs 356M 0 356M 0% /run/user/0
/dev/mapper/vg_p4metadata-lv_p4metadata 120G 890M 120G 1% /p4metadata
/dev/mapper/vg_p4logs-lv_p4logs 100G 747M 100G 1% /p4logs
/dev/mapper/vg_p4depots-lv_p4depots 500G 3.6G 497G 1% /p4depots
----
At this point, you are ready to install the Server Deployment Package (SDP) software.
== Volume Expansion
An LVM partition can usually be expanded while the system is running without any outage.
The procedure involves the following steps:
. Expand the associated VG by one of following
.. Expand the underlying PV disk ( such as a virtual disk )
... Rescan the storage buss
... Resize the PV
... Extend the LV -or-
.. Add additonal disks to the VG ( typically physical disks )
... Rescan the storage buss
... Extend the VG
... Extend the LV
=== Expand VG by expanding underlying disk
In a virtual environment it is typically easy to expand the size of the underlying disk.
CAUTION: Even though this procedure can be run without downtime, to be safe, ensure you have a recent backup of the system.
. Expand the virtual block storage device, by changing its size on AWS, VMware, or other environment.
. Rescan the VM's *scsi-buss* to pick up the change
--
Note replace *nvme1n1* with your block device
--
[source, console]
----
$ sudo su
# echo 1 > /sys/block/*nvme1n1*/device/rescan_controller
# exit
// Alternate procedure
$ sudo apt install scsi-tools
$ sudo rescan-scsi-bus -s
----
[start=3]
. Resize the PV to use the new expaned size
[source, console]
----
### Note replace *nvme1n1* with your block device
$ sudo pvresize /dev/*nvme1n1*
----
[start=4]
. Resize the LV to use the new storage
--
Note replace `/dev/mapper/vg_p4depots-lv_p4depot` with your device
Run one of the following
--
[source, console]
----
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4depots-lv_p4depots
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4logs-lv_p4logs
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4metadata-lv_p4metadata
----
[start=5]
. Grow the Filesystem
--
Note replace `/dev/mapper/vg_p4depots-lv_p4depots` with your device
--
[source, console]
----
# xfs_growfs /dev/mapper/vg_p4depots-lv_p4depots # Full Size
----
=== Expand VG by adding more disks
CAUTION: Even though this procedure can be run without downtime, to be safe, ensure you have a recent backup of the system.
. Expand the VG by adding additional disks to the VG.
. Hot-pluggable server disks are required to add without downtime. After adding the disk, rescan the VM's *scsi-buss* to pick up the change
To determine the exact device name of the added disk, look at the kernel ring buffer for the expected device name immediatly after adding the disk.
[source, console]
--
$ sudo dmesg | grep nvme # If device is expected to be nvme*
--
--
Note replace *nvme5n1* with your block device shown with `dmesg`
sudo dmesg # Look for new block device signature, e.g `nvme5n1`
--
[source, console]
--
$ sudo su
# echo 1 > /sys/block/*nvme5n1*/device/rescan_controller
# exit
#Alternate procedure
$ sudo apt install scsi-tools
$ sudo rescan-scsi-bus
--
[start=3]
. Mark new Disk as a PV
--
Note replace *nvme5n1* with your new device
--
[source, console]
----
$ sudo pvcreate /dev/*nvm5n1*
----
[start=4]
. Extend the VG
--
Replace *nvme5n1* with your new disk
--
[source, console]
----
$ sudo vgextend vg_p4metadata-lv_p4metadata /dev/*nvme5n1*
Volume group "vg_p4metadata" sucessfully extended
----
[start=5]
. Extend the LV
--
Replace the LV with your desired LV
--
[source, console]
----
$ sudo lvextend -l +100%FREE /dev/mapper/vg_p4metadata-lv_p4metadata
Size of logical volume "lv_p4metadata" changed fromm 205GB to 300GB
Logical volume "lv_p4metadata" sucessfully resized.
----
=== LVM Storage setup script
For reference if device names are known in advance, the LVM setup procedure may be scripted as follows:
[source, bash]
----
#!/bin/bash
#NOTE DRAFT, error checking not fully included
# Devices to format ( root on nvme1n1 )
DEV_META="/dev/nvme2n1"
DEV_LOGS="/dev/nvme3n1"
DEV_DEPOTS="/dev/nvme4n1"
# Mount points
MP_META="/p4metadata"
MP_LOGS="/p4logs"
MP_DEPOTS="/p4depots"
# VGs
VG_META=vg_p4metadata
VG_LOGS=vg_p4logs
VG_DEPOTS=vg_p4depots
# LVs
LV_META=lv_p4metadata
LV_LOGS=lv_p4logs
LV_DEPOTS=lv_p4depots
# Backup /etc/fstab
cp /etc/fstab /etc/fstab.bak.$(date +%F)
# If device unused, create PV's
for dev in $DEV_META $DEV_LOGS $DEV_DEPOTS; do
if blkid "$dev" &> /dev/null; then
echo "Warning: $dev already has a file system. Skipping mkfs.xfs."
else
echo "Creating PV on $dev"
pvcreate "$dev"
fi
done
# Create VG's
vgcreate $VG_META $DEV_META
vgcreate $VG_LOGS $DEV_LOGS
vgcreate $VG_DEPOTS $DEV_DEPOTS
#Create LV's
lvcreate -n $LV_META -L 100%FREE $VG_META
lvcreate -n $LV_LOGS -L 100%FREE $VG_LOGS
lvcreate -n $LV_DEPOTS -L 100%FREE $VG_DEPOTS
# Create mount points if they don't exist
[ ! -d "$MP_META" ] && mkdir -p "$MP_META"
[ ! -d "$MP_LOGS" ] && mkdir -p "$MP_LOGS"
[ ! -d "$MP_DEPOTS" ] && mkdir -p "$MP_DEPOTS"
# Add entries to /etc/fstab
echo "/dev/mapper/${VG_META}-${LV_META} ${MP_META} xfs defaults 0 0" >> /etc/fstab
echo "/dev/mapper/${VG_LOGS}-${LV_LOGS} ${MP_LOGS} xfs defaults 0 0" >> /etc/fstab
echo "/dev/mapper/${VG_DEPOTS}-${LV_DEPOTS} ${MP_DEPOTS} xfs defaults 0 0" >> /etc/fstab
# Reload systemd daemon
echo "Reloading systemd to apply changes to /etc/fstab."
systemctl daemon-reload
# Mount all file systems
mount -a
# Verify mounts
mountpoint "$MP_DEPOTS" || echo "Error: $MP_DEPOTS is not mounted."
mountpoint "$MP_LOGS" || echo "Error: $MP_LOGS is not mounted."
mountpoint "$MP_META" || echo "Error: $MP_METADATA is not mounted."
# Display current mounts
df -h
----
== Notes and References
A few possible issues that may come up are:
. Mounting duplicate named Disks
* If you clone a disk and mount it to the same machine you will get an error because the UUID's and LVM assigned name are the same.
* https://unix.stackexchange.com/questions/495669/how-to-mount-lvm-partitions-with-duplicate-names
. Snapshot and revert before dangerous ops.
* https://digitalcave.ca/resources/computer/lvm-snapshots.jsp
* https://askubuntu.com/questions/424225/setting-up-lvm-snapshot-as-a-backup-restore-point-in-ubuntu
* https://www.percona.com/blog/disaster-lvm-performance-in-snapshot-mode/