Preface

1. Introduction

Directory Structure Image (DSI) is a comprehensive comparison utility, used primarily to detect changes to a directory tree since initial installation. An image can be generated as part of the release process for a software product, and used subsequently to detect changes.

2. Overview

The image identifies elements which can be files, directories, or symlinks. Elements may be specified as absolute paths or paths relative to a variable installation root. In addition to files in an install root, additional elements may be specified that may be scattered sparsely across a filesystem.

Directories can be strictly or loosely controlled. A strictly controlled directory tree is one in which the image comprehends all elements in the tree, and excess or missing files are unexpected. A loosely controlled direcotry is one in which the image specifies specific files, but allows for excess files, directories, and symlinks to exist.

The DSI can be configured to understand a product that may include sparsely deployed files throughout a filesystem.

Detection of changes includes:

  • Newly added files/directories/symlinks.

  • Deleted files/directories/symlinks.

  • Differing last-modified timestamp of existing elements.

  • List files with differing content.

  • Changes in file/directory owner.

  • Changes in file/directory group.

  • Changes in file/directory permissions (e.g. +x bit, etc.)

Command line flags can suppress certain types of diffs that may not be important in certain scenarios, such as group/owner changes, as well as special verifications such as "owner can be anything except root."

This software is powered by the Perforce Helix version management software using Helix native DVCS features to show file and content differences.

3. Command Line Usage

Run the dsi tool with the -man flag to display a synopsis of options, plus documentation of commands and options, examples, and other info. Using the -h flag will display only the synopsis of options.

Command Line Usage
USAGE for dsi v1.0.9:

Usage varies based on mode, which can be one of: create, diff, info, list, rm, rmall, tags

dsi create [<tag>]

dsi diff [<tag>[@<rev>]]

dsi info

dsi list [<tag>]

dsi rm [<tag>@<rev>]

dsi rmall [<tag>]

dsi tags

Key environment variables:

* DSI_TAG defines a default for <tag>
* DSI_STORAGE, or specify as '-s <storage_dir>'
* DSI_PKG_ROOT, defaults to $PWD

or

dsi [-h|-man|-V]


DESCRIPTION:
	Directory Structure Image (DSI) is a comprehensive comparison utility,
	used primarily to detect changes to a directory tree since the initial
	installation.  A DSI can be generated as part of the release process
	for a software product, and then used subsequently to detect changes.

	Detection of changes includes:
	* Last-modified Timestamp
	* Newly added files/directories
	* Deleted files/directories
	* Changes to contents of files, including showing diffs
	* Changes in file/directory owner
	* Changes in file/directory group
	* Changes in file/directory permissions (e.g. +x bit, etc.)

	Command line flags can suppress certain types of diffs that may
	not be important in certain scenarios, such as group/owner changes,
	as well as special verifications such as 'owner can be anything
	except root.'

	This software is powered by the Perforce Helix version management
	software using Helix native DVCS features to show file and content
	differences.

TAGS AND REVISIONS:
	A DSI is assigned a tag on creation, which is generally the name of the
	product and optionally product version ID, e.g. 'fgs' or 'fgs-1.0'.

	DSI tag names can include alphanumeric characters, dashes, underscores,
	and dots.

	DSI revisions are positive integers associated with a tag and assigned
	upon creation.

MODES:
 Create, specified with 'create [<tag>]' (or 'c' for short)
	Create and store a DSI image with a given tag name.

 Diff, specified with 'diff [<tag>]' (or 'd' for short)
	Diff a DSI image with the given tag, using the latest version or a specified
   revision tag.  Optionally, specify the @<rev> to compare against a revision
   other than the head revision.

   If no flags are specified to limit the output, a comprehensive diff will
   report:
   * Timestamp differences
   * Checksum differences
   * Size Differences

   The Diff mode supports flags to exlcude certain things from comparison:

 Fix, specified with 'fix [<tag>]' (or 'f' for short)
	Fix a directory structure as it appears on disk to match the DSI. This may

 Info, specified with 'info' (or 'i' for short)
	Show DSI environment variables, default settings, and version info.

 List, specified as 'list [<tag>]' (or 'l' for short)
	List stored DSI images for the given tag.

 Remove, specified as 'rm [<tag>]@<rev>'
	Remove the specified revision for the given tag.  For this usage
	the revision must be specified.

 RemoveAll, specified as 'rmall <tag>'
	Remove all images for the given tag.

 Tags, specify as 'tags' (or 't' for short)
	List available image tags.

<tag>[@<rev>]
	Specify the DSI tag and revision.  A default tag value can be defined by
	setting the DSI_TAG environment variable.

	For Diff mode, the following can be specified:
	<tag>          Explicit value for <tag>, overriding DSI_TAG.
                       Revision defaults to @LATEST

	<tag>@<rev>    Explicit definition of tag and revision

	@<rev>         Uses DSI_TAG for the tag, with explicit rev.

        For Create or List modes, the following can be specified:
	<tag>          Explicit value for <tag>, overriding DSI_TAG.
	               Revision is not set with Create and List modes.

	For Remove mode the tag may be implied (with DSI_TAG) or specified
	explicitly.  The revision is required.

	For RemoveAll mode the tag must be specified explicitly.  A revision
	is not allowed.

OPTIONS:
 -d <package_dir>
	Specify the directory from which to create a DSI, or in which to
	do a comparsion against stored DSIs. The default is the current
	working directory as indicated by $PWD.

 -s <storage_dir>
	Specify the DSI storage dir. A default can be defined by setting
	the DSI_STORAGE environment variable

GENERAL OPTIONS:

 -v<n>	Set verbosity 1-5 (-v1 = quiet, -v5 = highest).

 -L <log>
	Specify the path to a log file to enable logging.  Or, specify
	'-L on' to enable logging using a default log file name:
	/tmp/dsi.v1.0.9.<datestamp>.log

	By default, logging is disabled.

	NOTE: This script is self-logging if logging is enabled. That is,
	output displayed on the screen is simultaneously captured in the log
	file.  Do not run this script with redirection operators like '> log'
	or '2>&1', and do not use 'tee.'

-si	Operate silently.  All output (stdout and stderr) is redirected to the
	log only; no output appears on the terminal.  This cannot be used with
	'-L off'.

 -D     Set extreme debugging verbosity.

HELP OPTIONS:
 -h	Display short help message
 -man	Display man-style help message
 -V	Dispay version info for this script and its libraries.

DSI FILE FORMAT:
	The directory structure image file format rocks.

FILES:
	DSI package definition files are stored in the DSI_STORAGE directory,
	and have a .dsi file name suffix.

EXAMPLES:

4. Image File Format and Samples

4.1. Image File Format

Following is the image file format:

Image File Format
# Directory Structure Image (DSI) File Format.

# Lines starting with hash ('#') characteres are comments.
# Hash characters inline are comments
# Support for characters in filenames is limited to ASCII (not extended ASCII)

The general format of data lines (i.e. non-commment lines) is:

<DirectiveOrElement> <Data>

where <DirectiveOrElement> is one of a fixed set of values, and <Data> depends on <DirectiveOrElement>.

<DirectiveOrElement> can be one of i, d, D, f, s.

The format of <Data> for element and directives types is:

i <PkgTag> <PkgMajorVersion> <PkgMajorVersionShort> <PkgMinorVersion> <PkgDescription>
  The 'i' Info entry contains package metadata provided by the user upon DSI creation.
  There can be only one 'i' entry per DSI file, and it must be the first entry.
  <PkgTag> alphanumeric+dash+underbar, e.g. fgs for Friendly Greeting System product..
  <PkgMajorVersion> alphanumeric+dash+underbar, the major version identifier, e.g. 2020.1.
  <PkgMajorVersion> alphanumeric+dash+underbar, a short form of the major version identifier, e.g. r20.1, or 'none'.
  <PkgMinorVersion> positive integer, typically a Perforce Helix changelist number of a product build.
  <PkgDesscription> ASCII string (can have spaces, but not extended ASCII)

d <RelPath> <Owner>:<Group> <Perms> <Timestamp>
  The 'd' is a directory element.
  <RelPath> is the relative path to the directory from the package root
  <Owner>:<Group> are the owner/group settings as used by the `chown` command.
  <Perms> are the file permissions as used by the `chmod` command.
  <Timestamp> is the file timestamp as reported by the `ls -l` command.

D <RelPath> <Owner>:<Group> <RecusrivePerms>
f <RelPath> <Owner>:<Group> <Perms> <Timestamp> <MD5Checksum>
s <RelPath> <Target> <TargetMD5Checksum>

<PkgTag> is the tag name for a product, e.g. 'fgs' for the Friendly Greeting System product.

The <Owner>:<Group> defines the Operating System owner and group, wit

<Perms> contains permission data, of the form <OwnerPerms>:<GroupPerms>:<OtherPerms>

4.2. Sample Image Files

Sample 1: Friendly Greeting System (FGS)

Sample image file:

# Sample Friendly Greeting System file format, with tag 'fgs':
i fgs 2020.1 r20.1 26937 Friendly Greeting System
d src unset:unset 750 20201123080422
f src/hello.h unset:unset 640 20201123080422
f src/hello.c unset:unset 640
f src/Makefile unset:unset 640
f src/build.sh unset:unset 750

Sample 2: FGS with sparse files.