Copyright
Copyright (C) 1997 Capella Computers Ltd.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
Version
This is version 0.1 of the wrappers, dated 7/97. It has just emerged from
preliminary testing and is being used by the developers in
Capella Computers Ltd. It is likely that numerous small problems will
be detected and corrected during the next few months. When no change will
be made to the system for a reasonable period of time (two months?), the
first stable release (1.0) will be "formally" declared.
If you have any questions, discovered any bugs,
would like the most up-to-date
version, or have any other sort of feedback, contact
the author,
oren@capella.co.il.
Introduction
This document gives a brief overview of the perl p4 wrappers.
The purpose of these wrappers is to implement a particular development
process on top of the generic p4 services. This process is an
adaptation of the process enforced by aegis. Aegis is a
free software available on the net, and is much more robust and complete
then this set of wrappers. If you develop on UNIX platforms and you can use
NFS to link the developer's workstations, you should probably use aegis.
A deliberate design choice made when implementing these wrappers was
not to attempt any form of co-existance with other practices. This allowed
the wrappers to rely on a particular structuring of the depot, thereby
allowing better protection schemes and higher-level baviour. As a result,
the wrappers completely replace the built-in p4 interface, with
a few notable exceptions which will be listed below.
The wrappers are implemented in perl and as such are rather portable.
However, they invoke a few external programs (such as cp, rm
and mkdir). While the access to these programs is isolated, it is
not exactly trivial to port the system to DOS, for example. However, you
can obtain all the necessary programs (including a shell) as free software
from GNU project or as commercial software (from MKS, for example).
Basic Concepts
The overview for aegis contains a good tutorial of the basic
concepts. It is also recommended that you become familiar with the
basic p4 concepts before you read this overview.
-
The definitive sources of the system reside in //depot/baseline/....
-
The baseline always "works". In practical terms
it means that there is a perl script, verify.pl, in the top level
directory of the baseline, and that running it is guaranteed to succeed.
Note that this file is part of the baseline and can be changed like any
other source file.
-
Development is done in terms of jobs. The best way to view a job
is as a transaction on the sources database. Like a transaction, it must
ensure that it brings the database to a consistent state (as defined by
verify.pl). It is atomic - either it is applied in full or it
is rejected completely. And like in any database, the transactions are
serialized.
-
The integration of a job into the baseline creates a new system version.
The result of serializing the jobs is to create a system history which is a
linear sequence of such versions, where it seems as if each job
begun with the version created by the previous job, made some changes,
and created the next version.
Job Life Cycle
The trick is, of course, how to enable a team of programmers to work
on several jobs at once and still maintain the above benefits.
The best way to understand this process is to follow the life cycle of a job.
- new
-
This is a status of a newly created job. At this point, the job exists
as a simple p4 job.
- work
-
Only a single developer can work on a job (a developr may, of course, work
on several jobs simultanously). For the developer to be able to work,
a new branch is created and populated with the current baseline version.
The new branch is named
//depot/user/job-attempt/.
The attempt
number is incremented whenever a developer undoes everything, returning
the job to the new state. If the same developer starts development
later on, p4 demands we use a different branch and directory name,
hence the count.
The developer's work is done inside a p4 change.
This change is submitted to
p4 when the job is submitted to review.
- review
-
When development is done, and the job is verified, it is moved to this state.
At this stage, all files modified by the job are locked, so no two active jobs
which are at this stage or onwards can touch the same file. This is necessary
since such jobs will be sent to integration without further synchronization
with the baseline.
Sadly, it is not possible to use the built-on p4 file locking mechanism
for this, since it only allows associating a lock with an active change which
actually tries to modify the locked files. Instead, there's a dummy p4
job which holds in its description the list of all locked files.
- fail
-
The system supports the notion of a code review. At the moment, anyone
(even the developer itself) can change the review status of a job. Aegis
does a much better job here (as in some other places) by limiting this to
developers which are specially denoted as reviewers, and also optionally
blocks a developer from reviewing himself.
At any rate, a review may fail the job. In this case, the developer is
expected to resume working on it. Note that this means that a new change will
be opened, and this change does not necessarily touch all the files which
were touched by the previous one, which means the built-in p4 reports
for change file lists are not sufficient by themselves to describe what has
been done in a particular job.
- pass
-
At this stage, the job comes to the attention of the integrator. In
aegis, this can be any developer who has been assigned the task.
In p4, you need to specify a special user (called integ)
to handle the task. This means an extra p4 license. However, Perforce
were kind enough to give us (Capella Computers) an extra license so we
could rotate the task between the developers.
At any rate, the integrator chooses between the jobs at this stage to
decide which job to integrate next. This may have consequences on developers
working on other changes, so the choice is not always easy.
- integ
-
Only one job can be at this stage at any given time. This is the job that
is being integrated into the baseline. The integration consists of opening
a new change, in which the development branch is merged into the baseline.
Note that this is a simple merge - there are never any conflicts (due to
the file locks mentioned before). In theory, it should be possible to merge
more then one branch in the same change, though this is not currently
implemented. Again, there is no danger of conflicts.
The resulting baseline version is re-verified (using verify.pl). This
is necessary since the verification done by the developer was on his machine,
and may have passed due to his special development environment or due to
other external factors. The integration verification is the last line of
defense - any problem that gets by it will go straight into the baseline.
Once it passes verification, the change is submitted, a new baseline version
is created, the job file locks are released, and there's no going back.
Of course, you can easily
recreate a previous version of the baseline. But development from this point
on would build on the version just created.
The baseline version created is given the label Lversion which
can be used to refer to this version in various commands (e.g., p4fp).
Note that the version number is different from the job number, and is always
increasing by one for each integration.
- clear
-
Since aegis assumes file-system access to all development hosts, it
cleans up all the development directories when a job is integrated. One of
the nice points about p4 is that it works through any Tcp/Ip connection.
This means that the developer needs to do something on his own host to clean
things up - both the development branch and the development directory. Note
that since there's no point in keeping history of the branch files, the
branch files are truly obliterated.
- done
-
Once a job is cleared, there's nothing left to do. It is still possible
to edit the job description, but any changes to the sources require opening
a new job.
Commands
Copyright
In order to follow the spirit of the GPL, the following command, useless
as they may seem to developers, are provided:
- p4notice
- This displays a short copyright notice, and refers to the copy of the
GPL which is supposed to be distributed with the software.
Environment
There are a several environment variables which are used by
the wrappers. With the exception of P4JOB, they are all mandatory.
- P4ROOT
-
The root directory of the client's view.
- P4PATH
-
Where the wrappers are installed.
- P4PORT
-
The Tcp/Ip address (including the port) of the p4 server.
- P4CLIENT
-
The identity of the current client.
- P4USER
-
The identity of the current user.
- P4JOB
-
The number of the default job (optional). Serves as the default job
number for any command which takes such an argument.
The environment can be manipulated by the following commands:
-
- p4el
-
List all relevant environment variables.
- p4ej [job]
-
Print (if given no argument) or set the default job; a value of '-'
means
unsetting the variable altogether.
- p4eu [user]
-
Print (if given no argument) or set the current user id. This command should
not be used by most users.
There are additional environment variables used, which are not p4
specific.
- TEMP
-
Where to create temporary directories. Mandatory.
- EDITOR
-
How to edit text files. Is only used by p4je, but you really want
to set it anyway (is used by a lot of utilities).
- DIFF
-
How to diff text files. Used by p4rdiff and by some
p4 commands. The safe bet is to set it - after all, any system
has a reasonable built-in diff, even if you need to get it of the net.
- MERGE
-
How to merge text files. You can do without it if you don't ask for a merged
version during conflict resolution, which is a good idea anyway. Using the
edit option is sufficient, really.
Report
The following are read-only operations which should be available for
all users on all clients, unless stated otherwise.
- p4dh
-
List all the changes in the depot, from recent to ancient. The output
is the raw output of p4. Todo: format output as a report.
- p4dv [release-specifier]
-
Print depot baseline directory. This requires you to have a copy of the
baseline on the current client. Such a copy can be obtained
by using p4fg. It takes as argument any valid p4 release
specifier, such as @Lversion-number.
- p4fh [file-pattern] ...
-
Display file history (as a list of jobs). Todo: list the system version
number as well as the job number. It takes as argument any file pattern
recognized by p4. Typically you just give it the path name(s) of
the files in the development or baseline directories, which p4
translates automatically to its internal file name specification.
- p4jd [job]
-
Get job details. It could probably be in a better format.
- p4je [job]
-
Edit job data. This one is very dangerous. As long as you only
change the description, you are safe; otherwise you can really shoot
yourself in the foot. Todo: provide a safer way to edit just the
description.
- p4jf [job]
-
Get list of job file operations - that is, the files changed in this job,
and the operation performed on each (add/delete/edit).
If the job is in work mode, this will
only work on the development machine. Otherwise, it will work anywhere
(very important for reviewers).
- p4jh [job]
-
History of a job (a list of changes). Same problems as the depot
history command.
- p4jl [filter] ...
-
List jobs. By default, shows all jobs, but accepts a multitude of filtering
commands. These commands are applied in the order they are given. A '+' turns
on a set of jobs, a '-' turns it off. If the first command
is a '+', then
it is assumed you start with the empty set; if it is a '-',
it is assumed
you start with the complete set. The '+/-Status'
command turns on/off the
listing of jobs of any status. '+/-New' controls listing of new jobs,
and likewise for any other job status.
It is also possible to filter by user: '+/-u user'
will turn on/off the
set of jobs for the specified user, where 'User' means
all users, and 'Me'
means the current user. You can mix filtering by status and by user; for
example, p4jl -done -u me lists all the jobs which aren't done
and are owned by the current user.
Any of the keyworks can be shortened to its first letter; the previous
example can be written as p4jl -d -u m.
Just for fun, a set of shell functions for common cases is provided:
- p4jln
- Is equivalent to p4jl +n
- p4jlw
- Is equivalent to p4jl +w
- p4jlr
- Is equivalent to p4jl +r
- p4jlf
- Is equivalent to p4jl +f
- p4jlp
- Is equivalent to p4jl +p
- p4jli
- Is equivalent to p4jl +i
- p4jlc
- Is equivalent to p4jl +c
- p4jld
- Is equivalent to p4jl +d
- p4jlu
- Is equivalent to p4jl +u; requires an argument.
- p4jlm
- Is equivalent to p4jl +u me
- p4uh [user]
-
History of a user (a list of changes). Same problems as the depot
history command.
- p4ul
-
all users. Again, the unformatted output of the p4 command.
- p4vl
-
List all versions of the depot, which job created them, who was the
developr and so on.
The following are file commands which are not commonly used. Unlike
any other command, they take as first argument either baseline
or a job number (which is P4JOB by default). They use this
argument to construct the necessary prefix for the file pattern
passed to p4. By default, this pattern is ....
When applied to files of a job, these commands work only for the developer
on the development client if the job is in work state, and for
anyone anywhere if the job is in review/pass/fail state.
In the second case, the (default) pattern ... is taken to mean
all the files opened in this job, and not all the files. In order to get
all the files, specify the .... (add another '.').
- p4fg [job|baseline] [file-pattern][release-specifier] ...
- Get depot file(s). For example, p4fg baseline @L7 would
bring version 7 of the whole depot to the current client.
If you got files you want to get rid of, the only way is to get them with
a release specifier of '#none'. This removes the files from the
client.
- p4fp [job|baseline] [file-pattern][release-specifier] ...
- Print depot file(s). For example, p4fp baseline verify.pl@L7 would
print the contents of verify.pl as it was in version 7 of the system.
- p4fr [job|baseline] [file-pattern][release-specifier] ...
- Refresh unopened file(s).
For example, p4fr verify.pl would
refresh verify.pl from the depot, in case it was deleted
by mistake using rm, for example.
New Jobs
- p4jn title [issue] ...
-
Create a new job. The first argument is the job's title (if it contains
multiple words, put quotes around it so it will be a single argument), and
a list of issue id (for recording relevant entries in an external
issue-tracking system).
- p4jnu [job]
-
Undo new job creation. This only works on new jobs, and onliterates
them from the system. This is the only way to delete a job.
Work Jobs
With the exception of p4wb, all these commands will only work
for the developer on the development client, for obvious reasons.
Job Commands
- p4wb [job]
-
Start working on a new job. Creates the branch, creates a development directory
in the current client and populates it. Will work for any developer
anywhere (except integ, which isn't really a developer). The
current user and client become the development user and client for this job.
- p4wbu [job]
-
Undo working on a job. Removes the branch and the development directory; all
work is lost forever, unless previouly backed up.
Increases the attempt counter by one.
- p4we [job]
-
End working on a job. It first verifies the job (as per p4wv), and
then locks all relevant files and submits the job for review.
- p4weu [job]
-
Undo end of working on a job. Which means you've just remembered something
which will cause the reviewer to fail the job anyway, so you quickly withraw
the submission and fix it, hoping nobody noticed :-) Or, you haven't
remembered, and the job has been failed; you need to restart work to
fix whatever is wrong.
- p4ws [job]
-
Synchronize a development directory with updated baseline. This expects
the developer to interact with resolve to handle potential
conflict. The best way is to edit each file - p4 is kind
enough to insert very clear markers into the source files, which allow
resolving the conflicts using any text editor.
- p4wv [job]
-
Verify work on a job. Basically, a predictor whether p4we will
work or not, but can also be used as an easy way to invoke make
or whatever.
File Commands
- p4fd [job] file-name...
-
Open a file for deletion. Informs p4 that the specified file will
not be included in the version created by this job. Will remove
the specified file.
- p4fdiff [job] [diff-flags] file-name...
-
Diff edited file from baseline. This one also works for the integrator
when the job is in integ mode, as it is useful for last-moment
reviews. P4 allows certain flags to be specified for diff
in this command; see the description of the diff command.
- p4fe [job] file-name...
-
Open a file for editing. Informs p4 that the specified file will
be changed in the version created by this job. It will make the file
writable (files are read-only by default).
- p4fn [job] file-name...
-
Open a new file for editing. Informs p4 that the specified file
will be added in the version created by this job. Does not actually create
or modify the file.
- p4fu [job] file-name...
-
Undo operation on an opened file. Informs p4 that whatever has been
said about the file is not true, whatever the operation was. p4
restores the file to its original state (which means refreshing it if it
was deleted or edited). It will not delete files which were added.
Review Job
These commands can be run by any developer (except integ, which is
not really a decveloper). It would have been nice to limit them just to
developers designated as reviewers, and prevent a developer from reviewing
himself, but as fooling the system is so easy (just set P4USER),
what's the point?
- p4rdiff [job] [diff-flags] file-name...
-
View difference for a file for a job. P4 is not kind enough to let
us difference any two files in the depot (not that I know of, anyway), so
you first have to get all the modified files from the development branch
by using p4fg job. The bonus is that you can use any
diff program, and give it any flags you like.
- p4rfail [job]
-
Fail a reviewed job; additional work should be done by the developer.
In theory,
there should be no "bugs" where (assuming that the verify.pl
script runs a good set of regression tests). Even so, there are issues
of style, design, performance, and so on.
While failing their jobs won't make you popular with other developers,
passing bad code won't make you popular with the integrator.
- p4rpass [job]
-
Pass a review job (clearing it for integration). By this the reviewer is
taking as much responsibility for the submitted code as the developer. This
keeps the reviewers honest :-)
Integration Job
- p4ib [job]
-
Start integration of a job. This creates a local integration directory
(ruining whatever copy of the baseline you had), and merges the changes
from the development branch. Due to p4 bug #350, this causes
getting all the added branch files as well, so you'll have to get rid of
them manually later. This bug is really a pain in the rear echelon.
- p4ibu [job]
-
Undo begin integration of a job. Restores the local baseline copy
to the latest version, removes the integration change, and so on.
- p4ie [job]
-
End integration of a job. This is the point of no return. It does a verify
first, of course, as per p4iv, for what it is worth. If a bug made
it this far, it can bask in the warm fuzzy certainty it will take a lot of
time and effort to root it out...
- p4iv [job]
-
Verify integration of a job. Basically, run verify.pl and hope
it checks the right things.
Clear Job
- p4ic [job]
-
Clear integrated job directories. As you've noticed, the above process
leaves a lot of junk after the job is integrated. All the reviewers and
the integrator need to do p4fg job #none, for starters.
Then the developer can destroy the whole works, obliterating the branch.
Locks
In principle, none of these commands should ever be used. The world
being an imperfect place, it is very probable that sometimes scripts will
crash leaving hanging locks and other inconsistencies.
The following commands allow access to the lock operations, so it would
be easy to recover from broken/hanging locks. Other inconsistencies
require manual care using the raw p4 interface.
- p4ld
-
Lock the whole depot. Note that this only stops operations which try
to lock the whole depot; operations that only lock a single job will succeed.
- p4ldu
-
Unlock the whole depot.
- p4lf
-
List all locked files by all jobs.
- p4lj [job]
-
Lock a single job. Note that this will succeed regardless of whether the
whole depot is locked.
- p4lju [job]
-
Unlock a single job.
- p4ll
-
List all (job, depot) locks.
Starting
Set Up New Depot
Add New Developer
In order to add a new developer (and his client) to the system:
-
As integ, add the new user to
the protection lists, using p4 protect.
-
As the new developer, copy the p4init file into the shell
startup file, with the appropriate customizations. Apply it.
-
Call p4 user to create the user.
-
Call p4 client to create the client.
Caveats
This is the second try at getting the system right. The first was intended
to be a much thinner set of wrappers on top of p4, with the goal
to use as much of the built-in p4 commands as possible. This turned
out to be a huge headache, due to problems with almost anything - change lists,
file locks, job status...
The result is that you have to do things through the wrappers exclusively.
It is true that a few of them are just calls to built-in p4
commands, but you've got to go through them in order to get the arguments
right.
An exception to the above is setting things up. There's no automated way to
set up the initial dept, create a new user, a new client, and so on. This
is not too bad since such operations are rare.
A second problem is that these wrappers go only so far. There is some
missing functionality, the output of commands is not consistent (some
is the raw outpout from p4, some is formatted reports), and so on.
The implementation is not very fast, either. It is not that they are
terribly slow, but compared to the lightning-quick response of the raw
p4 interface, they seem a drag. Most of the delays are due to
the horribly inefficent way locks are implemented. If p4
allows locking unopened files (or even better, nonexisting files),
this could be speeded up by a huge factor.