RELEASE TEST PROCEDURE
Richard Brooksby, Ravenbrook Limited, 2001-03-21
1. INTRODUCTION
This is the basic minimum test procedure to be applied to all releases
of the P4DTI software.
The readership of this document is project engineers.
This document is not confidential.
2. PROCEDURE
2.1. Clean your machine. Remove previous installations of P4DTI,
Bugzilla, TeamTrack, and other software.
For a really thorough test, remove all prerequisite software, or
start with a completely virgin machine with a freshly installed
operating system.
2.2. Obtain the release.
- Check that links to the release are correct.
- Check that the release is correctly described.
- Read and follow the instructions that accompany the release (the
readme.txt) file.
2.3. Verify that all the prerequisites are met.
- Ask Perforce for its version and check it against the manual.
- Ask Python what version it is and check it against the manual.
- Inspect the TeamTrack build number.
- Make sure you're running an _unpatched_ Bugzilla.
2.4. Install test repository, databases, etc. from the test materials
for the relevant version (e.g. teamtrack-test.mdb). These should be
constructed to meet the various "procedural prerequisites".
2.5. Copy the configuration for the test machine from the test
materials for the relevant version. Check it over by eye.
2.6. Follow the AG instructions for patching Bugzilla, updating the
registry, etc.
2.7. Start the replicator as described in the manual and compare the
output to that in the manual. Stop it as described. Start it again.
On Linux, test the startup script for starting and stopping the
replicator. On Windows, install the NT service for managing the
replicator and test it. In both cases log out and make sure the
replicator keeps running, then reboot the machine to make sure that
the system starts automatically as described.
2.8. Follow the instructions in the manual for testing the
configuration. (i.e. take an issue through a complete life cycle as
described in the UG, from various Perforce interfaces, and run
"check.py"). Deliberately simulate various user errors and
violations of permissions to check e-mail delivery and accuracy.
Look up error messages in the AG.
2.9. Run any automatic tests to stress the system.
2.10. Change the set of users and a user's e-mail address and re-start
the replicator to make sure it picks up the new configuration, as
described in "Maintaining the P4DTI".
2.11. Change the set of replicated fields and refresh the Perforce jobs
as described in "Maintaining the P4DTI".
2.12. Leave it running for as long as possible.
2.13. Uninstall the integration, following the instructions in the AG.
Make sure that Perforce and the DT are still functioning.
2.14. Test the TeamTrack integration in the three configurations we
support: TeamTrack 4.5, TeamTrack 5.0, and TeamTrack 5.0 with a database
upgraded from TeamTrack 4.5.
3. TESTING WITH MULTIPLE PERFORCE SERVERS
3.1. Follow the instructions in [GDR 2001-11-14, 5] to set up a system
with several Perforce servers, replicators and P4Web instances.
3.2. Start all the P4DTI replicators.
3.3. For each project, go through an issue lifecycle: create it in
TeamTrack, assign it to someone, check that it is replicated to
Perforce, close it in Perforce by fixing it with a changelist, check
that the appropriate transition happens in TeamTrack.
3.4. Check that issues get replicated only to one Perforce server, and
to the right Perforce server.
3.5. Check that the changelist link in TeamTrack goes to the right
instance of P4Web. Check that when two changelists have the same number
in two different Perforce servers, theie descriptions show up properly
in TeamTrack (that is, TeamTrack doesn't confuse changelist 17 on
Perforce server A with changelist 17 on Perforce server B).
3.6. Try simultaneous changes in two or more Perforce servers.
3.7. Try stopping one of the P4DTI replicators. Make some changes while
the replicator is stopped. Restart the replicator. Does it pick up the
changes that happened while it was stopped?
3.8. Try other user activity that you think might break the system.
A. REFERENCES
[GDR 2001-06-29] "Testing the P4DTI with multiple Perforce servers"
(e-mail message); Gareth Rees; Ravenbrook Limited; 2001-06-29;
<http://info.ravenbrook.com/mail/2001/06/29/14-15-22/0.txt>.
[GDR 2001-11-14] "Perforce Defect Tracking Integration Advanced
Administrator's Guide"; Gareth Rees; Ravenbrook Limited; 2001-11-14;
<http://www.ravenbrook.com/project/p4dti/version/2.0/manual/aag/>.
B. DOCUMENT HISTORY
2001-03-21 RB Drafted in e-mail.
2001-03-26 RB Edited into shape as text document in master sources.
2001-07-05 GDR Test TeamTrack in all three configurations.
2002-01-31 GDR Added section on testing with multiple Perforce servers,
based on [GDR 2001-06-29].
C. COPYRIGHT AND LICENCE
This document is copyright (C) 2001 Perforce Software, Inc. All rights
reserved.
Redistribution and use of this document in any form, with or without
modification, is permitted provided that redistributions of this
document retain the above copyright notice, this condition and the
following disclaimer.
THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
$Id: //info.ravenbrook.com/project/p4dti/version/2.0/procedure/release-test/index.txt#2 $