TODO for VCP - RETEST ON WINNT - TEST ON Win2K - TEST ON WinXP *NOTE* Doing a `grep -r TODO lib bin` will find lots of small things and future feature ideas. This file will grow to include the more important of those. - Bugfixes - if vcp::source::p4 fails to create temporary p4 client, don't try to delete it, or at least, don't complain when the delete fails. - deduce_rev_root should be called after all metadata is extracted (or perhaps as it is being extracted) and deduce the root from all of the revisions extracted. This will eliminate the need for --rev-root to be used with VCP::Source::p4's --follow-branch-into, for instance. - VCP::Source::cvs should either ignore a leading "/" before the module name or complain about it. - can't exe p4 argument list too long Run3.pm 428. I had done a xc/doc/hardcopy/... and a xc/doc/man/... conversion. [possibly fixed; needs to be tested] - xfree86 p4->cvs conversion error: vcp: no rev number found in cvs commit log output for X11R6/misc/INSTALL.ms(//depot/X11R6/misc/)#2 @3 (text) edit 1996-02-10 00:00:00Z dawes@vcp_tmp_4752 "X11R6.1 public patch 0\n": cvs revs parsed: $VAR1 = {}; - Port VCP::*::vss to use state files - Clean up vcp_#### integrate spec. - Implement branching in VCP::*::vss - Port to Win32 (ie tweak IPC::Run3 to work there) - Make it so that a dest of cvs:blah:foo is either an error or is just like cvs:blah:foo/... (if it looks like a directory given the source revs or the target repository topology) - Prevent keyword expansion on all checkouts. Found by Thomas Quinot - Get detailed spec of legal names (filenames, branch spec/label names) for p4. * Carry executable bit through (Nick Ing-Simmons) * Make , , etc. use binary escapes when needed * VCP::Dest::cvs needs to set the binary mode properly on files it creates and checks in [[check status]] * VCP::Source::cvs needs to deduce binary modes correctly [[check status]] - there should be an option to detect collisions between underscorified tags in VCP::Dest::cvs. Perhaps enabled by default. - use the upcoming -t option for p4 filelog (expected in 2002.2) to extract times with dates. Issue a warning recommending an upgrade or enabling a metadata reading mode if not present. Warn about metadata loss when using older p4. - Review cvs2p4 and (in the guest depot) vss2p4 for gotchas they catch that we should [[STATUS: tag underscorification is done and we've checked that cvs does not limit its tag lengths]] - get p4 forms parsing to lex *exactly* like p4 does it. - fix docs to give .vcp file examples before command line examples - Testing * Test the VCP::Source::cvs -r and -d options (-r *was* tested before --continue was added; now its not tested at all. Plus, -d was never tested and is not fully implemented in the RCS file scanning code. - VCP::TestUtils & all tests to using IPC::Run3, for speed and portability's sake. * use get_vcp_output() in the test suite whereever we're extracting revml. * test -r options in t/91cvs2revml.t. Need to test each of (start, end, start+end) with both rev numbers and tags - Test VCP::Source::cvs branching corner conditions, perhaps all corner conditions for this sort of thing. - when the first rev on a branch is a delete - when the first rev on a branch occurs as the first change in an incremental export - when a file is added on a branch - Test importing two bootstrap imports one on top of the other - should VCP warn when it detects this? - Test CVS vendor tags in cvs->p4 and cvs->cvs; not sure how to do this best; there is no VCP internal designation of a vendor branch. Such a flag would also have to be visible in RevML and would be needed for testing purposes. Right now, CVS seems to make vendor branches appear as branches of rev 1.1, which is OK, I think, but it means that the first p4 integrate looks backwards. - Test aliased CVS tags - Test unlabelled CVS branches - Feature Adds - Convert all ui_machines to use in_edit_mode and $is_current_value; - Converted so far: - VCP.stml - VCP-Source-vss.stml - VCP-Dest-vss.stml - Get all VCP-*-*.stml to use new_source and new_dest - Add more default filter wizards to DefaultFilters.pm, particularly 'vss2p4_default_filters' and maybe 'p42vss_default_filters' - config file writing - Need to add comments. - Perhaps tell the user that it's being written. - Add the following hints for VSS-> conversions (from gerry@perforce.com) ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Subject: RE: [p4] Migration hints from Visual SourceSafe Date: Sun, 13 Jul 2003 09:45:37 +0200 From: "Yariv Sheizaf" To: , Hi Thierry, We just finished to convert to big (3GB each) VSS databases to Perforce using the VSS2P4 script. The script itself running coorrectly, and its installation is easy. Pay attention to two elements: 1. Locate all components required for the conversion process: VSS DB, Perforce server+DB, Vss2P4 script - on the same machine. It reduces the conversion time very much (6 hours instead of 6 days). 2. Run "analyze" and "analyze -f" if needed on the VSS DB until it fixed - no error messages should be appeared. In any case that the VSS DB is not consistent (e.g. - does not passing "analyze" correctly) - the conversion process will fail. Regards, Yariv ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| - Finish Interactive UI - allow default values in prompts - test suites - This should warn and then ask whether they want to keep the value anyway: ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Enter the revml repo spec. 'revml:' will be prefixed to it then parsed as if it was entered on the vcp command line. revml filespec? foo.revml No such file or directory: /home/barries/src/VCP/foo.rebml ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| - gerry@perforce.com says: - Better examples needed for things like CVSROOT - Clean up the docs - Bundle HTML docs with the app, don't require HTML::Links as a prerequisite in the CPAN dist or vcp.pl file. - Allow the underlying executable name to be set as an option. - Transfer and translate branch specs and other metadata on p4->p4 transfers. - option to lock label specs (integrate specs)? - Handle differently localized timestamps. RevML should be in GMT or use ISO8601's tz indicators. - Allow vcp to log *everything* it outputs to make it easier for the user to capture things, and log it all by default. Right now, the user needs to use some external redirect or tee utility, which might not be available. - branch_label fields in the sections to allow cvs branch labels to be carried through to p4 and, possibly, p4 branch spec names to be assigned and used as directory names for branches. - VCP::Filter::branchmap to allow elements to be altered, for instance by assigning names to p4 branches' elements given their elements (which are a path to the parent directory of each file). - VCP::Filter::usermap to allow usernames to be altered - VCP::Filter::commentedit to allow comments to be edited (post-sort but pre change aggregation); this is to allow cvs2p4-like comments. - VCP::Filter::changes to allow change numbers to be created, edited, or removed. - warn when a *Map filter is used and the default <> action fires, because this indicates a possible missing or faulty rule. - improve error reporting for .vcp files. Either use recursive descent to delegate each value to the appropriate object or capture line and column with each value for error reporting. - in foo->p4 (where foo != p4), try to integrate - in p4->foo suppress revs with empty deltas immediately after an integrate so as to not introduce changless revs in other repos, including ->p4 conversions when --change-branch-rev-1 not specified. - give better diagnostics when the state file appears to be out of date: - when --continue is specified we can't. - when --continus is not specified and a revision already exists in the destination state files (but not in the destination rep). We could keep track of the last known change to the destination in the state file and probe to see if that file is at the indicated revision. Or just watch for revisions coming along that should be in the dest (according to the state file) but aren't. - Add a --skip-unchanged-revs option or VCP::Filter that skips unchanged revisions; all children of such a revision become children of that revision's parent. - Enable an "--append-revs" flag to allow a bootstrap file to be added. This is dangerous (there's no checking to be sure that the first new version is the first version after the existing version in the repository) but useful. This might be done already with --bootstrap specified on the source, but is completely untested - reports and queries against the state files to show: - head rev of each filebranch - what source revs ended up where (path & rev_id) - how branches got mapped - what the main branch_id is for foo->cvs imports. - Need to make the transfers more transactional, so we can recover from where we left off when something fails. We're part way there with the --continue support, but VCP needs to log what it's about to submit and sniff out how far the submittal got before it blew up. This would allow recovery by updating the state files to the correct state and not trying to double commit files. - Allow the state files to be checked in to the destination. Probably as text, in order to avoid sdbm byte ordering issues if they are checked out on a differently byte ordered system. - Perhaps allow keyword expansion, but convert the expanded texts so that they are no longer seens as RCS style keywords. This would allow imported files to have a "stamp of origin" in them. Would also need an option to leave the keywords in place in this case, since the user might presumably want expansion to work correctly in the new repository too. Suggested by Thomas Quinot . - Add a link checker to vcp html * An option to not bring over deleted files Steve James - A report destination that offers a preview of what a transfer will do, with summary and long views. - Limit the number of NtLkLy queries per command to prevent server lockup. Steve James (possibly URGENT, need to test). - VCP::Source::cvs guess what you mean when you specify a starting and/or ending tag using -r. The underlying cvs implementation emits all revisions for files that don't match the -r (assuming, I guess, that all of the file's is within the tagged limits). So VCP::Source::cvs looks through the files that did match the tags for the oldest and newest times of revisions and throws away revisions from files that *didn't* match and are older or newer than the oldest or newest. TODO: perhaps do this culling after change aggregation somehow so that an untagged file's rev that gets associated with a particular change gets included. * Set CVS_PASSFILE for all cvs invocations to prevent mucking with the users' current .cvspass - Use ptys to handle CVS login, if available. Recomend installing IO::Pty if needed but not installed. - finish off VCP::Source::cvs's simulation of the cvs (rlog) -r option - implement VCP::Source::cvs -d option when scanning files - PERHAPS checksum all non-binary files line by line, removing all \r's in order to reduce sensitivity to varying platform settings between the source and the destination. - allow VCP::{Source,Dest}::* to "sniff" at unknown directories / files to see if they can detect what kind of repository is there. This will make schemes optional, so tab completion will work again. - Efficiency - Batch cvs checkouts somehow (they're cvs prints now). - Batch p4 integrates by using a temporary branch spec - extend VCP::Source::cvs to build revisions directly from the RCS files, this will probably mean memorizing the offsets of the delta or full text chunk for each version in the RCS file, then applying them all as needed to get the desired version. They may need to be reversed as a speed hack since RCS files tend to store the most recent revision in full text and uses deltas from that to encode older revisions, and we'll probably want the oldest revision first. This means that we can build the more recent revisions from the older revisions by reversing the deltas as we apply them to build the older revisions, then apply those reversed deltas. Or something; not sure what's best here. - VCP::Source::revml should only keep on hand the versions it needs at each moment in order to conserve disk space. The problem with this is that the RevML may be coming from a non-seek()able byte stream, like STDIN, so we need to patch as we go. One alternative is to cache the revml off to the side and rescan it if this happens. Another is to only patch- as-you-go if the input is non-seek()able. NOTE: P'haps VCP::Source::revml and VCP::Source::cvs can share the RCS file scanner. NOTE: P'haps VCP::Source::revml and VCP::Source::cvs can share the an internal file revision format; RCS is bass ackwards for our needs. - Once a version has been used, it should be cleaned up out of the source directory unless it will be needed again. This used to be the way things were done, but it has been changed to support branching. STATUS: partially implemented. Revs now refer directly to their ancestor revs, and thus revs may be freed willy-nilly and needed revs won't disappear. TODO: Have each ::Dest do $r->previous( undef ) as aggressively as possible so that older revs will be garbage collected as needed. - in --continue mode, VCP::Source::p4 could do a p4 files to get all the source_filebranch_ids and then get the last_rev_in_filebranch for each, which would probably be a lot quicker than running a full filelog and throwing away most of the data (ie for a --continue on a large tree with lots of changes, but only one or two files have changed since the last export). * Consider offering (david d zuhn ) - benchmark different strategies for IPC::Run3's buffers, see if we can patch it to be faster by using a pipe. - the RevMapDB should be purged of any revisions descended from a revision being transferred. Right now, if you restore the set repository from an earlier backup and don't rewind the vcp_state directory, you will end up with a mix of RevMapDB entries from the prior transfer and the current. Fo now, a warning is generated. - Cleanup - Factor VCP::{Source,Dest}::new()s up in to VCP::Driver - Rename parse_foo_repo_spec() in VCP::Utils::foo to parse_repo_spec and make sure VCP::Utils is first in all use base;es. - use VCP::Utils::is_win32() whereever $^O is tested for win32. - Refactor File::Spec->rel2abs( ..., start_dir ) calls in to a new VCP::Utils::start_dir(); - Refactor $self->work_path( "co" ... ) calls in to a VCP::Plugin::checkout_path( ... ) or generalize the concept of a repository workspace. - use parent/child nomenclature instead of previous/next - have VCP::Rev use a next pointer, perhaps in a %next, as VCP::Dest does, to ensure garbage collection. - Centralize underscorification of tag names for VSS. - Rejected - VCP::Source::p4 should be able to create and read a metadata dump as an option. Watch out for different schemas in different p4d versions. Q: Read the btree files directly? 'twould be faster and more space efficient. (rejected by Perforce so as to not tie VCP to p4d's schema; this may still be contributed as OSS by others, but will not be done by the core team any time soon unless the plans change) - VCP::Dest::p4 should write a metadata file directly, and be able to merge new data in to a destination's exported metafile for reimport. Q: Write the btree files directly? This would bypass any checking p4d does on recovering from a metadata file. (rejected by Perforce so as to not tie VCP to p4d's schema; this may still be contributed as OSS by others, but will not be done by the core team any time soon unless the plans change) - An option to prefix all labels with some user-defined string Steve James (this is no longer necessary, as vcp does not add its own labels by default).