Source.pm #13

  • //
  • guest/
  • perforce_software/
  • revml/
  • lib/
  • VCP/
  • Source.pm
  • View
  • Commits
  • Open Download .zip Download (8 KB)
package VCP::Source ;

=head1 NAME

VCP::Source - A base class for repository sources

=head1 SYNOPSIS

=head1 DESCRIPTION

=head1 EXTERNAL METHODS

=over

=cut

use strict ;

use Carp ;
use UNIVERSAL qw( isa ) ;
use VCP::Debug qw( :debug ) ;

use vars qw( $VERSION ) ;

$VERSION = 0.1 ;

use base 'VCP::Plugin' ;

use fields (
   'BOOTSTRAP_REGEXPS', ## Determines what files are in bootstrap mode.
   'DEST',
   'CONTINUE',          ## Set if we're resuming from the prior
                        ## copy operation, if there is one.  This causes
                        ## us to determine a minimum rev by asking the
                        ## destination what it's seen on a given filebranch
) ;


=item new

Creates an instance, see subclasses for options.  The options passed are
usually native command-line options for the underlying repository's
client.  These are usually parsed and, perhaps, checked for validity
by calling the underlying command line.

=back

=cut

sub new {
   my $class = shift ;
   $class = ref $class || $class ;

   my VCP::Source $self = $class->SUPER::new( @_ ) ;

   $self->{BOOTSTRAP_REGEXPS} = [] ;

   return $self ;
}


###############################################################################

=head1 SUBCLASSING

This class uses the fields pragma, so you'll need to use base and 
possibly fields in any subclasses.  See L<VCP::Plugin> for methods
often needed in subclasses.

=head2 Subclass utility API

=over

=item parse_options

    $self->parse_options( \@options, @specs );

Parses common options including whatever options VCP::Plugin parses,
--bootstrap, and --rev-root.

=cut

sub parse_options {
   my VCP::Source $self = shift;
   $self->SUPER::parse_options( @_,
      "b|bootstrap=s"    => sub { $self->bootstrap( $_[1] ) },
      "continue"         => \$self->{CONTINUE},
      "rev-root"         => sub { $self->rev_root( $_[1] ) },
   );
}

=item dest

Sets/Gets a reference to the VCP::Dest object.  The source uses this to
call handle_header(), handle_rev(), and handle_end() methods.

=cut

sub dest {
   my VCP::Source $self = shift ;

   $self->{DEST} = shift if @_ ;
   return $self->{DEST} ;
}


=item continue

Sets/Gets the CONTINUE field (which the user sets via the --continue flag)

=cut

sub continue {
   my VCP::Source $self = shift ;

   $self->{CONTINUE} = shift if @_ ;
   return $self->{CONTINUE} ;
}



=back

=head1 SUBCLASS OVERLOADS

These methods should be overridded in any subclasses.

=over

=item copy

REQUIRED OVERLOAD.

   $source->copy_revs() ;

Called by L<VCP/copy> to do the entire export process.  This is passed a
partially filled-in header structure.

The subclass should call this to move all the revisions over to the
destination:

   $self->SUPER::copy_revs( $revs );

If $revs, an ARRAY containing revisions, is not passed in,
$self->revs->remove_all() is used.

=cut

sub copy_revs {
   my VCP::Source $self = shift ;
   my ( $revs ) = @_;
   $revs ||= $self->revs->remove_all;
   VCP::Revs->set_file_fetcher( $self );
   for my $i ( 0..$#$revs ) {
      $self->dest->handle_rev( $revs->[$i] );
      $revs->[$i] = undef;
   }
}


=item fetch_files

Calls get_rev( $r ) for each parameter.

Overload this if you can batch requests more efficiently.

=cut

sub fetch_files {
   my VCP::Source $self = shift ;
   map $self->get_rev( $_ ), @_;
}


=item handle_header

REQUIRED OVERLOAD.

Subclasses must add all repository-specific info to the $header, at least
including rep_type and rep_desc.

   $header->{rep_type} => 'p4',
   $self->p4( ['info'], \$header->{rep_desc} ) ;

The subclass must pass the $header on to the dest:

   $self->dest->handle_header( $header ) ;

=cut

sub handle_header {
   my VCP::Source $self = shift ;

#   my ( $header ) = @_ ;

   confess "ERROR: copy not overloaded by class '", ref $self, "'.  Oops.\n";
#      if $self->can( 'handle_header' ) eq \&handle_header ;

#   $self->dest->handle_header( $header ) ;
}


=item handle_footer

Not a required overload, as the footer carries no useful information at
this time.  Overriding methods must call this method to pass the
$footer on:

   $self->SUPER::handle_footer( $footer ) ;

=cut

sub handle_footer {
   my VCP::Source $self = shift ;

   my ( $footer ) = @_ ;

   $self->dest->handle_footer( $footer ) ;
   VCP::Revs->set_file_fetcher( undef );
}


=item parse_time

   $time = $self->parse_time( $timestr ) ;

Parses "[cc]YY/MM/DD[ HH[:MM[:SS]]]".

Will add ability to use format strings in future.
HH, MM, and SS are assumed to be 0 if not present.

Returns a time suitable for feeding to localtime or gmtime.

Assumes local system time, so no good for parsing times in revml, but that's
not a common thing to need to do, so it's in VCP::Source::revml.pm.

=cut

{
    ## This routine is slow and gets called a *lot* with duplicate
    ## inputs, at least by VCP::Source::cvs, so we memoize it.
    my %cache;

    sub parse_time {
       my VCP::Source $self = shift ;
       my ( $timestr ) = @_ ;
       return $cache{$timestr} ||= do {
           ## TODO: Get parser context here & give file, line, and column. filename
           ## and rev, while we're scheduling more work for the future.
           confess "Malformed time value $timestr\n"
              unless $timestr =~ /^(\d\d)?\d?\d(\D\d?\d){2,5}/ ;
           my @f = split( /\D/, $timestr ) ;
           --$f[1] ; # Month of year needs to be 0..11
           push @f, ( 0 ) x ( 6 - @f ) ;
           require Time::Local;
           return Time::Local::timelocal( reverse @f ) ;
        }
    }
}


=item bootstrap

Usually called from within call to GetOptions in subclass' new():

   GetOptions(
      'bootstrap|b=s' => sub {
	 my ( $name, $val ) = @_ ;
	 $self->bootstrap( $val ) ;
      },
      'rev-root'      => \$rev_root,
      ) or $self->usage_and_exit ;

Can be called plain:

   $self->bootstrap( $bootstrap_spec ) ;

See the command line documentation for the format of $bootstrap_spec.

Returns nothing useful.

=cut

sub bootstrap {
   my VCP::Source $self = shift ;
   my ( $val ) = @_ ;
   require Regexp::Shellish;
   $self->{BOOTSTRAP_REGEXPS} = [
      map Regexp::Shellish::compile_shellish( $_ ), split /,+/, $val
   ];

   return ;
}


#=item bootstrap_regexps
#
#   $self->bootstrap_regexps( $re1, $re1, ... ) ;
#   $self->bootstrap_regexps( undef ) ; ## clears the list
#   @res = $self->bootstrap_regexps ;
#
#Sets/gets the list of regular expressions defining what files are in bootstrap
#mode.  This is usually set by L</bootstrap>, though.
#
#=cut
#
#sub bootstrap_regexps {
#   my VCP::Source $self = shift ;
#   $self->{BOOTSTRAP_REGEXPS} = [ @_ == 1 && ! defined $_[0] ? () : @_ ]
#      if @_ ;
#   return @{$self->{BOOTSTRAP_REGEXPS}} ;
#}
#
=item is_bootstrap_mode

   ... if $self->is_bootstrap_mode( $file ) ;

Compares the filename passed in against the list of bootstrap regular
expressions set by L</bootstrap>.

The file should be in a format similar to the command line spec for
whatever repository is passed in, and not relative to rev_root, so
"//depot/foo/bar" for p4, or "module/foo/bar" for cvs.

This is typically called in the subbase class only after looking at the
revision number to see if it is a first revision (in which case the
subclass should automatically put it in bootstrap mode).

=cut

sub is_bootstrap_mode {
   my VCP::Source $self = shift ;
   my ( $file ) = @_ ;

   my $result = grep $file =~ $_, @{$self->{BOOTSTRAP_REGEXPS}} ;

   lg(
      "$file ",
      ( $result ? "=~ " : "!~ " ),
      "[ ", join( ', ', map "qr/$_/", @{$self->{BOOTSTRAP_REGEXPS}} ), " ] (",
      ( $result ? "not in " : "in " ),
      "bootstrap mode)"
   ) if $result || debugging;

   return $result ;
}

=back

=head1 COPYRIGHT

Copyright 2000, Perforce Software, Inc.  All Rights Reserved.

This module and the VCP package are licensed according to the terms given in
the file LICENSE accompanying this distribution, a copy of which is included in
L<vcp>.

=head1 AUTHOR

Barrie Slaymaker <barries@slaysys.com>

=cut

1
# Change User Description Committed
#46 5404 Barrie Slaymaker - SVN support added
- Makefile gives clearer notices about missing optional
  prereqs.
- VCP::Filter::labelmap and VCP::Filter::map: <<skip>> replaces
  deprecated <<delete>> to be clearer that no revisions
  are deleted from either repository but some just are
  skipped and not inserted.
- VCP::Filter::map: support added for SVN-like branch labels
- VCP::Source: support added for ISO8601 timestamps
  emitted by SVN.
#45 5082 Barrie Slaymaker - VCP::Source tells VCP::Rev to uncache the source to allow
  the source instance to be DESTROYed and thus clean up its
  working files.
#44 5078 Barrie Slaymaker - VCP::Source::parse_time() 0s out undefined/missing fields
#43 4500 Barrie Slaymaker - Minor POD cleanup
#42 4497 Barrie Slaymaker - --rev-root documented
       - All destinations handle rev_root defaulting now
#41 4487 Barrie Slaymaker - dead code removal (thanks to clkao's coverage report)
#40 4135 Barrie Slaymaker - Time fields may have trailing AM/PM or A/P without leading whitespace
#39 4134 Barrie Slaymaker - "AM", "PM", "A", and "P" (case insensitive) are now parsed
  properly when parsing time values
#38 4039 Barrie Slaymaker - VCP::Source::scan_metadata() API now in place,
- VCP::Source::copy_revs() is fully deprecated.
#37 4021 Barrie Slaymaker - Remove all phashes and all base & fields pragmas
- Work around SWASHGET error
#36 3982 Barrie Slaymaker - VCP::Source no longer leaks memory by delete()ing from a phash
- VCP::Source::cvs now flushes to disk more often to conserve RAM
#35 3970 Barrie Slaymaker - VCP::Source handles rev queing, uses disk to reduce RAM
- Lots of other fixes
#34 3922 Barrie Slaymaker - More paranoid paramter checking
#33 3916 Barrie Slaymaker - Reduce memory consumption
#32 3907 Barrie Slaymaker - Debugging cleanups
#31 3898 Barrie Slaymaker - VCP::Source::* --rev-root reinstanted
#30 3855 Barrie Slaymaker - vcp scan, filter, transfer basically functional
    - Need more work in re: storage format, etc, but functional
#29 3835 Barrie Slaymaker - VCP::Source supports queuing of revs and facilities for
  sending revs ASAP to conserve memory
#28 3820 Barrie Slaymaker - VCP::Source::revml now uses VCP::Source's queueing methods
    - For maintainability only, does not decrease memory util.
#27 3819 Barrie Slaymaker - Factor send & queueing of revs up in to VCP::Source
#26 3811 Barrie Slaymaker - fetch_*() and get_rev() renamed get_source_file()
#25 3806 Barrie Slaymaker - VCP::Source no longer tries to send to a missing dest
#24 3804 Barrie Slaymaker - Refactored to prepare way for reducing memory footprint
#23 3706 Barrie Slaymaker - VCP gives some indication of output progress (need more)
#22 3687 Barrie Slaymaker - Destinations may now use compile_path_re()
#21 3681 Barrie Slaymaker - VCP now scans much more of real_vss_1 and converts it to revml
#20 3679 Barrie Slaymaker - VCP::Source::vss respects --case-sensitive in more places
#19 3677 Barrie Slaymaker - rev_root sanity check is now case insensitive on Win32
- Parens in source filespecs are now treated as regular
  characters, not capture groups
- ** is not treated as '...'
#18 3477 Barrie Slaymaker - Make --rev-root only available in VCP::Source::p4
#17 3460 Barrie Slaymaker - Revamp Plugin/Source/Dest hierarchy to allow for
  reguritating options in to .vcp files
#16 3445 Barrie Slaymaker - Don't misparse YYYY/MM/DD dates as MMMM/DD/YY.
- t/61sort.t no longer blows up due to VCP::Rev's new
  BUG checks.
#15 3443 Barrie Slaymaker - Use BUG instead of Carp::confess
- Recognize MM/DD/YY format dates
#14 3157 Barrie Slaymaker debug conversion to VCP::Logger
#13 3155 Barrie Slaymaker Convert to logging using VCP::Logger to reduce stdout/err spew.
       Simplify & speed up debugging quite a bit.
       Provide more verbose information in logs.
       Print to STDERR progress reports to keep users from wondering
       what's going on.
       Breaks test; halfway through upgrading run3() to an inline
       function for speed and for VCP specific features.
#12 3133 Barrie Slaymaker Make destinations call back to sources to check out files to
       simplify the architecture (is_metadata_only() no longer needed)
       and make it more optimizable (checkouts can be batched).
#11 3131 Barrie Slaymaker Double the speed of the RCS file parser.
       Deprecate VCP::Revs::shift() in favor of remove_all().
#10 2824 John Fetkovich removed CVS_CONTINUE field from Source/cvs.pm, and added
       CONTINUE field and continue accessor to Source.pm.  Moved parsing
       of the --continue option also.
#9 2809 Barrie Slaymaker Implement --repo-id in Plugin.pm, refactor source & dest
       options parsing starting in VCP::Source::cvs (need to
       roll out to other sources and dests), get t/91cvs2revml.t
       passing again (first time in months! branching and
       --continue support works in cvs->foo!).
#8 2453 John Fetkovich removed compilation of revml.
 will be making that a separate executable.
#7 2293 Barrie Slaymaker Update CHANGES, TODO, improve .vcp files, add --init-cvs
#6 2015 Barrie Slaymaker submit changes
#5 1998 Barrie Slaymaker Initial, revml and core VCP support for branches
#4 1809 Barrie Slaymaker VCP::Patch should ignore lineends
#3 628 Barrie Slaymaker Cleaned up POD in bin/vcp, added BSD-style license.
#2 468 Barrie Slaymaker - VCP::Dest::p4 now does change number aggregation based on the
  comment field changing or whenever a new revision of a file with
  unsubmitted changes shows up on the input stream.  Since revisions of
  files are normally sorted in time order, this should work in a number
  of cases.  I'm sure we'll need to generalize it, perhaps with a time
  thresholding function.
- t/90cvs.t now tests cvs->p4 replication.
- VCP::Dest::p4 now doesn't try to `p4 submit` when no changes are
  pending.
- VCP::Rev now prevents the same label from being applied twice to
  a revision.  This was occuring because the "r_1"-style label that
  gets added to a target revision by VCP::Dest::p4 could duplicate
  a label "r_1" that happened to already be on a revision.
- Added t/00rev.t, the beginnings of a test suite for VCP::Rev.
- Tweaked bin/gentrevml to comment revisions with their change number
  instead of using a unique comment for every revision for non-p4
  t/test-*-in-0.revml files.  This was necessary to test cvs->p4
  functionality.
#1 467 Barrie Slaymaker Version 0.01, initial checkin in perforce public depot.