README.header_scan_cache #1

The header scan was originally implemented by Craig McPheeters.  Extensions
to the initial implementation were done by Matt Armstrong.  One of the
extensions is the documentation in this file, which other than this 
comment, is left un-altered from Matt's version.

The comment about changes in the Jambase are not true in this branch.  See
  //guest/matt_armstrong/jam/hdrscan_cache/...
for details.

Craig McPheeters, Jan, 2002.

---

This code is taken from //guest/craig_mcpheeters/jam/src/ on the
Perforce public depot.  Many thanks to Craig McPheeters for making his
code available.  It is delimited by the OPT_HEADER_CACHE_EXT #define
within the code.

Jam has a facility to scan source files for other files they might
include.  This code implements a cache of these scans, so the entire
source tree need not be scanned each time jam is run.  This brings the
following benefits:

    - If a file would otherwise be scanned multiple times in a
      single jam run (because the same file is represented by
      multiple targets, perhaps each with a different grist), it
      will now be scanned only once.  In this way, things are
      faster even if the cache file is not present when Jam is
      run.

    - If a cache entry is present in the cache file when Jam
      starts, and the file has not changed since the last time it
      was scanned, Jam will not bother to re-scan it.  This
      markedly increaces Jam startup times for large projects.

This code has improvements over Craig McPheeters' original
version.  I've described all of these changes to Craig and he
intends to incorporate them back into his version.  The changes
are:

    - The actual name of the cache file is controlled by the
      HCACHEFILE Jam variable.  If HCACHEFILE is left unset (the
      default), reading and writing of a cache file is not
      performed.  The cache is always used internally regardless
      of HCACHEFILE, which helps when HDRGRIST causes the same
      file to be scanned multiple times.

      Setting LOCATE and SEARCH on the the HCACHEFILE works as
      well, so you can place anywhere on disk you like or even
      search for it in several directories.  You may also set it
      in your environment to share it amongst all your projects.

    - The .jamdeps file is in a new format that allows binary data
      to be in any of the fields, in particular the file names.
      The original code would break if a file name contained the
      '@' or '\n' characters.  The format is also versioned,
      allowing upgrades to automatically ignore old .jamdeps
      files.  The format remains human readable.  In addition,
      care has been taken to not add the entry into the header
      cache until the entire record has been successfully read from
      the file.

    - The cache stores the value of HDRPATTERN with each cache
      entry, and it is compared along with the file's date to
      determine if there is a cache hit.  If the HDRPATTERN does
      not match, it is treated as a cache miss.  This allows
      HDRPATTERN to change without worrying about stale cache
      entries.  It also allows the same file to be scanned
      multiple times with different HDRPATTERN values.

    - Each cache entry is given an "age" which is the maximum
      number of times a given header cache entry can go unused
      before it is purged from the cache.  This helps clean up old
      entries in the .jamdeps file when files move around or are
      removed from your project.

      You control the maximum age with the HCACHEMAXAGE variable.
      If set to 0, no cache aging is performed.  Otherwise it is
      the number of times a jam must be run before an unused cache
      entry is purged.  The default for HCACHEMAXAGE if left unset
      is 100.

    - Jambase itself is changed.

      SubDir now always sets HDRGRIST to $(SOURCE_GRIST) so header
      scanning can deal with multiple header files of the same
      name in different directories.  With the header cache, this
      does no longer incurs a performance penalty -- a given file
      will still only be scanned once.

      The FGristSourceFiles rule is now just an alias for
      FGristFiles.  Header files do not necessarily have global
      visibility, and the header cache eliminates any performance
      penalty this might otherwise incur.

Because of all these improvements, the following claims can be
made about this header cache implementation that can not be made
about Craig McPheeters' original version.

    - The semantics of a Jam run will never be different because of
      the header cache (the HDRPATTERN check ensures this).

    - It will never be necessary to delete .jamdeps to fix obscure
      jam problems or purge old entries.