log2sql.py #26

  • //
  • guest/
  • perforce_software/
  • utils/
  • log_analyzer/
  • psla/
  • log2sql.py
  • Commits
# Change User Description Committed
#26 25220 Robert Cowham Moved project files to new location: //guest/perforce_software/log-analyzer/psla/...
Required by Swarm project structure for workshop.
#25 24991 Robert Cowham Add tests for all TableUse values.
Fix 'locks acquired by blocking after' processing
#24 24674 Robert Cowham handle [] in api names
#23 24639 sgcommon Correct usage and rpc tracking values.

Per the schema for the process table, the values were being written in
the wrong order. They are now written in the correct order making queries
return the expected results for usage and rpc track values.

Test data updated accordingly.
#22 24287 Robert Cowham Refactor - seperate concerns for line parsing
#21 24286 Robert Cowham Minor perf tweak
#20 24285 Robert Cowham Improve performance detecting block end
#19 24256 Robert Cowham Make sure columns are correct
#18 24253 Robert Cowham Handle UTF8 errors
#17 24170 Robert Cowham Improve regex and remove special case for pull command
#16 24169 Robert Cowham More replica pull tweaks
#15 24168 Robert Cowham Process edge server log pull records
Some tweaks to tests required.
#14 23977 Robert Cowham Minor refactor
#13 23934 Robert Cowham Clean database and auto-create on log upload
#12 23923 Robert Cowham Added tests for compressed files gzip and zip.
Fix for python2
#11 23922 Robert Cowham Fix bzip2 decode in python 3
#10 23921 Robert Cowham Handle reading .gz/.zip
#9 23920 Robert Cowham Handle multiple log files
#8 23919 Robert Cowham Tidied defaults for reporting interval and dbname
#7 23897 Robert Cowham Handle remote journal pulls.
Increase size of process key in case of duplicates
#6 23803 Robert Cowham Trim newlines - unfortunately all md5s different in tests!
#5 23802 Robert Cowham Handle trigger lapse output when track=1
tableUse has extra field: triggerLapse
#4 23798 Robert Cowham Added explanatory comments.
Minor refactoring
#3 23786 Robert Cowham Remove isolation level - doesn't seem to affect performance
#2 23785 Robert Cowham Tweak interface
Remove unused module
#1 23765 Robert Cowham Moved things down one level to psla dir to make it easier to see what belongs
//guest/perforce_software/utils/log_analyzer/log2sql.py
#4 23719 Robert Cowham Log warnings for insert exceptions
#3 23683 Robert Cowham Bring in latest changes - performance improvements
#2 23636 Robert Cowham Merge in latests changes to log2sql.py
- much faster
- better at detecting completed commands and out of order vtrack
- full test harness
#1 22465 Robert Cowham Added tests for r99.2 created depot (no filesize/digest stored)
Fixed problem with preview failing comparison
Add --p4d option to tests to use different p4d version
//guest/robert_cowham/perforce/utils/log_analysis/log2sql.py
#22 21796 Robert Cowham Handle rmt-FileFetch records which may be duplicates as they don't have 'completed' entries.
#21 21780 Robert Cowham Don't drop tables when creating sqlite db
Performance tweaks after profiling - particularly date parsing.
#20 21713 Robert Cowham Ignore temporary error inserting into trackuse table
#19 21712 Robert Cowham Check for correct line number when looking to update for track entries
#18 21708 Robert Cowham Tweak parameters to improve error messages.
Make output to stdout optional
#17 21705 Robert Cowham Minor refactoring for clarity.
Process all completed records as we go.
#16 21702 Robert Cowham Rework algorithm to process blocks at a time, and to avoid the SQL update statements.
Risk of increased memory vs improved speed.
#15 21697 Robert Cowham Parse rcpSnd/rpcRcv and add to database.
#14 21694 Robert Cowham Handle null values properly
#13 21672 Robert Cowham Improve performance when inserting rows
#12 21660 Robert Cowham Handle utf8 in python2 strings
#11 21656 Robert Cowham Avoid SQL quoting issues with alternative insert statement.
#10 21643 Robert Cowham Handle git-fusion entries (strip the json) - for --sql option
#9 21402 Robert Cowham Filter out swarm commands
Test for db content in a basic fashion.
#8 21393 Robert Cowham Add a swarm test - large record.
Improve performance by skipping easy matches early.
#7 21378 Robert Cowham Batch updates and make SQL date arithmetic work with SQLite
#6 21374 Robert Cowham Write to sqllite db
#5 21368 Robert Cowham Make tests work under python3
#4 21367 Robert Cowham Restructured for unit tests, and first unit tests.
#3 21366 Robert Cowham Fix problems with NULL values and int
#2 21365 Robert Cowham Equivalent of Ruby version on smaller sets
#1 21361 Robert Cowham Sort of works...