Helix SwarmHelix Swarm

index.html

  • //
  • guest/
  • matt_albrecht/
  • scripts/
  • logDaemon/
  • index.html #2
  • View
  • Commits
  • Open Download .zip Download (6 KB) 
  1. <html>
  2. <head>
  3.     <title>logDaemon - A P4 Journal Monitoring Daemon</title>
  4. </head>
  5. <body>
  6. <h1>logDaemon</h1>
  7. <p>
  8. A Journal Monitoring Daemon<br/>
  9. Written for Python 2.3 on *NIX platforms</br>
  10. Version 1.1</br>
  11. Placed into the Public Domain
  12. </p>
  13.  
  14. <h1>About</h1>
  15. <p>
  16. This tool monitors the live journal file from a Perforce server, and executes a
  17. chain of registered listeners based on certain events.  It is capable of
  18. correctly handling a journal file being truncated by searching for previous
  19. journal files (either compressed or not).
  20. </p>
  21.  
  22. <h1>Using The Tool</h1>
  23. <p>
  24. The first step is to initialize the Settings.py file.  You may either edit the
  25. Settings.py.template file by hand, or run the CreateSettings.py application
  26. to do it for you, based on your existing environment variables.  This should
  27. be the only file you need to alter.
  28. </p>
  29. <p>
  30. If you wish to use the hot-backup listener, then you should follow the steps
  31. below for setting up an SSH communication between the two servers.  This
  32. will mean that both servers should use a minimal-access user for
  33. communication.
  34. </p>
  35. <p>
  36. Since the tool will look in checkpointed journal files, it is a good idea to
  37. keep at least the most recent of those around, so that the daemon can
  38. go back and look up any data it may have missed.  If something happens that
  39. causes the daemon to skip several journal files, including one that was
  40. deleted, then you will need to manually intervene to correct the situation.
  41. </p>
  42. <p>
  43. Finally, you can extend the tool yourself with custom listeners.  See that
  44. section below.
  45. </p>
  46.  
  47. <h1>Tool Details</h1>
  48. <p>
  49. The daemon's main logic is handling the journal file, as it is updated and
  50. truncated during normal checkpoint operations.  The daemon will persist a
  51. file on the machine running it that describes where it last left off
  52. (journal file sequence number and line number), so that the daemon can
  53. correctly recover from either a reboot, abnormal shutdown, or any other
  54. possible terminating event.
  55. </p>
  56. <p>
  57. This little "where was I" file is very important.  It's what ensures that the
  58. remote system is passed exactly the information it needs.  If this file doesn't
  59. exist, then the tool will look backwards through the journals to find the
  60. one corresponding to the checkpoint number specified in Setup.py, under
  61. "SERVER.lastCheckpoint".  This number should represent the last checkpoint
  62. that was recovered on the remote server.
  63. </p>
  64. <p>
  65. If the tool fails to find a journal file in between the current journal and
  66. either the where-was-I file or the lastCheckpoint, then it will fail.  You
  67. will need to manually recover from this point.
  68. </p>
  69.  
  70. <h1>Correcting From Errors</h1>
  71. <p>
  72. The daemon can quit on its own due to several reasons.  The most common is
  73. a mis-configured Setup.py file.
  74. </p>
  75. <p>
  76. Another issue, which can mean serious investigation, is when the daemon
  77. attempts to find a journal file that is no longer on the host computer.
  78. In the case of a simple hot-backup usage, you will need to restore the
  79. remote machine from a recent checkpoint, and update the Setup.py file
  80. to this checkpoint number.  Other tools, such as e-mail reviewing,
  81. you may want to simply send a message to your users saying that no
  82. e-mails were sent during this time period, and that they'll have to
  83. perform a manual monitor themselves.
  84. </p>
  85.  
  86. <h1>Adding Listeners</h1>
  87. <p>
  88. The tool parses through the journal files, invoking the registered list of
  89. listeners on each line encountered.
  90. </p>
  91. <p>
  92. The list of registered listeners is stored in Setup.py, in the JOURNAL
  93. class.  The "listeners" list contains strings of the names of listeners
  94. (including module name) to load.
  95. </p>
  96. <p>
  97. Each listener is constructed once at daemon start-up time, and should
  98. extend the listener.Listener class.  For each line of the Journal that is
  99. processed, first the tool invokes the listener's "isHandled(self, obj)"
  100. method (more on the "obj" argument later).  If this method returns True,
  101. then the tool calls the "listen(self,obj)" method to handle the journal line.
  102. </p>
  103. <p>
  104. If the tool notices a "quiet" period where no Journal activity occurs, the
  105. tool will call each listener's "noData(self, hadData)" method.  Here, hadData
  106. is True if this is the first "noData" call after a period of activity.
  107. </p>
  108. <p>
  109. The object passed to the "isHandled" and "listen" methods is a
  110. Journal.JournalEntry instance.  This contains the following interesting
  111. attributes:
  112. </p>
  113. <ul>
  114.     <li><strong>lines</strong> List of journal lines related to this entry</li>
  115.     <li><strong>fields</strong> List of the database fields parsed out from the lines</li>
  116.     <li><strong>didRotate</strong> Boolean, <code>True</code> if
  117.     there was a Journal rotation right before this entry</li>
  118.     <li><strong>incomplete</strong> Boolean, <code>True</code> if we expect
  119.     more lines in this entry</li>
  120. </ul>
  121.  
  122. <h1>Setting Up SSH For Non-Password Authentication</h1>
  123. <p>
  124. Definitions:
  125. </p>
  126. <pre>
  127.     $REMOTE = remote machine to log into
  128.     $RUSER = remote user on the remote machine to log in as.
  129. </pre>
  130. <p>
  131. From the machine that will run ssh:
  132. </p>
  133. <pre>
  134.     # Create the keys for the local machine
  135.     $ ssh-keygen -t rsa
  136.         (use default location, and no passphrase)
  137.     
  138.     # Install the ssh keys on the remote machine.
  139.     $ scp ~/.ssh/id_rsa.pub $RUSER@$REMOTE:./pubkey
  140.         (this will force you to login and authenticate the server,
  141.         which must be done at least once)
  142.     $ ssh $RUSER@$REMOTE "mkdir ~/.ssh && chmod 700 ~/.ssh ; cat ~/pubkey >> ~/.ssh/authorized_keys ; chmod 600 ~/.ssh/authorized_keys ; rm ~/pubkey"
  143.         (this will force you to login again)
  144.     $ ssh $RUSER@$REMOTE
  145.         (you shouldn't need to log in.  If you do, check the local
  146.         and remote security files for instructions)
  147. </pre>
  148.  
  149. <h1>Standard Legaleze</h1>
  150. <p>
  151. The authors are in no way liable for any damage this tool might do.  You've
  152. been warned: use at your own risk.  Standard administration procedures should
  153. require that you sufficiently test out any new tool before bringing it into
  154. your production environment.  Remember to keep good backups.  This tool
  155. probably won't eat your lunch, but you still need to be careful.
  156. </p>
  157. </html>
# Change User Description Committed
#2 5810 Matt Albrecht New version of logDaemon that has better bug fixes.
 Also added server-status web p...age for a pure client-side AJAX app that pulls together different server-side statistics. «		 18 years ago
#1 5805 Matt Albrecht Update to version 1.3-alpha 18 years ago