README #1

The Perforce Client/Service Protocol

    Message Passing

	Though called RPC, it is in fact a message passing mechanism.
	The difference being that in a message passing scheme the response
	is both asynchronous and optional, and the driver of the connection
	is a matter of agreement rather than predestiny.

	Each message is a list of variable/value pairs, with one variable
	('func') whose value is associated with some function call.

	Rpc::Invoke() sends a single message and Rpc::Dispatch() loops,
	reading messages and executing the associated function calls,
	until a special 'release' message is received.

    Basic Flow

   	The client sends a message containing the user's request to the
	server and then dispatches, i.e. reads and executes requests from
	the server until the server sends the "release" message.

	The server initially dispatches, i.e. reads and executes a
	request from the user.  The final act of the request should
	either be to release the client (send the "release" message),
	or send a message to the client instructing it to invoke
	(eventually) another server request.  In the end, the final
	message sent to the client should be "release".

    Notes on flow control.

	This collection of functions orchestrates the flow control when
	data must move in both directions between the client and server
	at the same time.  This is done without async reads or writes.
	Instead we rely on cues from our caller for the amount of data
	expected to make a round trip, and periodically stop writing
	and read for a while.

	Normally, the caller uses Invoke() to launch functions to the
	other side, and then Dispatch() (which calls DispatchOne()) to
	listen for responses.  Dispatch() returns when the other side
	launches a "release" or "release2" in our direction.

	But if the caller is expecting data to come back, it can't just
	call Invoke() indefinitely without calling Dispatch(), as the
	other end's Dispatch() may be busy making the responding Invoke()s
	and not reading ours.

	So if the caller expects data to make a round trip, it calls
	InvokeDuplex(), which calls DispatchDuplex() to count the amount
	of data being sent (and thus expected to be received).  For every
	700 bytes so sent, DispatchDuplex() introduces a marker (a
	"flush1" message), which makes a round trip through the other
	and and returns as a "flush2" message.  If there are more than
	2000 bytes of data send out but not acknowledged by "flush2",
	DispatchDuplex() starts calling Dispatch() to look for the
	"flush2" message, processing any pending messages along the way.

	If the remote operation is expected to send a lot of data,
	like a file, we use InvokeDuplexRev().	This assumes that the
	reverse pipe is always full and so counts all data going forward,
	namely non-roundtrip data with Invoked().  It puts a marker in
	the "flush1" message so that it can track what data sent with
	InvokedDuplexRev() is outstanding.

	The caller is expected to call FlushDuplex() after the last call
	to InvokeDuplex() to ensure the pipe is clear.  Not doing so
	could cause a subsequent "release" message to get introduced into
	the conversation before it has fully quieted.

	In one case (in server/userrelease.cc) an operation dispatched
	from DispatchDuplex() may call InvokeDuplex() again, which could
	lead to a nesting of DispatchDuplex() calls.  To avoid this, the
	inDuplex flag gates entry into DispatchDuplex() to ensure there is
	only one active at a time.  XXX ughly.

    Lo Mark/Hi Mark Calculation and Accuracy

	The 700 byte threshhold before sending "flush1" is called the
	"lomark".  The 2000 byte threshhold before dispatching is called
	the "himark".  These historical values (particularly the himark)
	perform poorly over long latency, high bandwidth connections,
	just as small TCP windows do.  For this reason, the himark is
	recalculated at connection startup time, taking into account the
	TCP send and receive buffer sizes of the client and server.

	A himark that is too high can lead to a write/write deadlock,
	due to both client and server send and receive buffers being
	full, but slight miscalculation in the himark can be hard to
	detect because there are a number of places data can hide:

	1. It's always either the client->server side of the connection
	   (for submit), or the server->client side (for sync, etc) that
	   is congested, not both.  On the uncongested side, oustanding
	   duplex messages may still be in transit, except during large
	   file transfers (where file content fills the pipe).

	2. One duplex message can be in the RpcSendBuffer of the client.

	3. After the first server Dispatch(), the NetBuffer (4k) on the
	   server can be partially filled with duplex messages from the
	   client.

	4. We 'guess' a size of the flush1 message, because we have
	   to include its size in the fseq value in the message itself,
	   and we deliberately overestimate flush1 to be 60 bytes instead
	   of the more likely 45-50.  That gives us a little slop for
	   duplex messages.

    Flow Control Scenarios

	1. "primal": client sends initial command.

            Client                              Server
            ---------------                     -------------
            |             |                     |           |
            | Invoke      |  ====== >>> ======  | Dispatch  |
            |             |                     |           |
            ---------------                     -------------

	2. "callback": server instructing client.  Continues until
	   server sends 'release'.  Output-only commands all work
	   this way.

            Client                              Server
            ---------------                     -------------
            |             |                     |           |
            |             |                     | Dispatch  |
            | Dispatch    |  ====== <<< ======  | Invoke    |
            |             |                     |           |
            ---------------                     -------------

	3. "loop": server instructs client to send another message
	   to continue flow of control.  Spec editing commands work
	   this way: when the user editing is done, the client does
	   another Invoke().

            Client                              Server
            ---------------                     -------------
            |             |                     |           |
            | Dispatch    |                     |           |
            | Invoke      |  ====== >>> ======  | Dispatch  |
            |             |                     |           |
            ---------------                     -------------

	4. "duplex": server instructs client to send acks of operations.
	   InvokeDuplex() counts the amount of data in repsonses expected
	   from the client and periodically calls DispatchDuplex()
	   to handle responses.  Client file update commands use this to
	   send transfer acks to the server.

            Client                              Server
            ---------------                     ------------------
            |             |                     |                |
            |             |                     | Dispatch       |
            | Dispatch    |  ====== <<< ======  | InvokeDuplex   |
            | Invoke      |  ====== >>> ======  | DispatchDuplex |
            |             |                     |                |
            ---------------                     ------------------

	5. "duplexrev": like duplex, but InvokeDuplexRev() assume channel
	   from client to always full, and so counts the amount of data
	   sent from the server to the client.  Commands that send file
	   data to the server use this.

            Client                              Server
            ---------------                     ------------------
            |             |                     |                |
            |             |                     | Dispatch       |
            | Dispatch    |  ====== <<< ======  | InvokeDuplexRev|
            | Invoke      |  ==== >>>>>>> ====  | DispatchDuplex |
            |             |                     |                |
            ---------------                     ------------------

	6. "nested": happens with duplex when the response to the server
	   causes yet another Invoke() to be called.  Only 'revert -a' needs
	   this.

            Client                              Server
            ---------------                     ------------------
            |             |                     |                |
            |             |                     | Dispatch       |
            | Dispatch    |                     | InvokeDuplex   |
            |             |  ====== >>> ======  | DispatchDuplex |
            |             |  ====== <<< ======  | InvokeDuplex   |
            |             |                     |                |
            ---------------                     ------------------