The Perforce Client/Service Protocol
Message Passing
Though called RPC, it is in fact a message passing mechanism.
The difference being that in a message passing scheme the response
is both asynchronous and optional, and the driver of the connection
is a matter of agreement rather than predestiny.
Each message is a list of variable/value pairs, with one variable
('func') whose value is associated with some function call.
Rpc::Invoke() sends a single message and Rpc::Dispatch() loops,
reading messages and executing the associated function calls,
until a special 'release' message is received.
Basic Flow
The client sends a message containing the user's request to the
server and then dispatches, i.e. reads and executes requests from
the server until the server sends the "release" message.
The server initially dispatches, i.e. reads and executes a
request from the user. The final act of the request should
either be to release the client (send the "release" message),
or send a message to the client instructing it to invoke
(eventually) another server request. In the end, the final
message sent to the client should be "release".
Notes on flow control.
This collection of functions orchestrates the flow control when
data must move in both directions between the client and server
at the same time. This is done without async reads or writes.
Instead we rely on cues from our caller for the amount of data
expected to make a round trip, and periodically stop writing
and read for a while.
Normally, the caller uses Invoke() to launch functions to the
other side, and then Dispatch() (which calls DispatchOne()) to
listen for responses. Dispatch() returns when the other side
launches a "release" or "release2" in our direction.
But if the caller is expecting data to come back, it can't just
call Invoke() indefinitely without calling Dispatch(), as the
other end's Dispatch() may be busy making the responding Invoke()s
and not reading ours.
So if the caller expects data to make a round trip, it calls
InvokeDuplex(), which calls DispatchDuplex() to count the amount
of data being sent (and thus expected to be received). For every
700 bytes so sent, DispatchDuplex() introduces a marker (a
"flush1" message), which makes a round trip through the other
and and returns as a "flush2" message. If there are more than
2000 bytes of data send out but not acknowledged by "flush2",
DispatchDuplex() starts calling Dispatch() to look for the
"flush2" message, processing any pending messages along the way.
If the remote operation is expected to send a lot of data,
like a file, we use InvokeDuplexRev(). This assumes that the
reverse pipe is always full and so counts all data going forward,
namely non-roundtrip data with Invoked(). It puts a marker in
the "flush1" message so that it can track what data sent with
InvokedDuplexRev() is outstanding.
The caller is expected to call FlushDuplex() after the last call
to InvokeDuplex() to ensure the pipe is clear. Not doing so
could cause a subsequent "release" message to get introduced into
the conversation before it has fully quieted.
In one case (in server/userrelease.cc) an operation dispatched
from DispatchDuplex() may call InvokeDuplex() again, which could
lead to a nesting of DispatchDuplex() calls. To avoid this, the
inDuplex flag gates entry into DispatchDuplex() to ensure there is
only one active at a time. XXX ughly.
Lo Mark/Hi Mark Calculation and Accuracy
The 700 byte threshhold before sending "flush1" is called the
"lomark". The 2000 byte threshhold before dispatching is called
the "himark". These historical values (particularly the himark)
perform poorly over long latency, high bandwidth connections,
just as small TCP windows do. For this reason, the himark is
recalculated at connection startup time, taking into account the
TCP send and receive buffer sizes of the client and server.
A himark that is too high can lead to a write/write deadlock,
due to both client and server send and receive buffers being
full, but slight miscalculation in the himark can be hard to
detect because there are a number of places data can hide:
1. It's always either the client->server side of the connection
(for submit), or the server->client side (for sync, etc) that
is congested, not both. On the uncongested side, oustanding
duplex messages may still be in transit, except during large
file transfers (where file content fills the pipe).
2. One duplex message can be in the RpcSendBuffer of the client.
3. After the first server Dispatch(), the NetBuffer (4k) on the
server can be partially filled with duplex messages from the
client.
4. We 'guess' a size of the flush1 message, because we have
to include its size in the fseq value in the message itself,
and we deliberately overestimate flush1 to be 60 bytes instead
of the more likely 45-50. That gives us a little slop for
duplex messages.
Flow Control Scenarios
1. "primal": client sends initial command.
Client Server
--------------- -------------
| | | |
| Invoke | ====== >>> ====== | Dispatch |
| | | |
--------------- -------------
2. "callback": server instructing client. Continues until
server sends 'release'. Output-only commands all work
this way.
Client Server
--------------- -------------
| | | |
| | | Dispatch |
| Dispatch | ====== <<< ====== | Invoke |
| | | |
--------------- -------------
3. "loop": server instructs client to send another message
to continue flow of control. Spec editing commands work
this way: when the user editing is done, the client does
another Invoke().
Client Server
--------------- -------------
| | | |
| Dispatch | | |
| Invoke | ====== >>> ====== | Dispatch |
| | | |
--------------- -------------
4. "duplex": server instructs client to send acks of operations.
InvokeDuplex() counts the amount of data in repsonses expected
from the client and periodically calls DispatchDuplex()
to handle responses. Client file update commands use this to
send transfer acks to the server.
Client Server
--------------- ------------------
| | | |
| | | Dispatch |
| Dispatch | ====== <<< ====== | InvokeDuplex |
| Invoke | ====== >>> ====== | DispatchDuplex |
| | | |
--------------- ------------------
5. "duplexrev": like duplex, but InvokeDuplexRev() assume channel
from client to always full, and so counts the amount of data
sent from the server to the client. Commands that send file
data to the server use this.
Client Server
--------------- ------------------
| | | |
| | | Dispatch |
| Dispatch | ====== <<< ====== | InvokeDuplexRev|
| Invoke | ==== >>>>>>> ==== | DispatchDuplex |
| | | |
--------------- ------------------
6. "nested": happens with duplex when the response to the server
causes yet another Invoke() to be called. Only 'revert -a' needs
this.
Client Server
--------------- ------------------
| | | |
| | | Dispatch |
| Dispatch | | InvokeDuplex |
| | ====== >>> ====== | DispatchDuplex |
| | ====== <<< ====== | InvokeDuplex |
| | | |
--------------- ------------------