Wire Protocol

vgi-rpc implements the same wire protocol as the Python vgi-rpc framework, ensuring cross-language compatibility.

Overview

Communication uses multiple sequential Arrow IPC streams over stdin/stdout (subprocess mode) or HTTP request/response bodies (HTTP mode).

Request format

Each request is a single Arrow IPC stream containing one RecordBatch:

• The batch schema fields correspond to method parameters
• Batch metadata carries the routing key, the method name and the protocol version:
  • vgi_rpc.method — the method name to invoke
  • vgi_rpc.protocol — which of the server’s hosted protocols the method belongs to. Dispatch resolves the pair (protocol, method), so method names may collide across protocols. Required on stdio, unix and named pipes, where the metadata is the only carrier and there is no single-protocol exemption: an intermediary that rebuilds a request and drops the field must be told, not landed silently on whichever protocol happened to be first. Optional on HTTP, where the {prefix}/{protocol}/{method} path segment has already resolved the binding — but if present it must agree with the path, because otherwise edge policy is applied to one protocol while the worker dispatches another. Accepting absent on HTTP is what makes a path rewrite by an intermediary undetectable, which is a real and deliberate cost; see HTTP transport.
  • vgi_rpc.request_version — wire framing version (currently "1")
  • vgi_rpc.protocol_version — optional application protocol surface version (semver MAJOR.MINOR.PATCH)
  • vgi_rpc.request_id — optional request correlation ID

Response format

Unary responses

A single Arrow IPC stream containing one RecordBatch with the result fields.

Stream responses

Multiple batches within a single IPC stream:

1. Optional header batch (if headerSchema is defined)
2. One or more data batches from produce or exchange
3. Log batches interleaved between data batches

Log batches

Zero-row batches with metadata:

• vgi_rpc.log_level — log level string (e.g., "INFO", "DEBUG")
• vgi_rpc.log_message — the message text
• vgi_rpc.log_extra — JSON-encoded extra metadata (optional)

Error batches

Errors are reported as zero-row batches with EXCEPTION metadata:

• vgi_rpc.log_level set to "EXCEPTION"
• vgi_rpc.log_message containing the formatted error message ("<ExceptionType>: <message>")
• vgi_rpc.error_kind — optional top-level error category (only present for typed exceptions)
• vgi_rpc.log_extra — JSON blob carrying exception_type, exception_message, traceback, and (when present) error_kind

Over HTTP, error responses additionally set the X-VGI-RPC-Error response header.

vgi_rpc.error_kind values include:

• method_not_implemented
• session_lost
• server_draining
• protocol_version_mismatch

Streaming protocol

Producer streams

1. Client sends a single request batch
2. Server responds with a stream of output batches
3. Server signals completion by ending the IPC stream

Exchange streams

Exchange streams use a lockstep protocol:

1. Client sends one input batch
2. Server reads it and writes one output batch
3. Repeat until the client stops sending

This interleaved read/write pattern prevents deadlocks when both sides buffer data on stdin/stdout pipes.

Cancellation

A client cancels an in-flight stream by writing an input batch whose metadata carries vgi_rpc.cancel (the CANCEL_KEY). On receipt, the server ends the stream cleanly, running any server-side onCancel hook exactly once.

External location pointers

Large batches may be replaced on the wire by a zero-row “pointer” batch that carries:

• vgi_rpc.location — an out-of-band location the peer resolves to fetch the real payload
• vgi_rpc.location.sha256 — the SHA-256 digest of the referenced payload for integrity verification

A unary handler that returns a pre-published ExternalRef produces exactly this pointer (its URL, plus vgi_rpc.location.sha256 only when the ref has a digest) without serializing or uploading anything during the call; the wire format is unchanged, so clients resolve it like any other pointer. The digest is optional on the wire: when the key is absent the client skips the content check.

Protocol version enforcement

Requests may carry vgi_rpc.protocol_version, the application protocol surface version in canonical semver (MAJOR.MINOR.PATCH). At the dispatch boundary the server enforces an exact major + minor match (the patch component is ignored). A mismatch raises a protocol-version error with error_kind set to protocol_version_mismatch. This is distinct from vgi_rpc.request_version, which versions the wire framing itself.

Metadata keys

Key	Value	Description
vgi_rpc.method	Method name	Identifies the RPC method
vgi_rpc.request_version	"1"	Wire framing version
vgi_rpc.protocol_version	Semver string	Application protocol surface version
vgi_rpc.request_id	UUID string	Request correlation ID
vgi_rpc.server_id	ID string	Server identifier
vgi_rpc.protocol_name	Name string	Protocol name (in describe)
vgi_rpc.describe_version	"4"	Describe protocol version
vgi_rpc.protocol_hash	Hex digest	SHA-256 hash of the describe payload
vgi_rpc.log_level	Level string	Log/error level ("EXCEPTION" for errors)
vgi_rpc.log_message	Message string	Log/error message
vgi_rpc.log_extra	JSON string	Additional metadata
vgi_rpc.error_kind	Kind string	Error category (typed exceptions)
vgi_rpc.cancel	Marker	Client-initiated stream cancellation
vgi_rpc.location	Location string	External payload pointer
vgi_rpc.location.sha256	Hex digest	SHA-256 of the referenced payload
vgi_rpc.stream_state#b64	Token string	HTTP streaming state token

Introspection

Introspection is vgi_rpc.Reflection.v1, an ordinary co-hosted protocol reached the ordinary way — not a reserved method answered before dispatch. It is hosted by default (enableDescribe defaults to true on VgiRpcServer); opt out with enableDescribe: false.

__describe__, the reserved name it replaces, is retired. A server asked for it answers with a refusal naming the protocol and both of its entry points, rather than “no such method” — that answer cannot be told apart from “this server was built without introspection”, and the two need opposite fixes.

Two methods, in the order a client uses them:

Method	Parameters	Returns
list_protocols	none	Every protocol the server hosts, with versions and hashes
describe	protocol: utf8	One protocol’s full description, methods included

The first is the bootstrap: reflection’s own name is fixed, so it is the one name a client may know a priori, and everything else is discovered through it. A client that already knows which protocol it wants skips straight to the second.

Each reply rides as serialized Arrow IPC in a single result binary column — the framework’s ordinary convention for a structured return. Reflection is an ordinary protocol, so its replies are subject to the transport’s payload handling like any other method’s, externalization included.

One method’s description carries name, method_type, has_return, has_header, stream_kind, params_schema_ipc, result_schema_ipc, header_schema_ipc, idempotency, deprecated and deprecation_message. Schemas travel as Arrow IPC because a client’s whole purpose in asking is to get a schema it can hand to its own Arrow implementation.

Decoding is tolerant by contract: read by field name, ignore unknown columns, default absent columns that have a default — and raise for an absent one that does not, because zero-filling a required field hands a client a description that is wrong rather than absent. One rule follows and binds every port: a field added in a minor version must carry a default, or the addition is a breaking change wearing a minor version number.

protocol_hash is the canonical digest of the protocol’s description — SHA-256 over RFC 8785 canonical JSON of the decoded structure, not over Arrow IPC bytes. That is what makes it comparable across ports, where the bytes are not. It is also the digest every access record carries.

HTTP wire format

When using HTTP transport, the wire format wraps the same Arrow IPC streams in HTTP request/response bodies:

• Content-Type: application/vnd.apache.arrow.stream
• State tokens in vgi_rpc.stream_state#b64 batch metadata enable stateless streaming
• Tokens are sealed with XChaCha20-Poly1305 AEAD (wire format v4), base64-encoded, and time-limited. Tamper and forgery resistance comes from the Poly1305 authentication tag, and each token is bound to its authenticated principal via the AEAD associated data, so a token minted for one identity fails to open when presented by another.

See HTTP Transport for the HTTP-specific routing and state management details.