CEP-22 Oversized Payload Transfer

Oversized Payload Transfer

Status: Draft Author: @contextvm-org Type: Standards Track

Abstract

This CEP defines a bounded reassembly profile for ContextVM messages that are too large to publish as a single relay event. It reuses MCP notifications/progress as the framing envelope and uses the request progressToken as the transfer identifier.

The receiver reconstructs the exact serialized JSON-RPC message, validates a SHA-256 digest announced in start, and only then materializes the synthetic final request or response.

This CEP also defines an optional accept step for stateless client-to-server bootstrap.

Specification

Overview

ContextVM currently carries MCP JSON-RPC payloads inside Nostr events. This works well for ordinary request and response sizes, but large payloads may exceed relay event size limits and fail to publish even when the logical MCP operation completed correctly.

This CEP defines a bounded oversized-transfer profile that:

• reuses the existing single-kind ContextVM transport model
• reuses MCP notifications/progress as the transfer envelope
• uses the request progressToken as the transfer identifier
• supports ordered start, accept, chunk, end, and abort frames
• preserves normal MCP semantics through rendered reconstruction after bounded reassembly completes

This CEP is focused on bounded transfer only. It does not define open-ended streams, selective retransmission, or chunk repair procedures.

Capability Advertisement and Negotiation

Support MAY be advertised through the same additive discovery surfaces used by ContextVM features, following src/content/docs/spec/ceps/cep-6.md and src/content/docs/spec/ceps/cep-19.md.

Peers MAY advertise support using support_oversized_transfer tags.

Example:

[["support_oversized_transfer"]]

Advertisement surfaces:

• public announcements
• MCP initialization
• first exchanged request or response in stateless operation

support_oversized_transfer indicates support for this profile. Implementations that advertise it MUST support completionMode: "render".

Request-Level Activation

Oversized transfer is available only when the initiating request includes a valid MCP progressToken.

Rules:

• Clients that want to permit oversized transfer for a request MUST include a progressToken.
• Servers MUST NOT start an oversized transfer for a request that did not include a progressToken.
• When no progressToken is present, peers MUST use ordinary non-fragmented behavior or fail cleanly.

The progressToken identifies the transfer session.

Sender Behavior

For a logical message associated with a request carrying a progressToken:

• the sender SHOULD proactively use oversized transfer when it can predict that direct publication is likely to exceed relay limits
• the sender MAY reactively switch to oversized transfer when direct publication fails with a size-indicative error
• the sender MUST NOT assume that an ambiguous publish failure is caused by payload size

When oversized transfer is used, the sender MUST emit an ordered sequence of MCP notifications/progress messages carrying ContextVM transfer frames.

If the sender already knows the receiver supports this CEP for the exchange, it MAY proceed directly from start to chunk. Otherwise it MUST wait for accept before sending chunk frames.

Progress Notification Framing

Oversized-transfer frames are carried inside MCP notifications/progress params.

Example:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-123",
    "progress": 1,
    "message": "starting oversized transfer",
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "start",
      "completionMode": "render"
    }
  }
}

progress values MUST increase monotonically across the transfer.

Frame Types

This CEP defines five frame types:

• start
• accept
• chunk
• end
• abort

Common Fields

All oversized-transfer frames MUST include a ContextVM-specific transport object with:

• type: MUST be oversized-transfer
• frameType: one of start, accept, chunk, end, abort

The outer MCP progress params MUST include:

• progressToken
• progress

The outer MCP total and message fields MAY be used for progress reporting and UX hints, but they do not define transfer correctness.

start Frame

The start frame begins the transfer.

Required fields:

• completionMode: render
• digest
• totalBytes
• totalChunks

Rules:

• If completionMode is omitted, receivers MAY reject the transfer; senders SHOULD always provide it.
• In this CEP version, senders MUST use completionMode: "render".
• Receivers MUST reject unknown or unsupported completion modes.
• digest MUST be the SHA-256 of the exact serialized JSON-RPC message string, encoded as UTF-8.
• totalBytes MUST equal the exact byte length of the serialized JSON-RPC message string encoded as UTF-8.
• totalChunks MUST equal the number of chunk frames that the sender intends to transmit before end.
• Receivers MAY reject start immediately when totalBytes or totalChunks exceeds local policy limits.

chunk Frame

The chunk frame carries one ordered fragment.

Required fields:

• data: chunk payload

Rules:

• For oversized-transfer frames, MCP progress is the normative ordering field.
• Each chunk frame MUST use a progress value greater than the preceding transfer frame’s progress value.
• The payload represented by data is an ordered fragment of the exact serialized logical message associated with the transfer.
• Receivers MAY accept out-of-order arrival of valid chunk frames and buffer them in memory for later assembly, provided they still reconstruct the payload strictly by increasing progress order.
• Receivers SHOULD enforce bounded buffering limits for out-of-order chunks and MAY abort when those limits are exceeded.
• Receivers MAY track gaps between observed progress values while awaiting delayed chunks, but MUST NOT treat a gap alone as terminal failure before end or local timeout/policy conditions are reached.

accept Frame

The accept frame confirms that the sender may begin transmitting chunk frames.

Rules:

• A receiver MAY send accept after start.
• A sender that is required to wait for confirmation MUST NOT send chunk frames before receiving accept.
• accept does not change completion semantics.
• accept SHOULD remain minimal and does not negotiate additional transfer parameters in v1.

end Frame

The end frame signals successful sender-side completion.

Rules:

• end is required for successful completion.

abort Frame

The abort frame signals that the transfer did not complete successfully.

Optional fields:

• reason

Rules:

• Receivers MUST treat abort as terminal for the transfer.
• reason is advisory only.

Completion Mode

This CEP defines one completion mode: render.

In render mode:

• the chunk sequence represents one bounded logical JSON-RPC message
• the receiver MUST reassemble chunks in progress order
• the receiver MAY temporarily buffer valid out-of-order chunk frames before assembly, subject to local bounded-memory policy
• the receiver MUST NOT surface partial payloads upward
• the receiver MUST materialize a synthetic final request or response only after validation succeeds

The completionMode field remains part of the start frame as an extension surface for future CEPs, but this CEP defines only render.

Validation Rules

Ordering and Completeness

Receivers MUST validate transfer ordering using MCP progress.

Rules:

• a transfer MUST begin with start
• if confirmation is required for the transfer, accept MUST be received before the first chunk
• progress values for oversized-transfer frames MUST increase monotonically across the transfer
• receivers MUST treat progress as the canonical assembly index, not as a guarantee of arrival order from relays
• receivers MAY buffer valid out-of-order chunk frames within bounded local limits and later assemble them by progress order
• receivers MAY track missing progress positions as provisional gaps while the transfer remains in flight
• receivers MUST fail the transfer if the received chunk set cannot satisfy the declared totalChunks and totalBytes from start
• successful completion requires end
• if end arrives after malformed or non-monotonic transfer ordering, the transfer MUST fail
• if end arrives while provisional gaps remain unresolved, the transfer MUST fail

This CEP does not define selective retransmission or repair.

Digest Validation for render

Integrity is validated using SHA-256.

Rules:

• the sender MUST compute the digest over the exact serialized JSON-RPC message string
• the sender MUST encode that string as UTF-8 before hashing
• the start frame MUST carry the digest value
• the start frame MUST carry totalBytes and totalChunks
• the receiver MUST reconstruct the exact serialized string in progress order
• the receiver MUST verify that the reconstructed UTF-8 byte length equals totalBytes
• the receiver MUST verify that exactly totalChunks chunk frames were assembled before accepting end
• the receiver MUST compute SHA-256 over the reconstructed UTF-8 byte sequence
• the receiver MUST compare the result to the advertised digest before materializing the synthetic message

The digest is over the serialized message string, not an abstract JSON object.

The same digest rule applies symmetrically to:

• oversized client-to-server requests
• oversized server-to-client responses

Receiver Behavior

Receivers that support this CEP:

• MUST track transfer state by progressToken
• MUST process frames in bounded transfer order
• MUST reject or fail malformed frame sequences
• MUST treat abort as terminal
• MUST evaluate declared totalBytes and totalChunks against local limits before committing unbounded reassembly state
• MUST fail a transfer if end is received before a valid monotonic progress sequence has been observed
• SHOULD enforce a bounded out-of-order window or equivalent memory policy for unresolved chunk gaps
• MUST fail a transfer when local timeout or bounded-buffer policy makes successful completion no longer possible

Receivers MUST only surface the synthetic final request or response after digest validation succeeds.

Stateless Operation

This CEP is compatible with stateless ContextVM operation:

• peers MAY advertise support in tags on the first exchanged request or response
• transfer state is correlated by progressToken
• receivers MUST NOT rely on a persistent connection-local session beyond temporary bounded reassembly state

For stateless oversized client-to-server transfer where the client has not previously learned server support, the client MUST send start first and wait for accept before sending chunk frames.

Timeout Semantics

Because frames are carried by MCP notifications/progress, receipt of valid transfer frames MAY be treated as progress activity for request timeout handling.

Implementations:

• MAY reset soft request timeouts upon receiving valid transfer frames
• SHOULD still enforce a hard maximum timeout for the underlying request

Relay Size Guidance

Practical relay acceptance thresholds are often near a total serialized event size of approximately 64 KiB.

Implementations:

• SHOULD treat relay size limits as applying to the full serialized Nostr event, not only to the JSON-RPC payload or event content
• SHOULD use conservative margin below common practical thresholds when deciding whether to fragment proactively
• SHOULD prefer proactive oversized transfer when they can predict that direct publication is likely to exceed common relay acceptance limits
• MUST NOT assume that all relays enforce the same threshold or rejection behavior

Best Practices and Risks

Implementations should keep oversized transfer defensively bounded.

• Receivers SHOULD enforce strict limits on concurrent reassembly state, buffered bytes, and unresolved out-of-order chunks to reduce OOM risk.
• Receivers SHOULD bound the out-of-order buffer window and fail transfers that cannot plausibly complete within local resource policy.
• Senders SHOULD choose chunk sizes conservatively so each resulting Nostr event remains below practical relay acceptance limits.
• Receivers SHOULD treat declared totalBytes and totalChunks as admission-control inputs before committing memory.
• Implementations SHOULD use hard timeouts so incomplete transfers do not retain memory indefinitely.
• Implementations MUST assume relay delivery may be delayed, duplicated, or reordered, and MUST NOT infer successful completion until end, completeness checks, and digest validation all succeed.

Example: render Response Transfer

Client request:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "large_tool",
    "arguments": {},
    "_meta": {
      "progressToken": "req-123"
    }
  }
}

Server start:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-123",
    "progress": 1,
    "message": "starting oversized response",
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "start",
      "completionMode": "render",
      "digest": "sha256:8d969eef6ecad3c29a3a629280e686cff8fabcd1...",
      "totalBytes": 49,
      "totalChunks": 2
    }
  }
}

Server chunk frames:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-123",
    "progress": 2,
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "chunk",
      "data": "{\"jsonrpc\":\"2.0\",\"id\":1,"
    }
  }
}

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-123",
    "progress": 3,
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "chunk",
      "data": "\"result\":{\"content\":[...]}}"
    }
  }
}

Server end:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-123",
    "progress": 4,
    "message": "oversized response complete",
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "end"
    }
  }
}

The client reconstructs the exact serialized JSON-RPC response string, verifies the digest from start, and materializes a synthetic final response.

Example: Stateless Client-to-Server Bootstrap

Client start:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-789",
    "progress": 1,
    "message": "starting oversized request",
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "start",
      "completionMode": "render",
      "digest": "sha256:8d969eef6ecad3c29a3a629280e686cff8fabcd1...",
      "totalBytes": 10485760,
      "totalChunks": 160
    }
  }
}

Server accept:

{
  "jsonrpc": "2.0",
  "method": "notifications/progress",
  "params": {
    "progressToken": "req-789",
    "progress": 2,
    "message": "oversized request accepted",
    "cvm": {
      "type": "oversized-transfer",
      "frameType": "accept"
    }
  }
}

After accept, the client sends chunk frames and finishes with end.

Backward Compatibility

This CEP introduces no breaking changes:

• peers that do not advertise support continue using ordinary ContextVM request and response transport
• peers that do not include a progressToken on a request do not enable oversized transfer for that exchange
• future completion modes remain possible through completionMode, but this CEP defines only render

Implementations that ignore the new tags or do not understand the oversized-transfer framing continue to interoperate for ordinary non-fragmented messages.

Dependencies

• CEP-6: Public Server Announcements

Reference Implementation

A reference implementation is intended for the ContextVM SDK transport layer.