DiskpipeSTORAGE / OPERATIONS

Zero-Copy Descriptor Model

zcutils treats a zero-copy record as metadata plus one or more leased byte ranges. The hot path should reorder, split, join, and fan out descriptors without normalizing payloads into new contiguous buffers.

The first command implementation still includes byte-stream compatibility tools such as zccat, zcgrep, zctee, zcsink, and zcstat. Those are useful for debugging and smoke tests, but a real cross-process zero-copy path needs the descriptor protocol below over Unix sockets, shared memory, and fd passing.

Every descriptor stream starts with a fixed envelope header:

#[repr(C)]
pub struct ZcStreamHeader {
    pub magic: [u8; 8],       // b"ZCSTRM\0\1"
    pub protocol_version: u16,
    pub min_reader_version: u16,
    pub flags: u32,
    pub header_len: u32,
}

Version rules:

Descriptor Streams

A ZcDescriptorStream is the Unix-like primitive. It is an ordered control stream that carries pool attachments, credits, descriptor records, releases, checkpoints, and termination frames. Payload bytes stay in owned pools such as ZCRX areas, registered buffers, mapped files, shared memory, or WAL regions.

A normal shell pipe can carry byte-compatible fallback data, but it is not enough for true zero-copy because it cannot pass pool fds or enforce release accounting by itself. Descriptor-native pipelines can be started by zcflow or an equivalent supervisor that wires Unix sockets, fd passing, credits, and release channels before producers begin emitting descriptors.

zcflow should resolve stages through the normal PATH. A descriptor-native utility does not need to be compiled into this repository; it only needs to speak the descriptor protocol on the inherited control fd or deliberately opt into byte-compatible stdin/stdout mode. The expected launch contract is an environment such as ZC_DESCRIPTOR_FD, ZC_PROTOCOL_VERSION, ZC_STAGE_INDEX, and ZC_STAGE_COUNT, plus normal argv.

Normal shell composition should also work without an explicit manager command:

zcdemux ... | zcmap --preserve-lanes | zcmux --peer-addr C ...

In that form, the upstream producer creates or joins a session and writes the session identity into the descriptor stream header. Each downstream zc* command reads the header, connects to the same manager/session, and propagates a fresh header for the next stage. Explicit zcflow is still useful when the user wants one process to supervise the whole graph directly.

Required stream frame families:

Collections and Lists

A ZcDescriptorCollection is a bounded group of descriptors with shared lifetime, ordering, topology, and failure semantics. It is useful for batching a Raft append slice, WAL commit group, scatter/gather file extent, or fanout unit. Collections are stream frames, not separate ownership domains: every descriptor inside still releases back to the original pool authority.

A ZcDescriptorList is the concrete serialized form of a collection. It is for manifests, tests, checkpoints, and compatibility files. It can be replayed into a descriptor stream, but replay must either attach the original pools safely or materialize/copy bytes into a new owned pool. A stale list must not grant access to recycled buffer generations.

Snapshot Cuts

A point-in-time snapshot should be a descriptor/WAL metadata cut, not a block device operation. The minimal descriptor-native shape is:

The byte-compatible zcsnap command writes a zcsnap-manifest-v1 cut manifest for smoke tests and pipeline planning. It deliberately does not freeze block devices, clone volumes, manage RAID membership, or add snapshot behavior to zcbrd, zcnblk, or zcraid-*.

Soundness Contract

Every descriptor command must satisfy one of these roles:

No command should emit descriptors into an unbounded or unacknowledged stream. If a downstream branch is not listening, disconnects, or exits, the upstream owner must receive release or abort all leases held for that branch.

Slice Descriptor

#[repr(C)]
pub struct ZcSliceDesc {
    pub pool_id: u32,
    pub queue_id: u32,
    pub buffer_id: u64,
    pub generation: u32,
    pub offset: u32,
    pub len: u32,
    pub flags: u32,
    pub numa_node: i16,
    pub preferred_cpu: i16,
}

Fields:

Record Descriptor

#[repr(C)]
pub struct ZcRecordDesc {
    pub desc_version: u16,
    pub desc_len: u16,
    pub record_id: u64,
    pub stream_id: u64,
    pub group_id: u64,
    pub sequence: u64,
    pub lane_id: u32,
    pub preferred_worker: u32,
    pub total_len: u32,
    pub slice_count: u16,
    pub flags: u16,
    pub release_token: u64,
}

The record header is followed by slice_count ZcSliceDesc entries. Consumers seek by walking the scatter/gather list and translating a logical record offset to a (buffer_id, offset, len) view. Reordering should hold descriptors, not copy bytes.

desc_len lets newer writers append fields while older readers skip the tail when no required feature bit is set. desc_version names the schema used by the fixed prefix.

Topology Hints

Plan Control Frames

Topology-aware descriptors use a control-plane handshake before data frames start moving. The control frames are not an optimizer side channel; they are the descriptor contract that tells each stage how to preserve zero-copy lifetimes, ordering, lane locality, and backpressure.

ZC_CAPS_V1  -> every participant advertises CPUs, NUMA, NIC queues, memory
               pools, ZCRX/send-zc/liburing/libfabric support, and limits
ZC_INTENT_V1 -> the workload declares mirror/stripe/raid10/spill/passthrough,
                read/write mix, objective, durability, sync, and zero-copy policy
ZC_PLAN_V1  -> the planner compiles lane -> worker -> CPU -> queue -> WAL shard
               and lane -> branch/replica/stripe mappings for one epoch
ZC_STATS_V1 -> stages report copies, context switches, migrations, branch lag,
               pool pressure, and queue/ring overflows
ZC_REPLAN_V1 -> a new epoch with explicit lane/worker/queue/branch remaps

The hot data plane should not re-run placement policy. It validates plan_id, placement_epoch, lane_id, and sequence, then executes the compiled map. A descriptor-native data frame therefore carries a compact plan reference, not the full plan:

plan_id
placement_epoch
lane_id
sequence
descriptor lease(s)

ZC_PLAN_V1 projects onto existing descriptor fields:

Plan changes happen only through ZC_REPLAN_V1 at a placement epoch boundary. Silent remapping is forbidden because it breaks release accounting, same-sector ordering, result-log zipping, sync high-water marks, and benchmark attribution.

The zcplan command is the current executable prototype. zcplan caps emits local ZC_CAPS_V1 JSON, and zcplan plan --caps A.json,B.json emits a ZC_PLAN_V1 JSON document plus a descriptor_projection section showing how the compiled topology maps onto descriptor, tcpmux, WAL, and zcnblk fields.

Encryption Hint

Transport descriptors should distinguish encrypted and plaintext payloads. aes-256 means AES-256-GCM framed chunks and is the default encrypted zc stream format produced by zcencrypt and tcpmux. none is an explicit plaintext hint and should only be produced when the user passed --encryption none.

Descriptors should carry locality hints so downstream commands can keep work near the source queue and memory when possible:

These fields are hints, not correctness requirements. A consumer may ignore them if its topology is different, if the CPU is unavailable, or if applying the hint would break ordering/backpressure. When preserved, they let a pipeline avoid cross-NUMA copies, L3 cache disruption, queue bouncing, and avoidable worker migrations.

Lane Alignment Contract

lane_id is the primary descriptor field for keeping software lanes aligned across network, worker, WAL, and storage stages. A stage that receives a record with a known lane_id should preserve that value unless it intentionally remaps the record.

For a WAL or block-target pipeline, the preferred mapping is:

lane_id -> transport lane -> RX queue -> worker -> WAL shard -> byte range -> ack lane

The same lane_id should therefore select the same worker/shard class on each stage, and the descriptor should carry enough local placement hints for the next stage to avoid guessing:

If a stage changes lane count, rebalances a hot lane, or stripes one input lane over several output shards, it should emit an explicit remap descriptor or append an extension that records both the original lane and the new local placement. Silent remapping is forbidden for descriptor-native hot paths because it breaks per-lane ordering, completion accounting, and profiling.

Coalescing should preserve lane identity. A WAL coalescer may join many small records from the same lane into a larger append extent, but the resulting descriptor should still report the source lane, output WAL shard, byte range, and sequence range covered by the extent. Cross-lane coalescing requires an explicit barrier or sequence map so consumers do not infer false per-lane order. See docs/wal-extent-framing.md for the proposed fixed 4K logical record extent format.

The zcnblk fan WAL prototype follows that rule for batched 4K block writes: WRITE_BATCH carries one descriptor array followed by one coalesced payload area, and RESULT_BATCH carries one result descriptor array. The fan interleaves leaf result streams in the lane-local handler aligned to the upstream output lane; leaf socket return order is never the placement or freshness contract.

TCP Mux Topology Header

zc-tcpmux parallel lanes now use a versioned V2 lane header. It carries the same placement intent as descriptor records so a receiver can keep each lane on the matching local worker:

The header is advisory. zc-tcpmux-receive logs both the sender hint and the receiver's actual pinned CPU/NUMA placement. Use --pin-cpus --cpu-list LIST on zc-tcpmux-xfer to apply the same lane-to-CPU map to both the local send workers and the remote receive workers. --send-cpu-list and --receive-cpu-list can diverge the two sides when the machines have different topologies.

Zcnblk Topology Header

zcnblk request and response headers are v2 fixed 64-byte frames. The first 32 bytes keep the original block contract: op, flags, target shard, payload length, and offset. The second 32 bytes carry the placement contract:

The userspace target rejects topology-marked requests that arrive on the wrong lane or target a different tier than the encoded shard. That makes silent lane remapping visible before the tiering and RAID paths depend on it for ordering, backpressure, and locality.

Lifetime

zcutils owns the authoritative lease table. Consumers receive descriptors and must release by token.

Normal flow:

NIC/ZCRX produces buffer
  -> zcdemux creates a buffer lease
  -> zcflow or stream-header discovery has wired downstream credit/release channels
  -> zcdemux emits a ZcRecordDesc + sglist when credit is available
  -> zcmap/zctee/zcmaptee/zcmux/app/WAL pass or refcount references
  -> each async completion or app release decrements references
  -> refs hit zero
  -> buffer returns to the RX refill path or userspace pool

Required release events:

The app must never directly free RX memory. It sends release_token back to the owner, and the owner validates (pool_id, buffer_id, generation) before recycling.

Ordering

zcdemux should default to per-stream ordering, not global ordering.

global: unordered
peer pair: optional ordering
stream/group: ordered
fragment: ordered within record

The reorder window stores descriptor references. A late or duplicate record can be dropped by metadata without touching payload bytes.

Zero-Copy Defaults

The network-facing commands default toward zero-copy with portable fallback:

Byte-stream tools are intentionally marked as compatibility/debug paths until the descriptor socket protocol is implemented.

Built from Markdown with Rust. No client-side dependencies.