Proteus

Proteus is a compact, zero-allocation binary format and a Zig comptime codec generator. SPEC.md defines the wire contract. ROADMAP.md holds the project principles, the wire-format change process, the compatibility policy, and the complete development history; the Zig API does not define a separate encoding.

Toolchain

The project requires Zig 0.17.0 exactly. The version is pinned in:

• .zig-version for compatible version managers;
• build.zig.zon as package metadata;
• build.zig as an exact compile-time check;
• src/proteus.zig as the public required_zig_version, checked by a test.

Install Zig 0.17.0 from the official download page, then verify:

$ zig version
0.17.0

A Zig upgrade is a reviewed source change. Update all four pins together and run the complete test, fixture, fuzz, assembly, and packaging gates.

Commands

Run Zig tests:

zig build test

Build the type-generation baselines:

zig build type-generation-small -Doptimize=small
zig build type-generation-large -Doptimize=small

The measurement procedure and initial results are in benchmarks/README.md.

Run only the malformed-schema diagnostic fixtures:

zig build compile-errors

The codec implements serialization, borrowed views, explicit validation, allocator-owned decoding, and fixture conformance for primitives, arrays, structs, enums, sealed and unsealed tagged unions, and optionals. A custom schema returns the operation-specific Schema*Unavailable error unless its adapter provides that operation's callbacks.

Use schema.unitVariant for unit-like variants and schema.variant for payload variants. A payload of schema.Optional(T) uses null as the no-payload state, allowing a unit variant to evolve compatibly to that form. Validation recursively checks known payloads while leaving future payloads on known unit variants and unknown fallbacks uninterpreted. Successful validation enables the generated trusted-view APIs normally.

Serialize an owned value into the transactional message writer:

var writer = proteus.Writer.init(allocator);
defer writer.deinit();

const region_end = try Record.serialize(&value, &writer);
const bytes = writer.asSlice();
_ = region_end;
_ = bytes;

The value is borrowed, not copied. Serialization only ever reads it, and a by-value parameter whose address the callee takes is a copy of the whole value on any call the optimizer does not inline; for a root whose Owned is a large fixed array that copy costs more than the encoding does. This also matches the custom-adapter callbacks below, which have always taken *const Owned.

A failed serialization restores the writer length from before that value, including removal of child regions written before the failure. Existing bytes in the writer are retained.

A writer is also reusable across messages. clearRetainingCapacity keeps the backing storage, so a serialization loop grows the buffer once rather than per message; on the region-heavy benchmark workloads reuse measured worth 15% to 35% over a fresh writer per message, and it composes with the reservation the serializer already performs.

Materialize native data with the same generated codec:

var owned = try Record.decodeOwned(bytes, allocator);
defer Record.deinitOwned(&owned, allocator);

const borrowed = try Record.view(bytes);
var copied = try Record.copyView(borrowed, allocator);
defer Record.deinitOwned(&copied, allocator);

decodeOwned parses root framing once and then materializes through the generated view. copyView skips that root parse. Both operations allocate every dynamic slice with the supplied allocator and clean up transactionally if a later read or allocation fails. They do not run canonical validation; call validate explicitly when that policy is required.

An Owned type is a native representation, not an ownership marker encoded in the Zig type system. Values returned by decodeOwned or copyView may always be passed to deinitOwned exactly once with the same allocator. For manually constructed values, callers retain responsibility for slice storage and must only call deinitOwned when all recursively dynamic slices were allocated by the provided allocator.

Each dynamic slice is a separate allocation, and deinitOwned frees each one separately. That is what lets it run over a value assembled by hand as well as one produced by decoding, and it means the cost of decoding a schema with many dynamic slices is set by the allocator rather than by the message. A 914-byte message holding 32 byte arrays takes 33 allocations and 33 frees, and spends about three quarters of its decode time in them.

The allocator is the caller's choice, and that is the lever. Decoding into an arena replaces every one of those frees with a single release:

var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();

const owned = try Record.decodeOwned(bytes, arena.allocator());
// No deinitOwned call: the arena owns every block the decode took.

On the message above this is 608 ns against 248 ns, a 59% saving; a decode loop can reset the arena between messages and keep the backing block. Values decoded this way may still be passed to deinitOwned with the arena's allocator if a caller prefers the uniform shape. Schemas whose owned form is one slice, such as a struct wrapping a single byte array, allocate once either way and have nothing to gain here.

Construct a borrowed view over a complete root encoding and access only the requested fields:

const view = try Record.view(bytes);
const id = try view.get(.id);
const data = try view.get(.data);
const first = try data.get(0);
_ = id;
_ = first;

Struct get return types are selected at comptime. Variable and fixed array views provide len() and checked get(index) methods. Arrays of u8 also provide asBytes() for direct slicing. Nested views retain the original input slice and allocate no storage.

Arrays of integers and floats provide asSlice(), which returns every element at once without copying:

const samples = try reading.get(.samples);
const values: []align(1) const i32 = samples.asSlice();

Constructing the array view proves its whole element body in bounds, and an integer's or float's encoding is its native little-endian representation, so the elements are already values and none of them can fail to read. That is why the accessor exists for those element types and not for others: booleans and enums have encodings that get rejects and a borrowed slice could not. Regions are unaligned by design, which is what align(1) records; the slice borrows the message, so it lives exactly as long as those bytes.

Reading elements one at a time is not the slow path it sounds like. Once an array view exists its element body is proven, and for these element types nothing is left to reject, so get performs a bounds check and a load — the same load asSlice hands out in bulk. getTrusted drops even the bounds check and returns the element with no error union at all, so it needs no try:

for (0..samples.len()) |index| consume(samples.getTrusted(index));

What asSlice still buys is the one fact a per-element loop cannot state. Widening a loop that stores elements into caller memory requires knowing that the destination does not overlap the message bytes, and only the calling code knows that. Copying 1024 u32 elements takes 54 ns element-wise where the caller's own signature rules out overlap and 268 ns where it does not, against 49 ns through asSlice, whose @memcpy carries the fact itself. Reductions have no store and so no such question. Owned decoding takes the asSlice path for every integer, float, and fixed numeric array it materializes.

asSlice is available when the element type is an integer or a float, and for elements wider than one byte when the target is little-endian; anywhere else it is a compile error. Code that must build for any target can ask first, since the answer is a comptime constant:

if (comptime @TypeOf(samples).supports_slice) {
    for (samples.asSlice()) |value| consume(value);
} else {
    for (0..samples.len()) |index| consume(try samples.get(index));
}

The companion constant supports_infallible_element reports the half of that gate that byte order does not affect, which is what decides whether getTrusted returns an error union. Fixed-array views, checked and trusted, expose the same get, getTrusted, asBytes, asSlice, and both constants, so code written against one array view type reads through the other unchanged.

view parses only the root framing needed to construct the view. Dynamic child footers and offsets are parsed when their accessors are called; recursive validation remains the caller's explicit responsibility:

try Record.validate(bytes, .basic);
try Record.validate(bytes, .canonical);

A view reports the errors it must resolve to answer at all — framing, offsets, bounds, and the value encodings it has to interpret, such as a boolean byte or an enum tag. Rules that a view could skip past without misreading anything are validation's, not the view's. A non-canonical NaN used as a float value is the clearest case: SPEC.md lists it among what a basic validation pass rejects, and validate rejects it, but a view hands the bit pattern back as the float it is. Do not treat a successful view as a statement that the message is valid; that is what validate is for.

Callers that validate once and repeatedly access the same immutable bytes can construct a generated TrustedView:

const trusted: Record.TrustedView = try Record.validatedView(
    bytes,
    .basic,
);
const id = trusted.get(.id);
const data = trusted.get(.data);
const first = data.getTrusted(0);

validatedView performs the complete validation pass and returns the proof- carrying view. Schema-derived field, child framing, offset, tag, and value reads are then infallible. A runtime array index is still checked by get; getTrusted is infallible and requires index < len().

When validation and view construction must be separate, call validate first and then trustedViewAssumeValidated. The latter does not validate: the caller must use the same schema and byte slice and must keep those bytes unchanged while the returned view is in use. Ordinary view remains independent of validation and is still the appropriate lazy checked API.

A caller that already holds a checked View can upgrade it in place rather than reparsing the root framing:

const checked = try Record.view(bytes);
try Record.validate(bytes, .basic);
const trusted: Record.TrustedView = Record.assumeValidated(checked);

assumeValidated carries the same contract as trustedViewAssumeValidated, and reparses nothing: both view families store identical framing, so the upgrade is a field-wise copy that optimizes away entirely. Prefer it when a lazy View already exists; prefer trustedViewAssumeValidated when starting from bytes, because it skips the checked footer parse that view performs.

Basic validation checks known structure and values. It also requires every known child region to end before the slot body that refers to it begins, and rejects offsets that name one region from several slots, so a validation pass is linear in the message length. Canonical validation also requires zero schema-owned unused tagged-union payload bytes and the smallest valid region word size. Older struct schemas validate their known field suffix and ignore appended fields whose widths and child schemas are unknown; the canonical word-size check is skipped for a region carrying such fields and applied to one an older writer produced with fewer fields than the reader knows.

The runtime, generated serializer, checked and trusted views, validation, and owned decoding all use exact unaligned region framing. Generated niche assignment and canonical word-size selection use the three reserved offset words. The conformance corpus covers real offsets 3, 4, and 5 and rejects unassigned reserved offsets 0, 1, and 2; deterministic mutation fuzzing is biased toward the same boundary values.

validateWithDiagnostic optionally reports an allocation-free path of field, variant, and array-index segments together with the underlying runtime diagnostic:

var diagnostic: ?Record.ValidationDiagnostic = null;
Record.validateWithDiagnostic(
    bytes,
    .canonical,
    &diagnostic,
) catch |err| {
    const path = diagnostic.?.path();
    _ = path;
    return err;
};

Enums, tagged unions, and optionals use schema-selected representations. Codec.niches exposes invalid slot patterns left after an optional or niche-backed union consumes its assigned patterns. Tagged-union View types are generated unions whose payload fields contain the corresponding borrowed view types.

Fixed and sealed custom adapters opt into validation with:

pub fn validateSlot(
    bytes: []const u8,
    containing_region_end: usize,
    word_size: proteus.WordSize,
    slot_start: usize,
    mode: proteus.ValidationMode,
) proteus.runtime.Error!void;

An adapter whose slot contains active region words or child offsets can also provide canonicalRequirements returning proteus.CanonicalRequirements.

Field paths

A field path is a sequence of struct field numbers and tagged-union variant numbers from the schema root; the schema decides at each depth which of the two a number is. Array, FixedArray, and Optional carry no number and are transparent, including after the last number, so a path names zero or more values in one message: one per element of every array it crosses, and none past a null optional, an inactive variant, or a struct field the encoding stopped before. A path's target is never a wrapper; it is a struct, a union, a primitive, a unit enum, a custom adapter, or void, and a custom adapter is opaque to further numbers.

A compile-time path is checked against the schema and iterated through the target's view type:

const Events = proteus.Codec(proteus.schema.Struct(.{
    .sealed = false,
    .fields = .{
        proteus.schema.field("seed", 0, proteus.schema.U64),
        proteus.schema.field("events", 1, proteus.schema.Array(EventSchema)),
    },
}));

const root = try Events.view(bytes);
var ticks = Events.pathIterator(root, &.{ 1, 1 });
while (try ticks.next()) |tick| {
    const index = ticks.coordinates()[0];
    _ = .{ tick, index };
}

trustedPathIterator walks a TrustedView and its next is infallible. A number the schema cannot follow is a compile error naming the depth.

A path stored outside the program, in a database say, arrives as numbers at run time. DynamicPath(target) resolves it once against the schema and applies it to any number of messages through one iterator type, whichever node of that type the path named:

const Ticks = Events.DynamicPath(proteus.schema.U64);
const stored = try Ticks.resolve(numbers_from_database, null);
var ticks = stored.iterate(try Events.view(bytes));
while (try ticks.next()) |tick| _ = tick;

Resolution follows the compatibility rules. A number beyond the fields this schema knows in an unsealed struct, or beyond the variants of an unsealed union, is where a newer schema may have appended something: the path resolves to unknown, known() is false, and the iterator yields nothing. A number outside a sealed struct or union is error.PathNumberOutOfRange, a number following a leaf is error.PathThroughLeaf, a path longer than proteus.path.max_len is error.PathTooLong, and a known node of another type is error.PathTargetMismatch. The optional diagnostic reports the depth, the number, and the reason. In the other direction, a path from an older schema always resolves in a newer one, and a field the encoding stopped before yields its default or nothing rather than the MissingRequiredField error get reports.

DynamicPath admits the nodes whose descriptor is the one given. When several descriptors view alike, such as custom adapters that mirror one definition in different roles, DynamicPathWhere takes a compile-time predicate on the target descriptor instead. Every admitted node must view as one type, and a predicate that admits nothing is a compile error:

const Mirrors = struct {
    fn accept(comptime target: anytype) bool {
        const T = @TypeOf(target);
        return T.schema_kind == .custom and
            @hasDecl(T.Adapter, "Definition") and
            T.Adapter.Definition == Definition;
    }
};
const Values = Stamped.DynamicPathWhere(Mirrors.accept);

resolvePath returns a PathNode, an enum with one member per node named by the path's canonical text — / for the root, /1/1 for variant 1 of field 1 — plus unknown. An inline else switch over it hands a run-time path to the compile-time API when the target type is not known in advance:

switch (try Events.resolvePath(numbers_from_database, null)) {
    .unknown => {},
    inline else => |node| {
        const numbers = Events.PathNodes.numbersOf(node);
        var values = Events.pathIterator(root, numbers);
        while (try values.next()) |_| {}
    },
}

Names are for source code and numbers are what messages and stored paths carry: pathOf(.{ "events", "tick" }) turns field and variant names into the numbers a compile-time path takes. proteus.path.format writes a path's canonical text and proteus.path.parse reads it back into a caller's buffer, for a database column that holds text rather than numbers.

Schemas

proteus.schema contains zero-sized comptime descriptors. Codec(schema) validates the complete finite tree and generates Owned, View, TrustedView, and struct Field types. It exposes the root layout and niche metadata while keeping its normalized analysis and resolver types private:

const RecordSchema = proteus.schema.Struct(.{
    .sealed = true,
    .fields = .{
        proteus.schema.field("id", 0, proteus.schema.U64),
        proteus.schema.fieldDefault(
            "attempts",
            1,
            proteus.schema.U16,
            @as(u16, 0),
        ),
        proteus.schema.field(
            "payload",
            2,
            proteus.schema.Optional(proteus.schema.Bytes),
        ),
    },
});

const Record = proteus.Codec(RecordSchema);

comptime {
    std.debug.assert(Record.layout == .sealed);
    std.debug.assert(Record.niches.isEmpty());
}

Struct field declaration order is irrelevant. Codec normalizes fields by their explicit field numbers for generated native types, metadata, child serialization, slot layout, viewing, validation, and owned decoding. Numbers must still be unique and contiguous from zero. Field and variant numbers are proteus.schema.Number, a u32, everywhere the library exposes them.

Array and fixed-array element types must occupy at least one byte, and a zero-width element type — an empty sealed struct, a FixedArray(0, T), or a custom adapter whose slotWidth returns zero — is a compile error there. An array's element count is bounded by the message only through count * slot_width(element), so a zero-width element would let a few bytes claim an element count no reader loop could survive. Empty sealed structs remain usable everywhere else.

Custom descriptors provide type, layout, and explicit niche metadata. Fixed and sealed adapters use slot callbacks:

const TimestampSchema = proteus.schema.Custom(struct {
    pub const Owned = Timestamp;
    pub const View = TimestampView;
    pub const layout: proteus.Layout = .fixed;
    pub const niches: proteus.NicheSet = .none;

    pub fn slotWidth(word_size: proteus.WordSize) usize {
        _ = word_size;
        return 8;
    }
});

Fixed custom types expose invalid complete-slot byte patterns directly. For example, a 12-byte type that never contains twelve zero bytes can make that pattern available to an enclosing optional or niche-backed sealed union:

const zero_12 = [_]u8{0} ** 12;

pub const niches = proteus.NicheSet.explicit(&.{
    proteus.NichePattern{ .fixed = &zero_12 },
});

The pattern length must equal slotWidth for every W, and a zero-length pattern is rejected: it would compare equal to every slot. Both are compile errors.

The adapter's checked view and validation callbacks must return error.InvalidValue when the custom type itself encounters an exposed pattern. Enclosing schema-generated code recognizes an assigned niche before invoking those callbacks. Nothing at compile time can check that an adapter keeps this promise, so builds with runtime safety on assert it against the bytes each callback accepts; an adapter that returns a value for a pattern it also publishes would otherwise make Codec(T) and Optional(T) disagree about identical bytes.

The core fixed or sealed operation callbacks are:

pub fn resolveSlot(
    value: *const Owned,
    resolver: *const Resolver,
    slot_writer: anytype,
    word_size: proteus.WordSize,
    containing_region_end: usize,
) proteus.runtime.Error!void;

pub fn viewSlot(
    bytes: []const u8,
    containing_region_end: usize,
    word_size: proteus.WordSize,
    slot_start: usize,
) proteus.runtime.Error!View;

pub fn viewSlotValidated(
    bytes: []const u8,
    containing_region_end: usize,
    word_size: proteus.WordSize,
    slot_start: usize,
) View;

pub fn decodeOwned(
    value: View,
    allocator: std.mem.Allocator,
) proteus.runtime.Error!Owned;

pub fn deinitOwned(
    value: *Owned,
    allocator: std.mem.Allocator,
) void;

Resolver, serializeChildren, minChildRegionEnd, maxSlotWordValue, validateSlot, and canonicalRequirements are supplied when the sealed adapter has children or active W requirements. A sealed adapter that declares a Resolver must provide minChildRegionEnd, and canonicalRequirements when it also provides validateSlot. A fixed adapter is self-contained slot bytes and declares neither Resolver nor serializeChildren. Every adapter that provides decodeOwned also provides deinitOwned, empty when Owned contains no allocator-owned storage. Each of these is a compile error when violated, with the reason in the diagnostic. A custom decodeOwned implementation must clean up allocations it made before returning an error. viewSlotValidated is optional. When omitted, trusted viewing calls viewSlot after successful validation and treats an error as an adapter contract violation.

Dynamic custom adapters have a generated offset slot and use the region-level contract:

pub const layout: proteus.Layout = .dynamic;
pub const niches: proteus.NicheSet = .region_words;

pub fn slotWidth(word_size: proteus.WordSize) usize {
    return word_size.bytes();
}

pub fn serializeRoot(
    value: *const Owned,
    writer: *proteus.Writer,
) proteus.runtime.Error!usize;

pub fn viewRoot(bytes: []const u8) proteus.runtime.Error!View;

pub fn viewRootValidated(bytes: []const u8) View;

pub fn validateRoot(
    bytes: []const u8,
    mode: proteus.ValidationMode,
) proteus.runtime.Error!void;

Dynamic adapters use the same decodeOwned and deinitOwned callbacks. They do not declare Resolver; the codec generates child-region serialization, backward offsets, niche metadata, and enclosing canonical requirements. viewRootValidated has the same optional trusted-view contract as viewSlotValidated.

Schema references and recursive schemas are rejected explicitly in the initial scope.

Runtime

The schema-independent runtime is exposed through proteus.runtime. Reader and Writer use normal Zig error unions and retain a payload-bearing diagnostic for the most recent failure:

var reader = proteus.Reader.init(bytes);
const footer = proteus.runtime.RegionFooter.parse(&reader, bytes.len) catch |err| {
    const diagnostic = reader.diagnostic;
    _ = diagnostic;
    return err;
};

Create writers with an allocator and release their backing storage:

var writer = proteus.Writer.init(allocator);
defer writer.deinit();

var region = try writer.beginRegion(.one, 1, 4);
defer region.abort();
try region.writeIntLe(u32, 7);
const region_end = try region.finish();

defer region.abort() is required for transactional cleanup when a function returns before finish; it is a no-op after a successful finish.

A region writer holds the parent writer exclusively until finish or abort. Regions are written children-first, so a child region is completed before the region that refers to it is begun — appending to the parent writer while a region is open asserts in safe builds and is undefined behavior in optimized ones.

Run deterministic fixture mutation fuzzing:

tools/check-fuzz.sh 2000

The command must be run from the repository root. It mutates shared canonical fixtures, exercises checked viewing and both validation modes, and runs owned decoding with bounded allocation. CI uses a larger corpus on its weekly schedule.

Run the benchmark suite:

zig build benchmark

Runtime, assembly, compile-time, and binary-size methodology and baselines are recorded in benchmarks/README.md.

Generate API documentation and run the complete example:

zig build docs
zig build example -Doptimize=fast

Generated API documentation is written to zig-out/docs. The example in examples/task.zig defines a schema, constructs Owned, serializes it, performs explicit canonical validation, and reads fields through TrustedView.

Build and verify a versioned source archive:

tools/package.sh 0.1.0

The archive is written under zig-out/. Release tags and compatibility rules are covered by ROADMAP.md.

Run the complete pre-release gate, including Debug and ReleaseFast tests, compile-error fixtures, 100,000 deterministic mutations, assembly checks, checked and trusted benchmarks, docs, the example, repository hygiene, and the verified source archive:

tools/release-gate.sh 0.1.0

The fixture format and extension rules are documented in testdata/conformance/README.md.