zTensor Container Format, Version 2

Status: Draft 4 · File extension: .zt · Footer version integer: 2

1. Introduction

zTensor v2 is a container format for tensor data. It is designed around one premise: a format survives by separating what must never change from what is allowed to die.

The format is organized in three layers with explicit stability contracts:

Everything in the file is a blob: an aligned, contiguous, unnamed byte range. Tensor data is blobs. The manifest itself is a blob. The footer is a fixed-size trailer whose only job is to point at the root manifest blob and protect it with a hash.

1.1 Design principles

1. Container and semantics are fully separated. The container knows nothing about tensors. It stores aligned blobs and one manifest.
2. Everything is a blob. One mechanism; no special cases.
3. Validation is not optional. Every invariant in this document is a MUST, and readers MUST reject files that violate them. There are no silent-fallback rules.
4. The format owns only the minimal verifiable contract. Values derivable from the data (e.g., actual alignment) are not stored. Policy knobs (e.g., placement) belong to writers, not to the format.

1.2 Conventions

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119.

All multi-byte integers in the container (L0) are little-endian. All tensor element data is little-endian. The only exception is CBOR's internal length encoding, which is big-endian per RFC 8949 and handled by any CBOR library.

2. Container (L0)

2.1 File structure

+----------------------------------------+  offset 0
| Magic (8 bytes)                        |
+----------------------------------------+
| Blob heap                              |
|   blob, blob, ..., manifest blob       |  every blob offset ≡ 0 (mod 4096)
|   (padding between blobs = 0x00)       |
+----------------------------------------+
| Footer (40 bytes)                      |
+----------------------------------------+  EOF

A file MUST be at least 48 bytes long (8-byte magic + 40-byte footer).

2.2 Magic

The 8-byte magic appears at offset 0 and again at the end of the footer:

89 5A 54 32 0D 0A 1A 0A        (0x89 'Z' 'T' '2' CR LF SUB LF)

Following the PNG convention, the high-bit first byte, the CRLF pair, and the trailing LF detect 7-bit stripping, line-ending translation, and other transmission corruption.

2.3 Footer

The footer occupies the last 40 bytes of the file:

manifest_hash is an integrity check against corruption, not a cryptographic commitment; XXH3-64 is acceptable to freeze into L0 because its role is error detection. Cryptographic identity is provided by digests (§6) and canonical form (§6.3), which are algorithm-agile.

2.4 Blobs and alignment

• A blob is a contiguous byte range identified by (offset, length).
• Every blob offset MUST satisfy offset % 4096 == 0 and offset >= 4096.
• Every blob MUST satisfy offset + length <= file_size − 40 (blobs may not overlap the footer).
• Blob references within a file, plus the manifest blob, MUST be pairwise identical or disjoint: two references are valid iff they have exactly equal (offset, length) or do not overlap at all. Partial overlap MUST be rejected. Identical references are deliberate: they enable weight tying (e.g., input/output embeddings sharing one blob) and intra-file dedup, while the dangerous case (one range validated under one interpretation, read under another) remains impossible. (Zero-length blobs are exempt from this check but not from the alignment check.)
• All bytes between blobs, and between the magic and the first blob, MUST be written as 0x00. Readers MAY ignore padding content.

4096 is a floor, not a ceiling. Writers MAY place blobs at any coarser alignment (16 KiB, 64 KiB, ...); a multiple of a coarser power of two is automatically a multiple of 4096, so stricter placement is always valid. Because alignment is observable from the offsets themselves, the actual alignment used by a writer is not stored in the file. See §6.3 for the canonical placement (64 KiB).

2.5 Append and generations

The container is append-only. To amend a file, a writer appends new blobs, a new manifest blob, and a new footer at the end. Prior manifest blobs remain in the file as unreferenced blobs; the file's meaning is always defined by the footer at EOF. Writers MUST NOT truncate or overwrite existing bytes.

Files SHOULD nevertheless be treated as immutable artifacts once published. Canonical files (§6.3) contain exactly one manifest and no unreferenced blobs.

A file whose footer is not at EOF, for example after a crashed append, is invalid; there is no reader-side recovery (§10). Durable publication is a transport concern: write to a temporary name, then rename atomically (Appendix B).

3. Manifest (L1)

3.1 Encoding

The manifest blob is a single CBOR map encoded in the Core Deterministic Encoding profile of RFC 8949 §4.2.1 (definite lengths, shortest-form integers, bytewise-lexicographic key order).

Readers MUST reject a manifest that is not in deterministic encoding: non-shortest heads, indefinite lengths, unsorted or duplicate map keys, and non-canonical floats (NaN other than 0xf9 0x7e00, or a float wider than the value requires). Anything weaker is unsound, not merely lenient: a reader that accepts non-canonical floats will accept two NaN-payload map keys as distinct, which is duplicate keys in disguise. Readers MUST reject a manifest whose manifest_length exceeds 1 GiB, before parsing.

Readers MUST ignore map keys they do not recognize, at every level. This is the L1 minor-evolution mechanism; the footer version integer is the major one.

An attributes field MUST be a map whose keys are text strings obeying the name rules of §3.5. Attribute values (at every level) MUST NOT use CBOR tags; permitted types are integers, text strings, byte strings, booleans, null, floats, arrays, and maps. Nesting is limited to 32 levels measured from the manifest root, so a reader needs one depth counter rather than a per-value one. The manifest blob itself is always stored raw; it is never compressed or encoded.

3.2 Root schema

Shown in CBOR diagnostic notation:

{
  "attributes": { ... },            ; optional, arbitrary file metadata
  "shards": { "00001": {...}, ... }, ; optional, see §7; absent ⇒ single file
  "objects": {                      ; required in a root manifest
    "layer1.weight": <object>,
    "layer1.bias":   <object>
  }
}

3.3 Object

{
  "shape": [4096, 4096],
  "layout": "dense",                ; or a namespaced profile id, §5
  "attributes": { ... },            ; optional
  "parts": { "data": <part> }
}

3.4 Part

{
  "dtype": "bf16",
  "type": "...",                    ; optional logical type, §4.2
  "blob": [65536, 33554432],        ; [offset, length]
  "shard": "00001",                 ; optional; absent ⇒ this file, §7.1
  "encoding": "zt.zstd-seekable/1", ; optional; absent ⇒ raw
  "decoded_length": 67108864,       ; required iff encoding is present
  "digest": "xxh3:9f2c..."          ; optional, over decoded bytes
}

The decoded size of a part is length when encoding is absent, else decoded_length.

Digests cover decoded bytes because a digest identifies content, not its encoding; re-encoding a blob does not change its digest. Transport integrity of encoded bytes is the encoding profile's responsibility (e.g., zstd frame checksums; see profiles/zt.zstd-seekable-1.md).

3.5 Names

Object names, part names, and attribute keys MUST be non-empty, valid UTF-8 strings of at most 1024 bytes containing no U+0000. That is the entire reader-side check. Writers SHOULD emit names in Unicode Normalization Form C, and canonical form (§6.3) requires it, but readers do not verify normalization: NFC verification needs Unicode tables, which would break the minimal-reader contract (§11). Normalization is a writer duty, not a reader check.

3.6 Validation summary

A reader MUST reject a file if any of the following fail:

1. Footer: magic, version supported.
2. Manifest: bounds (manifest_offset/length inside the file, off the footer), size cap, XXH3-64 match, deterministic-CBOR parse, no duplicate keys.
3. Every blob reference: blob has exactly two elements; shard, if present, is a key of shards; offset % 4096 == 0; offset >= 4096; offset + length within the referenced file's data region.
4. Blob references are grouped by the file they point into (the shard name, or its absence) and checked per group: within each file, references, plus the manifest blob for the containing file, are pairwise identical-or-disjoint; any partial overlap is rejected (§2.4). References into a shard are checked from the root manifest alone, without opening the shard.
5. Layout rules for every object whose layout the reader interprets (e.g., the dense size equation, §5.1).
6. Name rules (§3.5); shape rank and overflow rules (§3.3); attribute value constraints (§3.1).

4. Type system (L2)

4.1 Storage types (dtype)

A closed set of 12 primitives. The storage layer's only responsibilities are byte width and endianness; therefore no storage type may have invalid bit patterns.

There is no bool storage type; bool is a logical type over u8 (Appendix A). This is deliberate: a type with invalid bit patterns (0x02..0xFF) does not belong in the layer whose contract is "any byte pattern is valid."

4.2 Logical types (type)

An open, registered set giving meaning to raw storage elements. When absent, the logical type equals the storage type and no interpretation is needed. Each registered logical type specifies its required dtype, its size function (the decoded byte size for a given logical element count) and, for packed sub-byte types, the bit order. A plain type has size(n) = n × width(dtype); a compound type like complex64 has size(n) = 2n × width; a packed type like f4_e2m1 has size(n) = ⌈n/2⌉. See Appendix A.

Unknown logical types are an error. A reader asked to decode a part whose type it does not recognize MUST refuse. Raw structural access (bytes plus metadata) MAY be offered through an explicitly structural API, but a reader MUST NOT silently reinterpret unknown-typed data as its storage type. Two conforming readers must either agree on a tensor or agree on an error.

5. Layouts (L2)

An object's layout selects how its parts combine into a tensor.

5.1 Core layout: dense

The only layout defined by this document.

• Exactly one part, named "data".
• Row-major (C-contiguous) element order.
• The part's decoded size MUST equal the logical type's size function applied to num_elements(shape) (for a part with no type, that is num_elements × width(dtype)). Readers MUST verify this equation.

5.2 Layout profiles

Every other layout, whether sparse, quantized or something future, is a profile: a separately published mini-specification identified by a namespaced, versioned id such as zt.sparse_csr/1 or pie.paged_kv/1. Profiles live beside this document under spec/profiles/; this core specification defines the mechanism and nothing else.

A layout profile MUST completely specify: its required and optional parts; each part's permitted dtype/type; how each part's decoded size derives from shape and the object's attributes; the meaning of every attribute key it uses; and its validation rules. A profile that cannot be implemented from its text alone is not a profile. The test is whether two independent implementations, working from the text, produce interchangeable files.

The zt. namespace is reserved for profiles blessed by the zTensor registry. Vendors MUST use their own namespace prefix. Version suffixes (/1, /2) are mandatory; any semantic change requires a new version.

A reader encountering an unknown layout MUST NOT interpret the object; it MAY expose the object structurally (shape, attributes, raw parts).

The registry at the time of writing: zt.sparse_csr/1, zt.sparse_coo/1, zt.quant_group/1, zt.mx/1, the gguf.<type>/1 family, and the encoding profile zt.zstd-seekable/1.

Parametric and opaque profiles

There are two kinds of profile, and choosing the wrong one is the common way to write a bad one.

A parametric profile describes a space, not a scheme: its attributes determine the decoder, and two different attribute sets under the same profile are two different schemes. zt.quant_group/1 is one: affine group quantization, where the bit width, group size, packing order, scale form and zero-point form are each stated. A parametric profile MUST make every parameter its decoder needs a required attribute; anything left unstated will be inferred from something incidental, which is how a file that happens to use a 32-element group comes to be read as the one scheme that used to have 32-element groups.

An opaque profile does not describe the payload's internal structure at all. It names an authoritative external definition and preserves the bytes verbatim. The gguf.<type>/1 family is opaque: a ggml block struct interleaves scales with data, and q4_k nests a second level of scales quantized to six bits inside a super-block, which is structure no attribute set describes without inventing a layout language. An opaque profile MUST still carry the constants a reader needs to validate sizes without knowing the layout (for the gguf family, elems_per_block and block_bytes), and its version suffix carries real weight: an external definition that changes under a stable name is exactly what the suffix distinguishes.

Which kind a profile is settled by one question: can the attributes alone determine the decoder? If they cannot, do not force it. A profile parameterized past what its payload actually admits is readable only by implementations of the invented language, which is worse than an opaque profile with an accurate name.

5.3 Encoding profiles

The core defines exactly one encoding: raw (the absence of the encoding field). Compression algorithms are the most obviously mortal component of any container, so they live in L2 like layouts: namespaced, versioned encoding profiles, published under spec/profiles/ like layouts.

A reader encountering an unknown encoding MUST refuse to decode the part (structural access to the encoded bytes MAY be offered).

6. Integrity, identity, and canonical form

6.1 Integrity ladder

Cheapest first; each rung is independent:

1. Structural: footer magic and bounds checks (always on).
2. Manifest: XXH3-64 in the footer (always on).
3. Transport: encoding-profile checksums, e.g. zstd frame checksums (on whenever the encoding is used).
4. Content: per-part digest over decoded bytes (optional; verify mode).
5. Artifact identity: whole-file hash of a canonical file, optionally signed (§6.5).
6. Model identity: content digest (§6.4), which is the same for a model however it was laid out or split.

6.2 What is deliberately not protected

Padding bytes are written as zero but not covered by any digest; unreferenced blobs (superseded generations, §2.5) are not covered. Canonical form closes both gaps by forbidding them.

6.3 Canonical form

A file is canonical iff all of the following hold:

1. Exactly one manifest blob; no unreferenced blobs.
2. Every blob offset ≡ 0 (mod 65536), assigned by packing blobs at successive 64 KiB boundaries starting at offset 65536, in the order of rule 3.
3. Blobs are ordered by the bytewise-lexicographic (object name, part name) of their first reference. Parts with byte-identical decoded content MUST share a single blob (aliased references, §2.4); a writer that finds sharing candidates by hashing MUST confirm equality on the bytes themselves, since a hash collision would otherwise alias two different tensors onto one blob. The manifest blob comes last; the footer immediately follows it.
4. All parts use raw encoding (no encoding field) and every part carries an xxh3 digest.
5. All names are in Unicode Normalization Form C.
6. The model is single-file (no shards table). A canonical multi-file profile additionally requires a fixed shard-partition policy and is deferred.

Because the manifest encoding is deterministic and nothing in a canonical file depends on time, randomness, or writer identity, identical tensors produce bit-identical files. The hash of a canonical file is therefore a stable identity for the model. Canonical form is the RECOMMENDED distribution format; non-canonical files are still fully conforming.

The 64 KiB canonical alignment guarantees that no two blobs share a page on any known page size (4 KiB x86/ARM, 16 KiB Apple Silicon, 64 KiB ARM64 distributions): every tensor can be independently memory-mapped, registered, and evicted (madvise) with exact page ranges, on every platform, with no fallback path. Average cost is 32 KiB per tensor, which is noise at weight-file scale.

6.4 Content digest

The whole-file hash of §6.1 rung 5 identifies an artifact: these bytes, this layout, this file. Two files holding the same tensors have different hashes if they were aligned differently, split differently, or written in a different order. That is what canonical form exists to pin down, and it is why canonical form is single-file (§6.3 rule 6): once a model is split, there is no fixed layout to hash.

The content digest identifies the model instead, and is defined so that layout cannot reach it. It is computed over this CBOR value, encoded with the deterministic rules of §3.1:

[
  "zt.content/1",
  {
    "<object name>": {
      "shape":   [u64, ...],
      "layout":  "<layout id>",
      "attributes": <the object's attributes, or omitted>,
      "parts": {
        "<part name>": {
          "dtype":  "<dtype>",
          "type":   "<logical type>",   ; omitted when absent
          "digest": "<algo>:<hex>"      ; the part's digest, over decoded bytes
        }
      }
    }
  }
]

The value is a two-element array: a version tag, and the manifest's objects map reduced to the fields above. Map keys are sorted by §3.1, so the object and part order is fixed. The digest of the whole is "<algo>:<hex>" for whichever algorithm the caller chose.

What is deliberately absent is every fact about layout: offsets, lengths, alignment, padding, blob sharing, encodings, the shard table, shard names, shard digests, and how many files there are. A part's digest covers decoded bytes (§3.4), so re-encoding a blob does not reach the content digest either.

Consequences:

• The same tensors give the same content digest whether they are in one file or fifty, at 4 KiB or 64 KiB placement, raw or compressed.
• It is computable from the manifest alone. A reader that has fetched a root and nothing else can compute it, at no I/O cost proportional to the model.
• It is defined only when every part carries a digest. Where one does not, there is nothing to stand for that part's content, and the content digest is undefined rather than approximated. Canonical form guarantees the digests (§6.3 rule 4).

The content digest MUST NOT be stored in the file. A stored value is a claim that can be false; a computed one cannot be.

6.5 Signing

Signing is out-of-band: a detached signature over the bytes (or a cryptographic hash) of a canonical file. For multi-file models, signing the root file suffices if the root's shard digests (§7) use a cryptographic algorithm (sha256), since the root manifest then commits to every shard byte (one-way Merkle direction).

7. Multi-file models (shards)

Sharding is not a feature bolted onto the format; it is the general case of the blob reference, with the single file as the degenerate case.

7.1 Shard table

"shards": {
  "00001-of-00002": { "size": 8589934592, "digest": "sha256:..." },
  "00002-of-00002": { "size": 8589934592, "digest": "sha256:..." }
}

• The shard table is a CBOR map from shard name to a shard identity.
• A shard name MUST be a non-empty text string of at most 64 bytes, made only of A–Z, a–z, 0–9, ., _, -, and MUST NOT begin with .. Readers MUST reject a name outside this set, and MUST reject a key that is not text.
• The containing file is never a key: it is named by the absence of a shard field on a part (§3.4). It needs no identity entry either, since its manifest is protected by the footer hash and a whole-file self-digest would be circular.
• Every entry MUST carry size (exact file size in bytes) and digest (whole-file). Because a name is only a label, the digest is the shard's identity; it is therefore required, not optional. xxh3 is the minimum; distribution files intended for signing SHOULD use sha256 (§6.5).
• When shards is absent, no part may carry a shard field.

A shard name is a label, not a location: no paths or URLs appear anywhere in the format, and a name carries no promise about what any file is called. A path is owned by the filesystem, is not verifiable by the format, and would couple model identity to naming, since a rename would change or break the model. Resolving a name to bytes is entirely the transport's concern (Appendix B gives the RECOMMENDED conventions); because identity lives in the content, a resolver can always ignore the name and locate a file by size and digest instead.

The character set is narrow for one reason: the conventional resolvers use a name as a single path component. Constraining it here means that a manifest cannot express a path such as ../../etc/passwd, so no consumer has to sanitize one. Spread across every implementation, that obligation would eventually be missed by one of them.

7.2 Data shards

A data shard is a minimal container: magic, blob heap, and a footer with manifest_offset = manifest_length = manifest_hash = 0. It carries no manifest and describes nothing; all meaning lives in the root manifest.

Knowledge flows one way: the root knows its shards, and shards do not know their root. This avoids circular hashing and keeps multi-file output deterministic.

A shard need not be a data shard: any .zt file, including one with its own manifest, may serve as a shard of another model. The referencing manifest treats it purely as a blob heap identified by size and digest. This is how overlay models work: a LoRA root can reference the base model file's blobs directly and store only its own deltas.

7.3 Verification ladder

Cheapest first: (1) shard footer magic and version; (2) actual file size equals the root's size; (3) whole-file digest match (deep verify / signed distribution).

Placement of blobs across shards is entirely the writer's policy. Writers MAY shard by layer, by pipeline stage, or by size threshold; the container imposes nothing.

8. Reading a file

1. Require file_size ≥ 48.
2. Read the last 40 bytes. Verify magic and version.
3. If manifest_length == 0: this is a data shard; only structural access
   is possible. Stop.
4. Verify manifest bounds and manifest_length ≤ 1 GiB.
   Read the manifest bytes; verify XXH3-64 == manifest_hash.
5. Parse deterministic CBOR; reject duplicate keys.
6. Run the validation summary (§3.6) over the whole manifest, including the
   global non-overlap check.
7. To load a part: resolve the shard, seek to offset, read length bytes,
   decode per the encoding profile (verifying decoded_length exactly),
   verify digest if requested, interpret as dtype/type elements.

For remote access, steps 2 and 4 are exactly two range requests (the footer is fixed-size).

8.1 Security rules

• No code execution. No field of this format is ever evaluated.
• Bounds before bytes. All §3.6 checks precede any data read.
• Decompression limits. decoded_length is enforced as an exact output size, pre-allocated and verified; a short or long decode is an error, never a silent zero-fill.
• Caps. Manifest ≤ 1 GiB; implementations SHOULD additionally cap total decoded allocation per read call.

9. Versioning policy

• L0 is frozen. The magic, footer layout, alignment floor, and byte order defined here never change. A hypothetical future major version changes the footer version integer and may point the footer at a different root structure. The container skeleton survives.
• L1 minor evolution: new OPTIONAL manifest fields; readers MUST ignore unknown fields. L1 major evolution: increment the footer version.
• L2 evolves by registry: new logical types, layout profiles, and encoding profiles are added without touching this document. Existing profile versions are immutable; semantic changes require a new /n.

10. Non-goals

This format deliberately does not attempt to be:

• A database. No random access over billions of objects, no query language, no in-place mutation. The manifest is a monolith by design; if you need an index over millions of entries, you need a different tool.
• A partial-update store. Amendment is generational append (§2.5), not in-place writes.
• A crash-recovery journal. A file whose footer is not at EOF is invalid, full stop. Durability comes from atomic publication (write to a temporary name, rename into place; Appendix B), not from reader-side recovery scans.
• An encryption container. Encryption belongs to the transport or filesystem layer.
• A compute or graph description. This format stores bytes and their interpretation, never behavior. (This is the tombstone on pickle's grave.)

Scope discipline is a futureproofing feature: these boundaries are what keep the format small enough to remain correct.

11. Conformance

A conforming implementation is one that passes the conformance corpus: a versioned set of golden files: valid files with their expected decoded contents, and invalid files that MUST be rejected with the corresponding §3.6 rule. The corpus and a minimal reference reader (a few hundred lines, no dependencies beyond CBOR and XXH3) are normative companions to this document: where prose and corpus disagree, the corpus is the bug tracker's problem, but implementations follow the corpus until resolved.

The mandatory-to-implement surface is deliberately small: the dense layout, raw encoding, and the 12 storage types. Everything beyond that is optional to support but mandatory to refuse cleanly (§4.2, §5.2, §5.3).

Appendix A: Registered logical types (initial)

A registry entry defines its required dtype, its size function, and, for packed sub-byte types, the bit order. The dense layout applies the size function to num_elements(shape); other layout profiles state which element count each part's size function receives (e.g., the reserved zt.mx/1 profile gives the scales part one f8_e8m0 element per 32-element block).

Appendix B: Recommended conventions (non-normative)

• Shard resolution (positional): for a root file named <stem>.zt, a shard named n resolves to a sibling file <stem>-<n>.zt (root model.zt with shards named 00001, 00002 → model-00001.zt, model-00002.zt). Naming shards after the segment they already carry in distribution, such as 00001-of-00003, reproduces existing file names exactly. Renaming the family together (model → llama) keeps it resolvable. The name is safe to use as a path component unchanged, because §7.1 admits no separators or ...
• Shard resolution (content-addressed): a store MAY resolve by digest instead, e.g. blobs/xxh3/<hex>; shard references are then location-independent, which also enables sharing shards across models.
• Shard resolution (by identity): a resolver MAY ignore names entirely and scan a directory, matching each file by size and whole-file digest. This is the only convention that survives arbitrary renaming, and it is the recommended fallback when a convention lookup fails.
• Writer alignment flag: reference writer default is canonical placement (64 KiB). A --align option may lower placement to any multiple of 4096 for space-sensitive, non-distribution files (e.g., small adapters).
• Advisory attributes: writers MAY record "alignment": <n> in file attributes; it is informational only and readers MUST NOT trust it over the observable offsets.
• Atomic publication: write to a temporary name in the destination directory, fsync, then rename(2) into place. A crashed write never leaves a partial file under a published name.