The Underlay protocol

Version 2 of the Underlay protocol specifies the canonical encoding and hashing of records, schemas and files; the input rules published data must satisfy; the construction of record and file trees; the version root and its hash; the repository layout in which collections are stored; the signed version log; the pack format by which versions are copied; and the HTTP interface by which servers serve and accept versions.

These pages restate the normative specification, docs/protocol-v2.md, section by section. Where they differ, the specification is authoritative.

Conformance

The key words MUST, MUST NOT, SHOULD, SHOULD NOT and MAY are to be interpreted as described in BCP 14 (RFC 2119, RFC 8174) when they appear in all capitals. Notes and examples are informative.

A conforming implementation accepts and rejects the same inputs, builds the same trees and computes the same hashes as the specification, byte for byte. A change to any rule that alters what is accepted, how a tree is built or what is hashed requires a new protocol version.

Terminology

• Collection: a named sequence of versions, identified by a collection id.
• Record: an id, a type and a JSON value data.
• Type: a named class of records, described by a schema (JSON Schema draft-07).
• File: a byte string, identified by its SHA-256 hash.
• Access set: one of a version’s two partitions, public and private.
• Tree: a sorted set of entries partitioned into tree nodes by their keys.
• Version: an immutable state of a collection, described by a root document. Its version hash is ulv2: followed by the SHA-256 of the root.
• Repository: the objects that represent collections in a storage location.
• Server (also Underlay node): an HTTP service that serves collections and may accept publications.
• Owner: a party permitted to read a collection’s private set, as determined by the server.

Properties

Informative. The rules below have these consequences:

• A tree depends only on its entries, not on the order of construction, so identical content yields an identical version hash.
• Every object is verifiable against the hash that names it, and a version’s public content against its version hash.
• A reader of the public set learns of the private set only whether it exists; the private set is committed to with a per-collection salt.
• Publication, comparison and transfer cost in proportion to what changed, since unchanged subtrees are shared by hash.
• Each collection’s versions are recorded in a signed, hash-chained log, so a copy of its repository in any location can be verified independently of the server that wrote it.

Contents

• Records and schemas (§§2–6): canonical JSON, input rules, record and schema hashes, the validation dialect, files.
• Trees and versions (§§7–10): key order, tree construction and encoding, access sets, the version root, semver.
• Repositories (§§11–11.3): the repository layout, the version log, packs, and the HTTP reads every server provides.
• Push and pull (§11.4, §13): delta push, errors, and security considerations.

Reference

Specification: docs/protocol-v2.md in the Underlay repository. Reference implementation: @underlay/protocol (packages/protocol), which runs in Node, Cloudflare Workers and browsers. Test vectors: packages/protocol/test/vectors/v2.json. The protocol is stewarded by Knowledge Futures.