Repositories

Layout

A repository is the representation of collections in a storage location. The layout is identical in a server’s own storage and in a mirror. Keys are relative to the location’s prefix.

nodes/<nodeHash>                             encoded node, gzip
bodies/<leafHash>.ndjson.gz                  the body of a record leaf
records/<recordHash>.json.gz                 an out-of-line record, gzip
schemas/<schemaHash>.json                    JCS(schema)
roots/<hex>.json                             JCS(root); <hex> is the version hash without "ulv2:"
private/<commitment>.json                    JCS(PrivateSetObject), where private sets are held
files/<fileHash>                             file bytes
collections/<collectionId>/collection.json   collection description
collections/<collectionId>/log/<seq>.json    version log entry
collections/<collectionId>/head.json         log head

• Bodies. One line per entry of the leaf, in entry order, each terminated by \n: the record’s canonical form, or a pointer {"$ref":"<recordHash>"} to records/. A body is one or more concatenated gzip members; readers MUST accept any number.
• Immutability. Objects outside collections/ MUST NOT change once written. A reader that does not trust a location MUST verify each object against its hash, and body lines against the leaf’s entries, before use.
• Write order. A writer MUST write every object a version reaches (leaves and bodies, interior nodes, the PrivateSetObject and root) before the log entry, and the log entry before head.json.
• Self-containment. No object refers to another location. Readers MUST ignore keys outside the layout.

Version log

Each collection has one signed log entry per version.

entry = {"actorId","appId","baseSemver","collectionId","createdAt","keyId",
         "message","prev","semver","seq","sig","versionHash"}

sig        = base64url(Ed25519 signature over JCS(entry without sig)), no padding
keyId      = first 16 hex characters of SHA-256(raw public key)
entry hash = SHA-256(JCS(entry)), sig included
prev       = entry hash of seq − 1, or null for seq 1
head.json  = JCS({"entryHash","seq","versionHash"}) of the latest entry

• collectionId is signed, so that an entry or log cannot be presented as another collection’s.
• createdAt is ISO 8601 UTC. appId, actorId, baseSemver and message MAY be null; writers SHOULD write actorId as null.
• collection.json holds the collection’s id, owner, slug, name and keys, an array of {"id", "alg": "Ed25519", "publicKey"}. It is neither hashed nor signed; readers MUST NOT depend on its serialization.
• A verifier MUST use a key only under the id derived from it.

A log is valid if and only if every entry from 1 to head.seq is present, each names the collection being read, each prev chains, each signature verifies under a trusted key, and head.entryHash and head.versionHash equal the last entry’s. Which keys are trusted is not yet specified.

Packs

A version is transferred as a pack: the objects it reaches that the receiver’s base version does not, under their repository keys, as an uncompressed POSIX tar (PAX headers for names over 100 bytes). File bytes and collections/ objects are not carried. Order:

1. the schemas the base lacks;
2. for each set sent, public first: for each record tree, the tree nodes not at the same position in the base’s tree of that type, parents before children, and after each new leaf the out-of-line records its body points to, then its body; then the new tree nodes of the set’s file tree;
3. the PrivateSetObject, if the private set is sent;
4. the root.

A receiver MUST NOT depend on any other order than leaf before body, out-of-line records before the body that points to them, and the root last.

A receiver MUST verify each object against its key before writing it, MUST re-derive every received tree by merging the entry changes into its base tree and obtain exactly the received root, count and bytes, MUST write the root last, and MUST refuse a pack that fails any check. Packs are pull-only: a server MUST NOT accept them from clients.

Serving over HTTP

A server serves each collection under a collection URL, an absolute URL without a trailing slash whose form is the server’s choice. On underlay.org it is https://www.underlay.org/api/collections/<owner>/<slug>. A server MUST provide these reads:

GET  <collection>/log?after=<seq>&limit=<n>
     200 {"collection": collection.json | null, "head": head.json | null, "entries": [entry, ...]}

GET  <collection>/versions/<v>/pack?base=<v>&sets=public|all
     200 application/x-tar; x-underlay-version, x-underlay-base, x-underlay-sets

GET  <collection>/versions/<v>/manifest?cursor=<c>&limit=<n>
     200 {"semver", "hash", "schemas": {slug: schemaHash},
          "records": [{"id", "type", "hash", "private"?}],
          "pagination": {"limit", "hasMore", "nextCursor"}}

GET  <collection>/files/<fileHash>     the bytes, or a redirect to them
HEAD <collection>/files/<fileHash>     content-length

<v>: a semver (leading "v" optional), a version hash (ulv2:<hex>), or latest.
     A hash that names several versions (a reverted change repeats one) selects the latest.

• Log. Entries with seq greater than after (default 0), ascending. A server MAY return fewer than limit; the client repeats from the last seq received until head.seq.
• Pack. Without base, every object the version reaches. sets defaults to public.
• Manifest. Records in order of type, then id. A caller who cannot read the private set receives the public set only. Cursors are opaque; a server MAY cap limit.
• Files. <fileHash> MAY carry a sha256: prefix. A server MUST serve a file only to a caller who may read a set that holds it.

Errors carry {"error": <message>}. 404: the collection, version, base or file does not exist or may not be read; a server MUST NOT distinguish these. 403: sets=all without access to the private set. 400: an invalid sets. 451: a file the caller could otherwise read, withheld for legal reasons (a file the caller may not read is a 404). Authentication is the server’s choice; a server without access control MUST serve public sets only.

A client MUST NOT trust a server’s responses: it verifies the log, receives packs under the rules above, and verifies files against their hash. The manifest is not verifiable on its own.