Concepts

Underlay has four core primitives. Everything else is built from these.

Collection

A collection (plural: collections) is a named, versioned body of structured data. It belongs to an account (a user or an organization) and is identified by :owner/:slug, e.g. knowledge-futures/pubpub-archive.

A collection can be public (browsable by anyone) or private (visible only to the owner and org members). Each collection has its own independent version history.

Version

A version is an immutable snapshot of a collection at a point in time. Each version contains:

A JSON Schema for each record type
A set of records (the actual data)
References to files (binary assets)
A metadata object that can contain readme, license, and other fields

Versions are identified by semver (e.g. v1.0.0, v1.1.0, v2.0.0). The semver is derived automatically from what changed:

A type added or removed, or a schema changed → major bump
Records added, removed or changed (including made private or public) → minor bump
Metadata or file changes only (readme, license, etc.) → patch bump

Each version also has a hash: ulv2: followed by the SHA-256 of the version root, which covers the metadata, every public schema, record and file, and a salted commitment to the private ones. Two versions with the same hash have identical content.

Record

A record has an id, a type and a data payload. Records are the rows of your data. Within a version, a type and an id identify one record.

{
  "id": "pub-001",
  "type": "Publication",
  "data": {
    "title": "The Structure of Scientific Revolutions",
    "doi": "10.1234/example",
    "authors": ["author-001", "author-002"],
    "pdf": { "$file": "sha256:a1b2c3..." }
  }
}

Records are content-addressed: each record is identified by the SHA-256 hash of its canonical JSON ({"id":...,"type":...,"data":...}). This means:

Records are stored in content-addressed trees, so versions share every part of the data they have in common.
Pushing a new version only transfers what changed (see push and pull).
A record’s hash finds the collections and versions you can read that include it (provenance).

Relationships between records are expressed as ID references (just strings). There are no joins, no foreign keys. An LLM or application can resolve references by reading the schema and records together.

Records are validated against their type’s schema as they are uploaded. If a record has top-level fields the schema’s properties don’t list, the upload is refused with a 422 listing them. Set strip_unknown_fields when opening the push to drop them instead.

Binary data is referenced via {"$file": "sha256:..."}, a pointer to a content-addressed file. The wire format for records is NDJSON, one record per line, independently hashable and streamable.

File

A file is a binary blob (PDF, image, dataset, anything) stored by its SHA-256 hash. Files are content-addressed: the same bytes always produce the same hash, so identical files are stored only once regardless of how many records reference them.

Files are uploaded before the commit that references them. The commit is refused unless every $file reference in your records points to a file this collection holds: uploaded to it, or already in one of its versions.

Accounts

Underlay has two account types:

Users: people, who sign in with KF Auth. Each user has a personal account that can own collections.
Organizations: group accounts with members who have roles (owner, admin, member)

API keys belong to a user or an organization and may be confined to some collections. A key has the scope read, write or admin, and never exceeds its holder’s role: read and write keys act as a member, and an admin key keeps an owner’s or admin’s powers. A collection-scoped key acts as a member of its collections whatever its scope, and is refused on account and organization endpoints.

Privacy & Visibility

Underlay has privacy at three levels, so private data can sit alongside public data in the same collection. Each version has a public set and a private set; members of the owning organization read both, everyone else the public set.

Collection-level

A collection can be public (listed in browse, readable by anyone) or private (visible only to the owner and org members).

A published version never changes, so anonymous reads of it are cached for up to about eleven minutes. Making a collection private can therefore take that long to reach every reader; making it public takes effect at once.

Type-level

Mark an entire record type as private in the schema by adding "private": true at the root of the type’s schema. All records of that type are hidden from public readers, and the type is absent from every read, schemas included.

Record-level

Mark an individual record private by adding "private": true to its line when you upload it. The flag is not part of the record’s data or its hash:

{"id": "pub-001", "type": "Publication", "data": {...}, "private": true}

The record is absent from listings, manifests, diffs, exports and the NDJSON stream for non-members; members of the owning org still see it.

Privacy belongs to the version, not the record. A record keeps its set from one version to the next until you upload it again: uploading it with "private": true makes it private, and uploading it without the flag makes it public. Read the current flags back from GET .../versions/:semver/manifest, which marks private entries with private.

Redaction is forward-only: marking a record private in a new version hides it from that version on. Earlier versions are immutable and still serve it. Because file access resolves across every version, a file referenced publicly in an earlier version also stays downloadable after the referencing record is made private.

Fields

"private": true on a field inside a schema is refused. Put private fields in a private type, or push the whole record as private.

A version’s hash covers the private set only through a salted commitment, so public readers can verify everything they can see without learning anything about the private content. Members can check the private set against the commitment.