Push and pull

Delta push

A client publishes a version by delta push: it opens a session against a base version, uploads the records it adds or changes and the (type, id) pairs it deletes, and commits. The server builds the trees, writes and signs the log entry, and assigns the semver (semver rules). Delta push is the only publication mechanism; a server MUST NOT accept tree nodes or packs from clients.

POST   <collection>/push                  open a session
POST   <collection>/push/<sid>/records    upload records (NDJSON)
POST   <collection>/push/<sid>/deletes    upload deletes (NDJSON)
PUT    <collection>/files/<fileHash>      upload a file
POST   <collection>/push/<sid>/commit     commit (?async=true for the background)
GET    <collection>/push/<sid>            session status
DELETE <collection>/push/<sid>            abandon the session

Opening a session

The body is a JSON object; every member is OPTIONAL.

The response is 200 with session_id; base (the semver the session is against, or null); needed_files, the declared files the server does not hold for the collection, which the client MUST upload before committing; expires_at, extended by each records or deletes request; and limits, the server’s own, by which a client MUST size its requests: open_bytes, batch_bytes, batch_lines, session_idle_seconds, open_sessions (per user, open or committing) and file_bytes.

Records and deletes

• …/records takes NDJSON record lines and answers {"received": n}. Each line MUST satisfy the input rules and its type’s schema, and its type MUST be in the session’s type set.
• A record whose data has top-level members absent from its schema’s root properties MUST be refused, unless the session set strip_unknown_fields, in which case they are removed before hashing. A schema without root properties admits any members.
• If any line fails, the server MUST answer 422 with validationErrors, one per failing line with its 1-based line among the request’s non-empty lines, and MUST store nothing from the request. A server MAY list only the first failures, with totalErrors giving their number.
• …/deletes takes NDJSON {"type", "id"} lines and answers {"received": n}. Each line MUST satisfy the input rules for syntax, duplicate keys, unsafe integers and lone surrogates, and its id and type the record id and type slug rules; the type MUST be in the session’s type set. Deleting a pair the base does not hold is not an error. A request with no lines is a 400.
• Within a session, the later upload of a (type, id) supersedes an earlier one, record or delete.

Files and commit

• PUT …/files/<fileHash> stores a file for the collection: 201, or 400 if the bytes do not hash to <fileHash>. A server MAY offer other upload mechanisms. Every file a new record references, and every declared file, MUST be held for the collection at commit: in a file tree of the base, in the public set of any earlier version, or uploaded to the collection. A file held only for another collection is not held, and every upload is hashed, even of a file the server already stores.
• …/commit answers 201 with the version, or 202 when it runs in the background. A client MAY request the background with ?async=true; a server MAY choose it for any commit. The client then polls GET …/push/<sid> until status is committed (result holds the 201 body) or failed (error holds the failure).
• Committing a session that has already committed answers 201 with the same body.
• A client that holds the base SHOULD compute the new version hash and compare it with the returned hash.

POST <collection>/push
{"base": "v1.2.0", "schemas": {"Publication": {...}}, "metadata_patch": {"readme": "..."},
 "files": {"add": ["9f86d0..."]}, "message": "Weekly update"}
→ 200 {"session_id": "...", "base": "v1.2.0", "needed_files": ["9f86d0..."],
       "expires_at": "...", "limits": {"open_bytes": ..., "batch_bytes": ..., "batch_lines": ...,
       "session_idle_seconds": ..., "open_sessions": ..., "file_bytes": ...}}

PUT <collection>/files/9f86d0...                       (file bytes)
→ 201

POST <collection>/push/<sid>/records
{"id":"pub-004","type":"Publication","data":{...}}
{"id":"pub-005","type":"Publication","data":{...},"private":true}
→ 200 {"received": 2}

POST <collection>/push/<sid>/deletes
{"type":"Publication","id":"pub-003"}
→ 200 {"received": 1}

POST <collection>/push/<sid>/commit
→ 201 {"semver": "v1.3.0", "hash": "ulv2:...", "recordCount": 4, "fileCount": 1,
       "changes": {"added": 2, "removed": 1, "updated": 0}}
→ 202 {"session_id": "...", "status": "committing"}         (background commit)

Clients without a copy

Informative. A client that keeps no copy of the collection reads the base’s manifest, compares each of its records with it by (type, id), record hash and set, uploads the records that are new, changed or moved between sets, and deletes the pairs it no longer has. The upload is then proportional to the changes. If another publication intervenes, the session or commit answers 409 and the client repeats the comparison against the new head.

Pull

A client that keeps a copy reads the log after the last entry it holds, verifies it, and requests a pack of the new head against the version it last received, which it receives under the pack rules. Reads beyond the log, packs, the manifest and files are a server’s own API (for underlay.org, the Versions API).

Errors

Errors carry {"error": <message>}. Authentication and 404 follow the reads’ rules: content the caller may not read is 404, never 403.

• 403: the caller may read the collection but not publish to it, or the session belongs to another user.
• 400: a malformed body, a metadata_patch that is not an object, a metadata that is neither an object nor null, or a request with no lines.
• 409: base is not the head (with currentVersion); the head changed before the commit; the session is not open; or the publication changes nothing (with the head’s hash).
• 413: a body over open_bytes or batch_bytes, a request over batch_lines, or a file over file_bytes.
• 422: records or deletes that fail, a refused schema, records carried over from the base that fail a changed schema (validationErrors), or a commit lacking files (filesNeeded).
• 429: open_sessions sessions already in progress, or a rate limit (with Retry-After).

Security considerations

• A server MUST NOT serve a tree node or body by hash alone, and MUST serve a record, schema or file located by hash only where it occurs in a set the caller may read.
• A server MUST NOT treat content it holds for other collections as present in a session: records are always uploaded in full, and a file counts only if it is held for the collection.
• The private-set salt prevents confirmation of guessed private content from the commitment; the root reveals only whether a private set exists.
• A version hash does not prove that a server shows every reader the same versions; the signed, hash-chained log makes omission and reordering detectable.
• Open issue: how a verifier obtains trusted signing keys is not specified. Trusting the keys in a collection.json served by an untrusted server admits a forged history on first contact.
• File bytes SHOULD be served from an origin separate from the server’s pages, with Content-Disposition: attachment, so that an uploaded HTML or SVG file cannot run in the server’s origin.