Records and schemas

Canonical JSON

Every hashed JSON document MUST be serialized as RFC 8785 (JCS): no insignificant whitespace; strings escaped as by JSON.stringify; numbers serialized as by ECMAScript Number.prototype.toString, with -0 as 0; object members ordered by the UTF-16 code units of their keys.

Note: ECMAScript objects enumerate integer-like keys ("9", "10") first regardless of insertion order. Sorting keys into a new object and calling JSON.stringify does not produce JCS; an implementation MUST emit object members as strings itself.

Input rules

A client MUST apply these rules to every record line it publishes, and a server to every record line it receives in a publication. They apply to the whole line, including members that are otherwise ignored, and are evaluated on the source text, since a parsed value no longer carries the information they require. Schemas are subject to the duplicate key, unsafe integer and lone surrogate rules.

A line that breaks several rules is reported with the code of the first in this order: the first violation in text order of an unterminated string or invalid escape (syntax), duplicate_key, unsafe_integer, lone_surrogate or too_deep, whether or not the text is otherwise JSON; syntax for any other text that is not JSON; bad_envelope for a non-object; bad_id; bad_type; bad_envelope for absent data or a non-boolean private; record_too_large. The codes are part of the protocol.

Members of a record line other than id, type, data and private MUST be ignored. Strings are not Unicode-normalized: é as U+00E9 and as U+0065 U+0301 are distinct ids.

Records

A record is an id (string), a type (type slug) and data (any JSON value). It is published as one NDJSON line; an optional "private": true assigns it to the private set and is not part of the record.

{"id":"pub-001","type":"Publication","data":{"title":"Notes","pdf":{"$file":"sha256:9f86d0…"}}}
{"id":"pub-002","type":"Publication","data":{"title":"Draft"},"private":true}

The canonical form is a fixed envelope, id, type, data in that order, with only data canonicalized:

'{"id":' + JCS(id) + ',"type":' + JCS(type) + ',"data":' + JCS(data) + '}'

record hash = SHA-256(canonical form)     64 lowercase hex characters
record size = length of the canonical form in bytes

Within a version, a (type, id) pair MUST identify at most one record, across both access sets.

Schemas

A type’s schema is a JSON Schema draft-07 object. Schema hash = SHA-256 of JCS(schema). A schema MUST be rejected if:

• its root private is present and not a boolean;
• a schema that is the value of a member of a properties object, at any depth, has "private": true (field-level privacy is not supported);
• its canonical form exceeds 262,144 bytes;
• any pattern or patternProperties key exceeds 256 UTF-16 code units. A pattern inside const, enum, default or examples is data and is not limited;
• the slug it is given under is not a valid type slug.

A root "private": true makes the type private.

Validation

A schema MUST be rejected unless:

• its root $schema, if present, is http://json-schema.org/draft-07/schema, with or without a trailing #;
• it is valid against the draft-07 meta-schema;
• every pattern compiles as an ECMAScript regular expression with the u flag;
• every $ref resolves within the schema or to the draft-07 meta-schema, against the base URI https://schema.underlay.invalid/ unless a $id sets another.

Records are validated under draft-07 with these refinements:

• keywords adjacent to $ref apply, as in draft 2019-09;
• keywords draft-07 does not define are ignored (unevaluatedProperties, prefixItems, draft-04 id, …); $defs and $anchor are honoured;
• multipleOf m accepts x when r = x mod m satisfies |r| < 1.1920929 × 10⁻⁷ or |m − r| < 1.1920929 × 10⁻⁷;
• string length is measured in Unicode code points;
• format constrains strings only, for the formats ajv-formats 3.0 defines in full mode (date-time and time require a time zone); other format names are ignored.

Only the verdict is normative; error messages are not.

Files

File hash = SHA-256 of the file’s bytes. A record references a file through any object, at any depth of data, whose $file member is a string sha256: followed by 64 lowercase hex characters. A reference object is not searched further; an object whose $file has any other value is not a reference and is searched as usual.