Sync and integrations API

Endpoints for copying collections elsewhere and connecting them to other systems: tree sync for mirrors and clients, whole-version exports, webhooks, ARK identifiers, storage you bring yourself, and a couple of service endpoints. Paths below that start …/:owner/:slug are under /api/collections. Where a version is asked for, it is a semver (v1.2.0), a version hash (ulv2:…) or latest.

Tree sync

How a mirror or client keeps a full copy of a collection’s repository: read the signed version log, then fetch each new version as a pack of only the objects it doesn’t already have. What a receiver must check is specified in the protocol’s version log and packs sections. A collection the caller can’t read is a 404.

GET …/:owner/:slug/log?after=&limit=

No auth for public collections

The collection’s collection.json (including the public keys that sign its log), its current head (null before the first version), and the log entries with seq greater than after (default 0), oldest first. limit defaults to and is capped at 1,000; to read further, call again with after set to the last seq you received.

{
  "collection": {
    "id": "uuid",
    "owner": "kf",
    "slug": "archive",
    "name": "PubPub Archive",
    "keys": [{ "id": "k1", "alg": "Ed25519", "publicKey": "base64url..." }]
  },
  "head": { "seq": 12, "entryHash": "a1b2...", "versionHash": "ulv2:e5f6..." },
  "entries": [
    {
      "collectionId": "uuid",
      "seq": 1,
      "semver": "v1.0.0",
      "versionHash": "ulv2:0c1d...",
      "baseSemver": null,
      "message": "First import",
      "appId": null,
      "actorId": "uuid",
      "createdAt": "2026-01-15T00:00:00.000Z",
      "prev": null,
      "keyId": "k1",
      "sig": "base64url..."
    }
  ]
}

GET …/:owner/:slug/versions/:n/pack?base=&sets=

No auth for public collections; sets=all needs membership

A pack of version :n: an uncompressed tar (application/x-tar) of the repository objects it reaches that version base does not, each under its repository key. Without base, the pack holds everything. sets is public (the default) or all, which adds the private set and needs membership in the owning organization (403 otherwise). base must be a version of the same collection (404 otherwise). The response headers x-underlay-version, x-underlay-base (empty without a base) and x-underlay-sets say what was packed. This request counts as 10 requests against your rate limit.

curl -o v3.2.0.tar \
  "https://www.underlay.org/api/collections/kf/archive/versions/v3.2.0/pack?base=v3.1.0"
# x-underlay-version: ulv2:e5f6...
# x-underlay-base: ulv2:9a8b...
# x-underlay-sets: public

Export

GET …/:owner/:slug/export?version=&format=

No auth for public collections

A whole version as one archive, of what the caller can read: non-members get the public set only. version defaults to latest; format is tar.gz (the default) or tar. The archive is named <owner>-<slug>-<semver>.tar.gz in Content-Disposition. This request counts as 20 requests against your rate limit.

Entries, in this order:

`manifest.json`	Always first: the collection, the version, its schemas, and file notes
`README.md`	The version’s `metadata.readme`, when it has one
`records/<Type>.ndjson`	One file per type, one `{"id", "type", "data", "hash"}` per line
`files/<hash>`	The bytes of each file, named by its SHA-256 hash

curl -o archive.tar.gz \
  "https://www.underlay.org/api/collections/kf/archive/export?version=v3.2.0"

tar -tzf archive.tar.gz
# manifest.json
# README.md
# records/Author.ndjson
# records/Publication.ndjson
# files/a1b2c3d4e5f6...

manifest.json: files_missing lists files the version references whose bytes the platform doesn’t hold, and files_withheld files that have been blocked; neither is in files/. Blocked records are left out of the NDJSON. totalBytes is the records’ canonical size plus the files’ size.

{
  "collection": {
    "owner": "kf",
    "slug": "archive",
    "name": "PubPub Archive",
    "description": "Full archive of PubPub publications"
  },
  "version": {
    "semver": "v3.2.0",
    "hash": "ulv2:e5f6...",
    "message": "April sync",
    "recordCount": 4521,
    "fileCount": 892,
    "totalBytes": 1073741824,
    "createdAt": "2026-04-01T00:00:00.000Z"
  },
  "schemas": { "Author": { "type": "object" }, "Publication": { "type": "object" } },
  "files_missing": [],
  "files_withheld": []
}

The archive streams as it is built, with no limit on the number of records; for very large collections, format=tar is quicker to produce. Entries are stamped with the time the version was created, so exporting the same version again, with the same access, gives a byte-identical archive. If something fails partway, the response is cut off rather than completed, so treat an archive that doesn’t end cleanly as failed.

Webhooks

A webhook sends a signed POST to your URL each time a version of the collection is published. Managing webhooks takes an owner or admin of the owning organization, signed in or with an admin API key; write and read keys, and keys confined to specific collections whatever their scope, get 403.

GET …/:owner/:slug/webhooks

Auth: org owner or admin

{"webhooks": [{id, url, bumpFilter, enabled, createdAt, lastDeliveryAt}]}, newest first. The secret is never listed.

POST …/:owner/:slug/webhooks

Auth: org owner or admin

Body: {"url", "bumpFilter"?, "enabled"?}. url must be https and may not name a local or private address (422). bumpFilter is a non-empty list of major, minor, patch, the kinds of version that trigger a delivery (default: all three). enabled defaults to true. The 201 response is the only time the signing secret is shown; store it.

curl -X POST https://www.underlay.org/api/collections/kf/archive/webhooks \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.org/hooks/underlay", "bumpFilter": ["major", "minor"]}'

{
  "id": "uuid",
  "url": "https://example.org/hooks/underlay",
  "bumpFilter": ["major", "minor"],
  "enabled": true,
  "createdAt": "2026-04-01T00:00:00.000Z",
  "lastDeliveryAt": null,
  "secret": "ulwhsec_3f9a..."
}

PATCH …/:owner/:slug/webhooks/:id

Auth: org owner or admin

Change any of url, bumpFilter and enabled, checked as on create. Returns the updated webhook (without the secret).

DELETE …/:owner/:slug/webhooks/:id

Auth: org owner or admin

Returns {"ok": true}.

POST …/:owner/:slug/webhooks/:id/test

Auth: org owner or admin

Queue a signed test delivery, returning {"ok": true, "deliveryId"}. Its event is ping, its version is null and it carries "test": true. It is logged with the other deliveries.

GET …/:owner/:slug/webhooks/:id/deliveries?limit=

Auth: org owner or admin

Recent deliveries, newest first (limit default 50, max 200). status is pending, success or failed. Deliveries are kept for 30 days.

{
  "deliveries": [
    {
      "id": "uuid",
      "event": "version.created",
      "semver": "v3.2.0",
      "bumpType": "minor",
      "status": "success",
      "attempts": 1,
      "responseCode": 200,
      "error": null,
      "durationMs": 184,
      "createdAt": "2026-04-01T00:00:01.000Z",
      "deliveredAt": "2026-04-01T00:00:02.000Z"
    }
  ]
}

POST …/:owner/:slug/webhooks/:id/deliveries/:deliveryId/retry

Auth: org owner or admin

Send a delivery again, with its attempts reset. Returns {"ok": true, "status": "pending"}.

Webhook deliveries

Each delivery is a JSON POST to the webhook’s URL. Any 2xx response within 10 seconds is a success; redirects are not followed. A failed delivery is retried up to 5 attempts in all, waiting 1 minute and doubling each time (at most 6 hours between attempts). Deliveries to a disabled webhook fail without being sent.

POST /hooks/underlay HTTP/1.1
Content-Type: application/json
User-Agent: Underlay-Webhook/2.0
X-Underlay-Event: version.created
X-Underlay-Delivery: 0b6f...
X-Underlay-Signature: sha256=5d41402abc4b2a76b9719d911017c592...

{
  "event": "version.created",
  "collection": { "owner": "kf", "slug": "archive" },
  "version": {
    "semver": "v3.2.0",
    "hash": "ulv2:e5f6...",
    "major": 3,
    "minor": 2,
    "patch": 0,
    "recordCount": 4521,
    "fileCount": 892
  },
  "bumpType": "minor",
  "delivery": { "id": "0b6f...", "timestamp": "2026-04-01T00:00:01.000Z" }
}

X-Underlay-Delivery is the same for every attempt of one delivery, so you can use it to ignore repeats. X-Underlay-Signature is sha256= followed by the hex HMAC-SHA256 of the raw request body, keyed with the webhook’s secret. Check it against the body bytes as received, before parsing:

import { createHmac, timingSafeEqual } from 'node:crypto'

// rawBody: the request body exactly as received (a Buffer or string), before JSON.parse.
function verifyUnderlaySignature(secret, rawBody, header) {
  const expected = 'sha256=' + createHmac('sha256', secret).update(rawBody).digest('hex')
  const a = Buffer.from(expected)
  const b = Buffer.from(header ?? '')
  return a.length === b.length && timingSafeEqual(a, b)
}

// e.g. in an Express handler with express.raw({ type: 'application/json' }):
// if (!verifyUnderlaySignature(SECRET, req.body, req.get('x-underlay-signature')))
//   return res.sendStatus(401)

ARK identifiers

A collection can have an ARK, a persistent identifier that redirects to the collection, to one of its versions, or to a URL a record names. Resolution sees what the caller can read, as everywhere else.

https://www.underlay.org/ark:<NAAN>/<shoulder><id><check>[.vX.Y.Z][/<Type>/<id>]

https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0                    the collection
https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0             a version
https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0/Publication/pub-1  a record

GET /ark:<NAAN>/<name>

No auth required

302 to the target: the collection or version page, or the collection’s custom URL when it has one. A record ARK redirects to the URL in the record field set for its type (below), and is a 404 when none is set, the record isn’t readable, or the field isn’t an http(s) URL. Append ?info or ?? for an Electronic Resource Citation (ERC) as plain text, or ?json for the metadata as JSON. /ark:<NAAN>/ alone returns the NAAN’s policy statement.

GET /api/ark/resolve?path=

No auth required

Resolve an ARK without following it. path is anything containing ark:<NAAN>/… (400 otherwise). Returns {"type": "redirect", "url", "metadata"}, or 404 with {"type": "not_found"}. A relative url is a page on this site.

{
  "type": "redirect",
  "url": "/kf/archive/v/3.2.0",
  "metadata": {
    "type": "version",
    "who": "Knowledge Futures",
    "what": "PubPub Archive v3.2.0",
    "when": "20260401",
    "where": "https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0",
    "naan": "12345",
    "collectionName": "PubPub Archive",
    "ownerName": "Knowledge Futures",
    "semver": "v3.2.0",
    "message": "April sync",
    "appId": null,
    "createdAt": "2026-04-01T00:00:00.000Z",
    "arkUrl": "https://www.underlay.org/ark:12345/ulb9bq4n5gmv3k0.v3.2.0"
  }
}

GET | PATCH …/:owner/:slug/ark

Auth: org member (PATCH: signed in, or a write or admin key)

GET returns {"enabled", "customUrl", "arkUrl", "shoulder", "arkId"}. PATCH takes {"enabled"?, "customUrl"?}: customUrl is an http(s) URL to redirect the collection and version ARKs to, or null to use the Underlay pages. The first PATCH mints the collection’s ARK. Returns {"ok": true}.

GET | PUT | PATCH | DELETE …/:owner/:slug/ark/record-types

Auth: org member (changes: signed in, or a write or admin key)

Which record field a type’s record ARKs redirect to. GET returns [{"recordType", "redirectUrlField"}]. PUT or PATCH {"recordType", "redirectUrlField"} sets one; PATCH with "redirectUrlField": null, or DELETE …/record-types/:recordType, removes it. Changes return {"ok": true}.

PATCH /api/accounts/:slug/ark

Auth: org owner or admin, signed in or with an admin key

Set the NAAN the organization’s ARKs are minted under: {"naan": "…"} (digits, at most 16), or null for the default. A NAAN another organization already uses is a 409. Read and write keys, and keys confined to specific collections, get 403.

Storage locations and mirrors

An organization can add its own S3-compatible buckets as locations and mirror collections to them: Underlay keeps a copy of each version’s repository there. The credentials must be able to both write and read the bucket. Managing locations and mirrors takes an owner or admin of the organization, signed in or with an admin API key not confined to specific collections. Removing a location or mirror never deletes what was copied to the bucket.

GET | POST /api/orgs/:org/locations

Auth: org owner or admin

GET returns {"locations": [...]} (credentials are never returned). POST adds one:

{
  "name": "Library mirror",
  "endpoint": "https://s3.us-east-1.amazonaws.com",
  "bucket": "kf-underlay-mirror",
  "prefix": "underlay",
  "accessKeyId": "AKIA...",
  "secretAccessKey": "..."
}

endpoint is an https address with no path; prefix is optional. Before keeping the location, Underlay writes a check object, reads it back, tests whether the bucket serves objects without credentials, and looks for lifecycle rules that would delete mirrored objects. If the check fails the location is not kept and the response is 422; otherwise 201 {"location", "check"}.

POST /api/orgs/:org/locations/:id/check

Auth: org owner or admin

Run the check again and update the location’s status. Returns {"location", "check": {"ok", "publicRead", "readBack", "error", "warnings"}}.

DELETE /api/orgs/:org/locations/:id

Auth: org owner or admin

Remove a location and every mirror to it. Returns 204.

GET | POST | DELETE /api/orgs/:org/placements[/:id]

Auth: org owner or admin

Organization defaults: every collection of the organization, existing and new, is mirrored to each default location. POST {"locationId", "sets"}, where sets is public (the default) or public+private. Private sets need an https endpoint and a bucket that doesn’t serve objects without credentials (422). Deleting a default also removes the organization’s mirrors to that location.

GET …/:owner/:slug/placements

Auth: org member

Where the collection is stored: the primary and each mirror, with its state (active, backfilling, lagging, error, paused), how many versions it is behind (lag), and whether it comes from an organization default (inherited).

{
  "headSeq": 12,
  "placements": [
    {
      "id": "uuid",
      "role": "mirror",
      "sets": "public",
      "inherited": false,
      "state": "active",
      "syncedSeq": 12,
      "lag": 0,
      "lastError": null,
      "updatedAt": "2026-04-01T00:00:00.000Z",
      "location": {
        "id": "uuid",
        "name": "Library mirror",
        "kind": "s3",
        "bucket": "kf-underlay-mirror",
        "prefix": "underlay",
        "status": "active"
      }
    }
  ]
}

POST …/:owner/:slug/placements

Auth: org owner or admin

Mirror this collection to a location: {"locationId", "sets"}, as for organization defaults. The mirror starts in backfilling. Returns 201.

POST …/:owner/:slug/placements/:id/sync

Auth: org owner or admin

Start a mirror catching up again, for example after fixing a broken location. Returns 202.

DELETE …/:owner/:slug/placements/:id

Auth: org owner or admin

Stop mirroring the collection there. Returns 204, or 409 for a mirror that comes from an organization default (remove the default instead).

Health

GET /api/health

No auth required

Whether the service is up. deployment names the deployment answering (for example production or staging).

{
  "ok": true,
  "version": 2,
  "deployment": "production",
  "time": "2026-04-01T00:00:00.000Z"
}

Abuse reports

POST /api/abuse-reports

No auth required

Report content that shouldn’t be served. Body: {"hash"?, "url"?, "reason", "contact"?}. Name the content with a file or record hash (64 hex characters, sha256: prefix optional), a url, or both. reason is required (up to 4,000 characters); contact is how to reach you (up to 320). Returns 201 {"ok": true, "id"}.

{
  "hash": "sha256:a1b2c3d4e5f6...",
  "reason": "This file republishes copyrighted material.",
  "contact": "rights@example.org"
}