Quickstart

Push your first version in 5 minutes. All you need is curl and a running Underlay instance.

Fastest path: use an AI agent

Point your coding agent at llms.txt and tell it what data you want to push. It has everything it needs to create a collection, write the push script, and batch the uploads for you. The steps below explain the same flow manually.

1. Sign in and create an API key

Sign in at www.underlay.org/login via KF Auth SSO. Your account is created automatically on first sign-in. Then go to Settings → API Keys and create a write-scoped key.

# Sign in via KF Auth SSO at https://www.underlay.org/login
# Your account is created automatically on first sign-in.
# Then create an API key at https://www.underlay.org/settings/keys

Save the key value. It's shown only once.

2. Create a collection

export KEY="ul_abc123..."

curl -X POST https://www.underlay.org/api/accounts/yourname/collections \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "slug": "my-dataset",
    "name": "My Dataset",
    "public": true
  }'

3. Push a version

A push is a delta push: open a session against a base version, upload the records that are new or changed and the ids to delete, then commit. The server builds the version and numbers it.

3a. Open a session

Send the schemas and metadata. The response includes the server’s limits, such as how many lines and bytes one upload may hold.

# Open a push session. base is null for the first version.
curl -X POST https://www.underlay.org/api/collections/yourname/my-dataset/push \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{
    "base": null,
    "message": "Initial import",
    "app_id": "my-app",
    "metadata": {
      "description": "A curated book list",
      "readme": "# My Dataset\nA collection of notable books."
    },
    "schemas": {
      "Book": {
        "type": "object",
        "properties": {
          "title": {"type": "string"},
          "author": {"type": "string"},
          "year": {"type": "integer"},
          "pdf": {"type": "object"}
        }
      }
    }
  }'
# → {"session_id":"SESSION_ID","base":null,"needed_files":[],"expires_at":"...","limits":{...}}

3b. Upload records

Send records as NDJSON (one JSON object per line). For large datasets, split them into batches within limits.batch_lines (10,000 on underlay.org) and limits.batch_bytes. Add "private": true to a line to keep that record out of public view.

# Upload records as NDJSON, one per line
curl -X POST https://www.underlay.org/api/collections/yourname/my-dataset/push/SESSION_ID/records \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: Bearer $KEY" \
  --data-binary @- << 'EOF'
{"id":"book-1","type":"Book","data":{"author":"Douglas Hofstadter","title":"Gödel, Escher, Bach","year":1979}}
{"id":"book-2","type":"Book","data":{"author":"Thomas Kuhn","title":"The Structure of Scientific Revolutions","year":1962}}
EOF
# → {"received":2}

3c. Commit

Large commits answer 202; poll GET .../push/SESSION_ID until its status is committed or failed.

curl -X POST https://www.underlay.org/api/collections/yourname/my-dataset/push/SESSION_ID/commit \
  -H "Authorization: Bearer $KEY"
# → {"semver":"v1.0.0","hash":"ulv2:...","recordCount":2,"fileCount":0,"changes":{...}}

4. Read it back

# Get collection info
curl https://www.underlay.org/api/collections/yourname/my-dataset

# Get the records of v1.0.0 (or /versions/latest/records)
curl https://www.underlay.org/api/collections/yourname/my-dataset/versions/v1.0.0/records

# Get the manifest (list of record hashes)
curl https://www.underlay.org/api/collections/yourname/my-dataset/versions/v1.0.0/manifest

5. Push an update

Set base to the current version and send only what changed. If someone else pushed in the meantime, opening the session answers 409 with currentVersion.

# Open a session against the current version. Schemas and metadata
# you leave out are kept.
curl -X POST https://www.underlay.org/api/collections/yourname/my-dataset/push \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{"base": "v1.0.0", "message": "Add a book, drop another"}'
# → {"session_id":"SESSION_ID","base":"v1.0.0",...}

# Upload the new or changed records
curl -X POST .../push/SESSION_ID/records \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: Bearer $KEY" \
  --data-binary '{"id":"book-3","type":"Book","data":{"author":"Ludwig Wittgenstein","title":"Philosophical Investigations","year":1953}}'

# Delete records by type and id
curl -X POST .../push/SESSION_ID/deletes \
  -H "Content-Type: application/x-ndjson" \
  -H "Authorization: Bearer $KEY" \
  --data-binary '{"type":"Book","id":"book-2"}'

curl -X POST .../push/SESSION_ID/commit -H "Authorization: Bearer $KEY"
# → {"semver":"v1.1.0","hash":"ulv2:...","recordCount":2,"fileCount":0,"changes":{...}}

If your app exports its whole dataset each time rather than tracking changes, read the current version’s manifest and diff against it first. See clients without a copy.

6. Diff versions

curl https://www.underlay.org/api/collections/yourname/my-dataset/versions/v1.1.0/diff?from=v1.0.0
# → {"from":"v1.0.0","to":"v1.1.0","added":[{"id":"book-3",...}],"updated":[],
#    "removed":[{"id":"book-2","type":"Book"}],"pagination":{...},"meta":{...}}
# Without ?from=, the diff is against the version before (here v1.0.0).

Record hashing

You don’t need to hash records to push them: the server hashes what you upload. You need the hash to compare your data with a version’s manifest. It is the SHA-256 of a fixed {id, type, data} envelope with data in canonical JSON (RFC 8785), so any client produces the same hash for the same data regardless of key insertion order.

// Record hashing: SHA-256 of the canonical form
//   '{"id":' + JSON(id) + ',"type":' + JSON(type) + ',"data":' + JCS(data) + '}'
// JCS is RFC 8785 canonical JSON: no whitespace, object keys sorted.
import { createHash } from 'node:crypto'

// Written out as a string: a sorted object passed to JSON.stringify
// would put integer-like keys ("9", "10") first.
function jcs(value) {
  if (value === null || typeof value !== 'object') return JSON.stringify(value)
  if (Array.isArray(value)) return '[' + value.map(jcs).join(',') + ']'
  const keys = Object.keys(value).sort()
  return '{' + keys.map((k) => JSON.stringify(k) + ':' + jcs(value[k])).join(',') + '}'
}

function hashRecord(record) {
  const canonical =
    '{"id":' + JSON.stringify(record.id) + ',"type":' + JSON.stringify(record.type) +
    ',"data":' + jcs(record.data) + '}'
  return createHash('sha256').update(canonical).digest('hex')
}

Working with files

To attach files (PDFs, images, etc.) to records, upload them first by hash:

# Compute hash
HASH=$(shasum -a 256 paper.pdf | cut -d' ' -f1)

# Upload
curl -X PUT "https://www.underlay.org/api/collections/yourname/my-dataset/files/sha256:$HASH" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/pdf" \
  --data-binary @paper.pdf

# Reference in a record
# {"id": "book-1", "type": "Book", "data": {"title": "...", "pdf": {"$file": "sha256:..."}}}
# A commit whose records reference a file this collection doesn't hold is refused (422, filesNeeded).
# The Book schema above lists "pdf", so the field is accepted.

Next steps