Files API

Files are content-addressed by SHA-256 hash. The same bytes always produce the same hash, so identical files are stored only once. Upload files before committing a version that references them.

Workflow

Compute the SHA-256 hash of your file locally
Open a push session that declares the file in files.add; its needed_files lists the declared files this collection doesn’t hold yet (see Versions)
Upload each needed file with PUT (up to 32 MiB) or a direct upload (larger)
Reference it in records as {"$file": "sha256:<hash>"}
Commit the version. The server checks that this collection holds every referenced file

Every collection uploads the bytes of its files itself, even when the server already has the same file for another collection. This is proof of possession: a hash alone never lets a collection use, or learn about, another collection’s file.

HEAD /api/collections/:owner/:slug/files/:hash

Access follows the collection’s visibility

Check whether you can read a file through this collection. Returns headers only, no body. For everyone, a file is readable here when it was in a public set of a published version. Members can also read every file the collection holds: the latest version’s private files, and any file uploaded to this collection and verified, which covers earlier versions’ private files and uploads not yet committed. To learn what to upload, use needed_files from opening a push session.

Parameters

:hash SHA-256 hash, optionally prefixed with sha256:

Response

`200`	The file is readable. `Content-Length` and `Content-Type` headers set.
`404`	Not found, not readable by you, or the collection is private. A file you can’t read is `404` even when it has been withheld.
`451`	The file is readable by you but has been withheld.

Example

curl -I https://www.underlay.org/api/collections/kf/archive/files/sha256:a1b2c3...
# HTTP/2 200
# Content-Length: 1048576
# Content-Type: application/pdf

GET /api/collections/:owner/:slug/files/:hash

Access follows the collection’s visibility

Download a file. After an access check in the context of this collection, the endpoint 302-redirects to a short-lived, presigned storage URL; follow the redirect (e.g. curl -L) to fetch the bytes. The same files are readable as for HEAD: public-version files anonymously, and for members (a session, a key, or a share/agent token sent as a Bearer header or ?token=) every file the collection holds. Inaccessible files return 404, withheld or not; a readable file that has been withheld returns 451.

The presigned URL lasts 300 seconds and downloads the file as an attachment. The redirect carries Cache-Control: private, max-age=240, so a browser may reuse it until shortly before the URL expires. This API path is the durable locator for the file; the redirect target is ephemeral and must not be persisted or shared, so always re-fetch through the API path. To resolve many files in one request, see the presign endpoint below.

Example

# -L follows the 302 redirect to the short-lived presigned URL
curl -L -o paper.pdf \
  https://www.underlay.org/api/collections/kf/archive/files/sha256:a1b2c3...

GET /api/collections/files/:hash

No auth for files in public collections

Download a file by hash alone. Redirects (302) to a presigned URL, like the download above, when the file is readable through any collection you can read: a public collection’s public files, or any file of a collection in an organization you belong to. Otherwise 404, withheld or not; a readable file that has been withheld returns 451.

POST /api/collections/:owner/:slug/files/presign

Access follows the collection’s visibility

Presign up to 500 files in one request (more is 400). The response maps each hash, exactly as sent, to a presigned URL (300 seconds), or to null when the hash is invalid, the file is not readable through this collection, or it has been withheld. Same access model as the single download; avoids one round-trip per file.

Request

{ "hashes": ["sha256:a1b2c3...", "sha256:f6e5d4..."] }

Response 200

{
  "sha256:a1b2c3...": "https://storage.example/...",
  "sha256:f6e5d4...": null
}

PUT /api/collections/:owner/:slug/files/:hash

Auth: write access

Upload a file of up to 32 MiB. The server verifies that the SHA-256 hash of the uploaded bytes matches the hash in the URL, then records that this collection holds the file. The bytes are always required, even when the server already has the file.

Request

Send the file as the raw request body with the appropriate Content-Type header, or as multipart/form-data with the file in a file field. HTML, XHTML, SVG and XML types are stored as application/octet-stream.

# Compute hash
HASH=$(shasum -a 256 paper.pdf | cut -d' ' -f1)

# Upload
curl -X PUT \
  "https://www.underlay.org/api/collections/kf/archive/files/sha256:$HASH" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/pdf" \
  --data-binary @paper.pdf

Response 201

{
  "hash": "a1b2c3d4e5f6...",
  "status": "stored",
  "size": 1048576
}

Errors

`400`	Not a SHA-256 hash; a multipart body with no `file` field; or a hash mismatch, where the uploaded bytes don’t match the hash in the URL: `{"error": "Hash mismatch", "expected": "..."}`.
`413`	Over 32 MiB. Use a direct upload (`POST .../files/uploads`).

POST /api/collections/:owner/:slug/files/uploads

Auth: write access

Start a direct upload to storage, for files over the 32 MiB the PUT above takes. A file is at most 5 TiB (larger: 413); a missing hash or size is 400.

Request

{
  "hash": "sha256:a1b2c3d4e5f6...",
  "size": 21474836480,
  "mimeType": "video/mp4"
}

Response 201

Up to 5 GiB, the ticket has one presigned url to PUT the bytes to, valid for an hour:

{
  "id": "uuid",
  "url": "https://storage.example/...",
  "expiresIn": 3600
}

Larger files are multipart. The ticket gives partBytes (every part but the last is that long: at least 100 MiB, more when needed to fit), partCount (at most 10,000) and presigned parts for the first 100 parts. Each part URL is valid for an hour.

{
  "id": "uuid",
  "partBytes": 104857600,
  "partCount": 205,
  "parts": [
    { "partNumber": 1, "url": "https://storage.example/..." },
    { "partNumber": 2, "url": "https://storage.example/..." }
  ],
  "expiresIn": 3600
}

GET /api/collections/:owner/:slug/files/uploads/:id/parts

Auth: write access

The next 100 presigned part URLs of a multipart upload, starting from part ?from=n (default 1). Only while the upload is pending; otherwise 404. Each part’s PUT returns an ETag; keep them for completing.

Response 200

{
  "parts": [
    { "partNumber": 101, "url": "https://storage.example/..." }
  ]
}

POST /api/collections/:owner/:slug/files/uploads/:id/complete

Auth: write access

Finish an upload. A multipart upload sends every part’s number and ETag; a single PUT upload sends no body. A multipart upload without a non-empty parts list is 400. The server then hashes the bytes in the background; poll GET .../files/uploads/:id until status is verified (or failed, with an error).

Request

{
  "parts": [
    { "partNumber": 1, "etag": "\"9b2cf535f27731c974343645a3985328\"" },
    { "partNumber": 2, "etag": "\"6f1ed002ab5595859014ebf0951522d9\"" }
  ]
}

Response 202

{ "id": "uuid", "status": "verifying" }

GET /api/collections/:owner/:slug/files/uploads/:id

Auth: write access

An upload’s state: pending (waiting for the bytes), verifying, verified (this collection now holds the file) or failed, with the reason in error.

Response 200

{
  "id": "uuid",
  "hash": "a1b2c3d4e5f6...",
  "size": 21474836480,
  "status": "verified",
  "error": null
}

File references in records

To link a file to a record, use the $file convention:

{
  "id": "pub-001",
  "type": "Publication",
  "data": {
    "title": "An Example Paper",
    "pdf": {"$file": "sha256:a1b2c3d4e5f6..."},
    "thumbnail": {"$file": "sha256:f6e5d4c3b2a1..."}
  }
}

A reference is any object, at any depth in a record’s data, whose $file is the string sha256: followed by 64 lowercase hex digits. Each referenced file must be held by this collection: uploaded to it (a PUT or a verified direct upload), or already in the version the push builds on (or, for public files, in any earlier version). A file that only another collection holds does not count. If any are missing, the commit returns 422 listing up to 100 of them:

{
  "error": "Missing files",
  "filesNeeded": ["a1b2c3d4e5f6..."],
  "statusCode": 422
}