Self-hosted dataset exposure through the collection API

[clementbiron]

Context

The engine periodically generates a dataset of a collection: a ZIP archive of all versions of all tracked terms. In production it runs on a weekly schedule (dataset.publishingSchedule) that generates and publishes the archive in one step.

Distribution is push-based: depending on the credentials configured on the tracking machine, ota dataset --publish uploads the archive to a Git forge release (GitHub or GitLab) and to data.gouv.fr. These are the dataset's sole distribution channel: the local archive is deleted after upload.

Problem statement

A collection's dataset cannot be retrieved from the instance that produces it; its distribution depends entirely on external platforms. This weakens the decentralised architecture: each instance is the authoritative source of its versions and snapshots, but not of its own dataset, which exists only as a copy hosted by a third party. And with no platform credentials configured, the dataset cannot be distributed at all.

This RFC proposes to reverse the flow, from push to pull: each instance stores its latest dataset locally and exposes it, with its metadata, through the existing collection API, becoming the authoritative source of its own dataset.

The existing publications to GitHub, GitLab and data.gouv.fr are kept for backward compatibility and catalogue visibility, but become downstream consumers of this API.

Out of scope:

• Dataset history: only the latest dataset is exposed. History remains available through the third-party releases for collections that publish there.
• On-the-fly generation: the API serves the latest pre-generated archive and never triggers a generation itself.

Proposed solution

Local storage

The generated dataset is kept on the instance in a configurable directory (dataset.storagePath, defaulting to ./data/datasets, alongside the existing ./data/versions and ./data/snapshots). The directory contains the archive under its dated name and a metadata.json sidecar file written at generation time:

data/datasets/
├── demo-2026-07-06.zip
└── metadata.json

{
  "filename": "demo-2026-07-06.zip",
  "title": "demo",
  "license": "ODbL-1.0",
  "releaseDate": "2026-07-06T08:30:12Z",
  "firstVersionDate": "2022-01-01T12:00:00Z",
  "lastVersionDate": "2026-07-05T23:12:00Z",
  "servicesCount": 262,
  "termsCount": 641,
  "versionsCount": 38294,
  "size": 104857600,
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}

Field	Type	Description
filename	string	Name of the archive file
title	string	Dataset title, from the dataset.title configuration
license	string	SPDX identifier of the dataset license
releaseDate	ISO 8601 string	Datetime when the dataset was generated
firstVersionDate	ISO 8601 string	Fetch date of the earliest version in the dataset
lastVersionDate	ISO 8601 string	Fetch date of the latest version in the dataset
totalServices servicesCount	number	Number of services in the dataset
totalTerms termsCount	number	Number of distinct terms (service and terms type pairs) in the dataset
totalVersions versionsCount	number	Total number of version files in the dataset
size	number	Size of the archive in bytes
sha256	string	SHA-256 checksum of the archive, allowing consumers to verify integrity and detect changes without downloading

The counter names follow the vocabulary already used by the dataset generator (servicesCount), and avoid reusing the totalServices / totalTerms names of GET /metadata: those are computed live on the current state of the collection, while these are frozen at generation time.

Only the latest dataset is kept; each generation replaces the previous one. The filename is a human-friendly label, not an identifier (two datasets generated on the same day share it), so consumers should detect changes through releaseDate and sha256.

Generation writes atomically: the archive is written to a temporary file, then renamed into place on the same volume, so a concurrent read never sees a truncated archive. metadata.json follows the same rule and is written after the archive, so it never references a file that does not exist yet.

Collection API extension

The existing collection API is extended with the following endpoints, using the existing naming conventions (singular for a specific resource, as /service/:id a latest selector on a singular resource, as /version/{serviceId}/{termsType}/latest):

Method	Endpoint	Description
GET	/dataset /dataset/latest	Returns the metadata of the latest dataset
GET	/dataset/download /dataset/latest/download	Returns the latest dataset archive

Both endpoints return 404 Not Found when no dataset has been generated yet on the instance. GET /dataset alone is not defined and returns 404 like any other unknown route; the /dataset/:date space is left free should dataset history ever be exposed.

GET /dataset /dataset/latest

Returns the content of metadata.json, enriched with an absolute downloadURL built from the request (as the Atom feed endpoints already do for their links):

GET /collection-api/v1/dataset/latest

HTTP 200
Content-Type: application/json
- - -
{
  "filename": "demo-2026-07-06.zip",
  "title": "demo",
  "license": "ODbL-1.0",
  "releaseDate": "2026-07-06T08:30:12Z",
  "firstVersionDate": "2022-01-01T12:00:00Z",
  "lastVersionDate": "2026-07-05T23:12:00Z",
  "servicesCount": 262,
  "termsCount": 641,
  "versionsCount": 38294,
  "size": 104857600,
  "sha256": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "downloadURL": "http://162.19.74.224/collection-api/v1/dataset/latest/download"
}

When no dataset exists:

HTTP 404
- - -
{ "error": "No dataset has been generated yet" }

This endpoint lets any consumer check whether a dataset is available and whether it changed (via releaseDate or sha256) without downloading it. It is also what lets the existing publication targets become consumers of the API rather than relying on a local push, the broader goal of this change.

GET /dataset/download /dataset/latest/download

Streams the archive:

GET /collection-api/v1/dataset/latest/download

HTTP 200
Content-Type: application/zip
Content-Disposition: attachment; filename="demo-2026-07-06.zip"
Content-Length: 104857600
Last-Modified: Mon, 06 Jul 2026 08:30:12 GMT
ETag: "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
- - -
<binary content>

The endpoint supports HTTP caching and resumable downloads: a client that already holds the current archive receives a 304 Not Modified (through If-Modified-Since or If-None-Match) instead of transferring it again, and an interrupted download can resume where it stopped (through Range requests) rather than restarting.

Rejected solutions

Single endpoint with content negotiation

A single GET /dataset endpoint could return either the metadata or the archive depending on the request Accept header (application/json for the metadata, application/zip for the archive).

Not retained: metadata and archive are two different resources, a few hundred bytes of JSON versus a multi-gigabyte file. Behind one URL, the archive cannot be linked to directly (a browser or plain curl gets whichever representation the Accept default selects), caching needs Vary: Accept on a huge resource, and one OpenAPI operation ends up with two unrelated payloads. Two URLs keep each resource independently linkable, cacheable and documented.

Single endpoint returning the archive, with metadata in HTTP headers

A single GET /dataset endpoint could return the archive directly, with the metadata in response headers (Last-Modified, ETag, and custom X- headers for the counts). A HEAD request would then read the metadata without transferring the body.

Not retained: structured metadata (counts, date range, filename) belongs in a documented JSON body, not in bespoke X- headers that carry no schema or types and have no place in the OpenAPI model. Headers are for transfer metadata, not domain data.

Plural `/datasets/latest` naming

The endpoints could be named /datasets/latest and /datasets/latest/download, a plural collection with a latest selector, as the versions API already does with /version/{serviceId}/{termsType}/latest.

This is a valid shape; the choice is a naming decision, not a technical constraint. Not retained: the plural advertises a collection the instance does not expose (only the latest dataset exists locally), and raises questions with no answer here, such as what GET /datasets returns. The singular /dataset states exactly what is available, one dataset, and mirrors /metadata. The /datasets namespace stays free should history ever be exposed.

Following review (see @Ndpnt's comment), the bare singular /dataset initially proposed was replaced by the singular with an explicit latest selector, /dataset/latest: unlike the /metadata singleton, a dataset is dated, and the explicit selector keeps /dataset/:date free for later.

Feedback expectations

We invite you to provide feedback to:

• Point out any limitations or edge cases you see in the proposed solution.
• Suggest improvements or refinements to the current proposal.
• Share any alternative approaches you believe could address the problem more effectively.

If you support the proposal as it stands, please react with a 👍 or leave a positive comment. This helps us confirm the RFC has been read and is appreciated 🙂.

Please provide your feedback by July 29th.

[Ndpnt]

Thanks @clementbiron for this detailed RFC. Here are my remarks:

• Route naming

I agree with you about the plural: /datasets would suggest a collection that the instance will not really expose at first. So I am not asking for the plural.

But I think the /metadata comparison is not the right one. /metadata is a true singleton, there is only one collection metadata, and it has no history. A dataset is different, it has a date, and older datasets exist (today on third-party platforms, but the RFC itself says this is "for now"). So /dataset mixes two things, the resource (the dataset) and the time selection (which one).

I think the versions resource has a better model which already exists in the API.

• GET /version/:serviceId/:termsType/latest
• GET /version/:serviceId/:termsType/:date

Here latest is a selector on the singular resource, not a plural collection (there is no /versions/latest). We can do the same for datasets, keep the singular /dataset, but make the time selection explicit.

• GET /dataset/latest (metadata) + GET /dataset/latest/download
• this keeps GET /dataset/:date (+ /download) free for later, with no conflict

You asked "what would GET /datasets return?". If we ever add it, the natural answer would be the list of datasets the instance can serve (the available :date entries, only one today). But this is optional and can come later.

• Counters

The /dataset response has totalServices / totalTerms. But /metadata already returns totalServices / totalTerms, computed live. These are not the same thing, the dataset numbers are frozen at generation time, the /metadata numbers show the current state. Same names, different values, so this can confuse users. A few options:

• clear suffix: totalServicesAtRelease / totalTermsAtRelease (releaseDate is already there, so it reads well)
• group them: statistics: { services, terms, versions }, clearly linked to the dataset
• follow the generator, which already returns servicesCount (scripts/dataset/export); reusing servicesCount / termsCount / versionsCount would keep the names consistent everywhere

Any option is fine. The goal is just to avoid the same names as the live counters.

• Never serve a half-written file

The versions repository uses git, so commits are atomic and a reader always sees a complete state. A dataset is different as it is one big file, written in place. If the API reads the file while a generation is rewriting it, it can serve a truncated archive. It would be good to say that generation must write atomically. Write to a temporary file, then rename it on the same volume. This way the served path always points to a complete file.

[clementbiron]

Thanks @Ndpnt for this thorough review. I agree with all three points.

Route naming

You're right, the /metadata comparison doesn't hold. /dataset/latest keeps the singular, matches the existing latest selector of the versions API, and leaves /dataset/:date free for later. Adopted: GET /dataset/latest and GET /dataset/latest/download. GET /dataset alone stays undefined and returns 404.

Counters

Confirmed: /metadata already returns totalServices / totalTerms computed live, so reusing the same names for values frozen at generation time would be confusing. I'll go with your third option, servicesCount / termsCount / versionsCount, since the generator already uses servicesCount internally and in the dataset README. It's the option that introduces the least new vocabulary.

Atomic writes

Good catch, the generator currently writes the archive in place (createWriteStream(targetPath)), so a concurrent read could indeed serve a truncated file. The RFC will state that generation must write to a temporary file and rename it on the same volume. The same rule applies to metadata.json, written after the archive so it never references a file that does not exist yet.

I'll update the RFC accordingly.