A free bucket on GitHub Releases

SunCVE is a dashboard for searching and analyzing CVEs. The interesting part is not the front end, it is the data problem behind it: more than 300k CVEs, from CVE-1999-0001 to the most recent record, enriched with repositories, advisories, PoCs, and package metadata. That turns into a SQLite database of about 130 MB.

And the project’s goal was always radical: run 100% in the browser, published on GitHub Pages, with no backend server to maintain. No managed database, no API, no bill at the end of the month.

The technical problem is straightforward: serve those 130 MB of data with no infrastructure of your own, using GitHub Releases as object storage. And, as a bonus, keep the git history clean.

The original problem: versioning data in git

The first version of the pipeline did the most obvious thing possible: every processed record was written to JSON, zipped, and committed to the repository. The front end downloaded those JSONs, decompressed them in the browser, and ran the queries right there, all local and fast (the SQLite that runs today came only later; the idea of processing everything on the client predates it). Each processing run generated new files, and they went into git alongside the code.

It worked for a few weeks. Then the problem showed up:

The practical effect was the worst possible for an open project: cloning the repository became slow and heavy. As a concrete example, git clone took forever and, at times, would even freeze GitHub’s own web interface while rendering the repository. Anyone who wanted to contribute had to pull all the historical data junk just to tweak one button in the UI. The barrier to collaborating went up, against the whole point of being open source.

Cleanup: rewriting the git history

The first step was accepting a simple thing, one that works as a general rule: if you are using git as a backup for data, for processed information, for generated files, you are probably doing it wrong and it is time to revisit your concepts, friend. A code versioning tool is not data storage.

So we pulled the data files out of version control, put the directories in .gitignore and, more importantly, rewrote the git history to remove the old blobs. Deleting the file at HEAD fixes nothing when the weight is in the past: you have to rewrite the git history for the repository to actually slim down.

With the cleanup done, the repository went back to being just code: processing scripts, UI components, configuration. The git log went back to recording only what matters, features and fixes, and not data dumps.

But that reopened the question we had been putting off: if the data does not live in git, where does it live?

The right question: where to store the data with no server?

The options on the table were the usual ones:

• A server with a database. It solves the problem, but it creates infrastructure to maintain, a monthly cost, and a single point of failure. Against everything the project wanted to be.
• An object storage bucket (S3, R2, etc). Cheaper, but still one more account, credentials to manage, and one more external dependency.
• No server at all. The project already ran directly in the browser, going back to the zipped-JSON days. With SQLite it only got more robust: the database ships whole to the client and the queries happen right there, via WebAssembly. If the data already traveled to the browser, why would we need a backend?

The third option was clearly the right one in spirit. All that was missing was a place to host the database file, a place that:

1. Was not git (so as not to dirty the git history again).
2. Was not new infrastructure to maintain.
3. Was free and already right there, next to the project.

First attempt: process everything on Actions

The initial idea was to move the whole pipeline into GitHub Actions. Actions would process the data and spit out the static files GitHub Pages needed, including the SQLite that runs in the browser. Elegant on paper: the same place that hosts the code also generates and publishes the data.

The problem showed up in the time budget. Processing from the first CVE in history to today, with all the enrichment (hitting the advisory database, verifying repositories, PoCs, npm and Packagist metadata), takes almost a week of execution.

The lesson is the same as always in data processing: you do not reprocess everything on every run. You process incrementally, from the last known point forward. Except that, to start from the last point, you need to have stored that point somewhere. And that brings us back to the storage question.

The insight: GitHub Releases as a bucket

The hypothesis was: what if GitHub Releases is that place? Releases accept arbitrary binary assets, of generous size, served over a stable URL, at no cost, and without touching the git history. That is, in practice, an object bucket glued to the repository.

The idea worked. The final design has five stages:

1. Local bootstrap. Most of the data is processed on the local machine, where there is no 6h ceiling. The final snapshot becomes a single file and is uploaded to the release.
2. Snapshot in the release. A release with the fixed tag db-snapshots holds the compressed SQLite as an asset. That is the bucket.
3. Daily incremental job. A schedule on Actions downloads the latest snapshot, processes only the new CVEs from there on, and uploads an updated snapshot.
4. Retention. We keep the last few snapshots as backup and delete the old ones, so the release does not grow without limit.
5. Deploy to Pages. The deploy pulls the most recent snapshot from the release, bakes it into the static build, and publishes to GitHub Pages.

The thing that ties it all together: the Actions job never starts from zero. It starts from the last snapshot it published itself. The work of a single run stops being “process 27 years of CVEs” and becomes “process what showed up since yesterday”. That fits comfortably within the runner’s limit.

One release, many snapshots

A design detail worth highlighting: we do not create a new release on every run. There is a single release, with the fixed tag db-snapshots, and the snapshots are assets inside it. Each asset’s name carries the CVE range and a timestamp:

snapshot-cve-1999-0001-to-cve-2026-28296-20260227T031447Z.tar.gz

To “read the bucket”, you just list the release’s assets and pick the most recent one by creation date. This primitive shows up identically everywhere in the project. In the local setup script, for instance, it is a gh call with a jq filter:

# scripts/setup-db.sh
LATEST_ASSET="$(gh release view "$RELEASE_TAG" \
  --repo "$REPO_SLUG" \
  --json assets \
  --jq '.assets
    | map(select(.name | test("^snapshot-.*\\.tar\\.gz$")))
    | sort_by(.createdAt)
    | reverse
    | .[0].name // ""')"

gh release download "$RELEASE_TAG" \
  --repo "$REPO_SLUG" \
  --pattern "$LATEST_ASSET" \
  --output "$TMP_FILE"

tar -xzf "$TMP_FILE" -C .

It filters assets matching snapshot-*.tar.gz, sorts by createdAt, reverses, and takes the first. It is the GET latest of our bucket, written in three lines of shell.

The local bootstrap

The initial load, the one that would take a week on Actions, runs on the local machine via scripts/manual-db-snapshot.sh. It reproduces the same pipeline as the workflow, but with no time ceiling, and publishes the result with plain gh commands:

# scripts/manual-db-snapshot.sh (excerpts)

# create the release-bucket if it does not exist yet
gh release create "$SNAPSHOT_TAG" \
  --repo "$REPO_SLUG" \
  --title "DB Snapshots" \
  --notes "Rolling snapshots for incremental CVE DB state"

# ... process the data ...

# upload the snapshot as an asset
gh release upload "$SNAPSHOT_TAG" "$SNAPSHOT_NAME" --repo "$REPO_SLUG"

Once the first heavy snapshot is up, Actions takes over the maintenance. A human only runs this again if a large reprocessing is needed, like a backfill that does not fit into the incremental flow.

The incremental job, step by step

The heart of the system is the db-snapshots.yml workflow. It fires every day on a schedule and can also be run by hand:

# .github/workflows/db-snapshots.yml
on:
  workflow_dispatch:
    # ... manual inputs ...
  schedule:
    - cron: "0 7 * * *"   # every day at 07:00 UTC

permissions:
  contents: write   # needed to write to the release
  actions: write    # needed to trigger the deploy at the end

The job’s first step makes sure the release exists (creating it on the first run) and finds out which asset is the most recent. Then comes the step that makes everything incremental, the restore of the latest snapshot:

- name: Restore Latest Snapshot
  if: steps.release.outputs.latest_asset_id != ''
  run: |
    mkdir -p data public/db
    curl -L \
      -H "Authorization: Bearer ${GITHUB_TOKEN}" \
      -H "Accept: application/octet-stream" \
      "https://api.github.com/repos/${{ github.repository }}/releases/assets/${{ steps.release.outputs.latest_asset_id }}" \
      -o /tmp/latest-snapshot.tar.gz
    tar -xzf /tmp/latest-snapshot.tar.gz -C .

A detail that closes the design off elegantly: the resume cursor lives inside the SQLite itself. The database has a sources table that stores, among other things, which was the last cvelistV5 delta release already processed. When the job restores the snapshot, it needs no external state to know where to continue: the resume point came inside the file. The snapshot is, at the same time, the data and the progress marker.

With yesterday’s SQLite restored, the pipeline runs the incremental update. Each step is an enrichment stage that only processes what was missing:

- name: Incremental CVE Update
  run: python scripts/create-manifest.py cves --year-auto

- name: Enrich from GitHub Advisory Database
  run: python scripts/create-manifest.py advisories

- name: Enrich POCs from PoC-in-GitHub
  run: python scripts/create-manifest.py pocs

- name: Incremental Repository Verification
  run: python scripts/create-manifest.py repos --batch-size "${REPO_BATCH_SIZE}"

# ... npm, packagist, wordpress, osv, fix recalculation ...

At the end, it rebuilds the artifacts the browser consumes (build-db-artifacts.sh generates the compressed SQLite and a manifest.json), and before publishing it goes through a sanity gate that avoids uploading a broken database:

- name: Validate Snapshot DB Size
  env:
    MIN_CVE_COUNT: "1000"
  run: |
    total=$(sqlite3 data/source.sqlite "SELECT COUNT(*) FROM cves;")
    if [ "${total}" -lt "${MIN_CVE_COUNT}" ]; then
      echo "[ERROR] Snapshot aborted: only ${total} CVEs. Likely stale/incomplete DB state."
      exit 1
    fi

This guard matters when the storage is written by automation: a network failure in the middle of processing could produce an almost empty database, and without this check it would overwrite the good snapshot. The rule is simple: a suspicious snapshot does not become the new state.

Anatomy of a snapshot

What goes to the release is not the raw SQLite. The Pack Snapshot step bundles two files:

tar -czf "${SNAPSHOT_NAME}" \
  public/db/manifest.json \
  public/db/source_com_repositorios.sqlite.gz

The source_com_repositorios.sqlite.gz is the database compressed with gzip (level 9), and the manifest.json is a small index with metadata: version, gzip URL, size, sha256 for integrity checking, and the CVE range that snapshot covers. The manifest is what the front end reads first to know what to download and how to validate it.

Retention: the last few as backup

A bucket with no retention policy becomes a dump. Since each run uploads a new snapshot, without cleanup the release would grow forever. The final maintenance step keeps only the last few and deletes the rest:

- name: Prune Old Snapshots (Keep 3)
  uses: actions/github-script@v7
  with:
    script: |
      const keep = 3;
      const snapshots = (release.data.assets || [])
        .filter(a => /^snapshot-.*\.tar\.gz$/.test(a.name))
        .sort((a, b) => new Date(b.created_at) - new Date(a.created_at));

      for (const asset of snapshots.slice(keep)) {
        await github.rest.repos.deleteReleaseAsset({ owner, repo, asset_id: asset.id });
      }

Keeping the last three gives you something a plain bucket does not hand you for free: versioned backup built in. If a run produces bad data that slips past the sanity gate, the previous snapshots are still there for a manual rollback. It is storage and backup in the same structure.

From the release to the browser

Having the data in the release is half the story. The other half is how it reaches the user without ever passing through an application server.

The deploy pulls the snapshot

When the incremental job finishes, it triggers the deploy workflow. The deploy makes the same “read the bucket” move: it finds the most recent snapshot and restores it into the build:

# .github/workflows/deploy.yml (excerpt)
- name: Restore DB Files from Release Snapshot
  run: |
    curl -L \
      -H "Authorization: Bearer ${GITHUB_TOKEN}" \
      -H "Accept: application/octet-stream" \
      "https://api.github.com/repos/${{ github.repository }}/releases/assets/${{ steps.snapshot.outputs.asset_id }}" \
      -o /tmp/latest-db-snapshot.tar.gz
    tar -xzf /tmp/latest-db-snapshot.tar.gz -C .
    test -f public/db/manifest.json
    test -f public/db/source_com_repositorios.sqlite.gz

Then Next runs in static export mode, with the compressed SQLite already inside public/db/, and the result is uploaded to GitHub Pages. From Pages’ point of view, the database is just one more static file served alongside the HTML and the JS. Before building, there is yet another sanity check that decompresses the database and counts the CVEs, so a Pages build with broken data never gets published.

SQLite runs in the browser

On the client, there is no API to query. The browser downloads the whole database and runs SQL locally via sql.js (SQLite compiled to WebAssembly). The flow is:

1. Read the manifest.json to find out the gzip URL, the size, and the sha256.
2. Download the .sqlite.gz and decompress it in streaming, in the browser itself, with the native DecompressionStream API.
3. Load the resulting SQLite into sql.js and run the queries straight in the client’s memory.

Streaming decompression avoids having to load the entire compressed file into memory before decompressing:

// src/lib/sqlite/sqlite-loader.ts (excerpt)
if (encoding && encoding !== 'identity') {
  stream = stream.pipeThrough(
    new DecompressionStream(encoding as CompressionFormat)
  );
}

The result is stored in the browser’s local storage (OPFS) so it does not re-download everything on every visit. From there, searching, filtering, and aggregating CVEs is all local, with no network round-trip. The user is, in practice, running a database of 300k records inside their own tab.

What we got out of it

Adding it all up, a design with no server at all delivers the entire project:

• Zero infra and zero cost. No server, no managed database, no paid bucket. Everything lives on GitHub, next to the code.
• Clean git. The git history goes back to being only features and fixes. Cloning the repository is light again, and the barrier to contributing drops.
• Processing that fits the runner. The incremental flow starts from the last snapshot, so no run hits the Actions time limit.
• Backup built in. Snapshot retention gives you rollback for free, something a raw bucket does not offer.
• Runs in the browser. The database travels to the client and the queries happen there, via WebAssembly. Nothing to scale on the server side, because there is no server.

The conceptual trick is seeing GitHub Releases as a versioned object bucket, with one read primitive (“grab the most recent asset”) and one write primitive (“upload a new asset and delete the old ones”). Two simple operations, plus a sanity gate, and you have large-scale data storage glued to your code repository.

Caveats: how far this approach goes

It is not a silver bullet. It is worth knowing where the model pinches:

For SunCVE the fit is perfect: CVE data is a large dataset that only grows, daily updates are more than enough, and the client always wanted the database local anyway. Outside that profile, a real bucket or a database is still the right choice.

Closing

The constraint, “I do not want to maintain a server”, is what forced the good design. With no place to dump the data, we were forced to look at what was already right there: GitHub Releases, which nobody thinks of as storage, but which plays the bucket role very well when access is by snapshot.

In the end, three decisions carry the project: taking the data out of git so the git history can breathe, processing incrementally from the last snapshot to fit the runner, and storing that snapshot on Releases to close the loop with no infrastructure. The result is a database of 300k CVEs that anyone opens in the browser, served by a repository that remains, essentially, just code.

SunCVE GitHub Actions SQLite sql.js serverless architecture