A 150,000 image library on one GPU

The object library is a set of isolated subjects on white backgrounds, generated locally and served as WebP. At the time of writing the corpus defines 158,592 unique entries and the renderer is grinding through them at roughly one every eleven seconds. That is a job measured in weeks, running on the same GPU that serves interactive customer work, on a machine that will be rebooted, redeployed and interrupted many times before it finishes.

Everything below follows from that: it has to yield, it has to resume, and it has to tolerate the corpus changing underneath it.

Background priority is a scheduling problem, not a nice-to-have

The obvious approach — run the generator when the machine looks idle — does not survive contact with reality, because "idle" is a lagging indicator. By the time you have measured idleness and dispatched a job, a customer request has arrived and is now queued behind an eight-second denoise.

The right place to solve this is the inference server, which knows what is actually running. Ours admits work in tiers, and the background tier is only admitted when nothing else is running or waiting. The generator simply asks for that tier:

{
  "prompt": "...",
  "width": 1024, "height": 1024,
  "num_inference_steps": 9,
  "low_priority": true
}

The client-side half of the contract is just as important: when the server says it is saturated, the generator does not retry immediately and it does not back off exponentially into uselessness. It holds for ninety seconds and tries again, up to three times, then records the failure and moves on. A generator that hammers a busy GPU is worse than one that stops.

A deterministic corpus beats a queue

The naive design is a work queue: enumerate every prompt, write them to a table, pop them off. It works, but it makes the prompt set a database migration. Adding a category means writing rows; changing a phrase means updating them; and the queue becomes the source of truth for something that should live in code and be reviewable in a diff.

Instead the corpus is a pure function. Categories, subjects, finishes and presentations are Go literals, and entry i is derived arithmetically:

n := len(subjectIndex)
ref := subjectIndex[idx%n]
rest := idx / n
presentation := rest % presentationsPerKind
treatment := (rest / presentationsPerKind) % treatmentsPerKind

The subject is the fastest-varying digit on purpose. If the treatment varied fastest you would generate a thousand variations of a moka pot before ever reaching a chair, and any partial library would be useless. With subject first, index order sweeps every subject in the corpus before repeating one — so the first pass is one honest rendition of every single thing, and the second pass starts the variations. A library that is two percent complete is still a broad, browsable sample.

The migration that mattered: key on slug, not index

The first version recorded progress by corpus index. It worked perfectly right up until we added fifteen categories, at which point len(subjectIndex) changed, every index remapped to a different entry, and 1,291 finished images were suddenly anchored to the wrong rows.

The fix is to key stored work on something that survives corpus edits. Each entry has a stable slug derived from its category, subject, finish and presentation — tools-a-cordless-power-drill-t00p0 — and that became the primary key. The index is demoted to an ordinary column used only for browse ordering, and on every start-up the generator re-anchors it:

current, ok := IndexOf(slug)
switch {
case !ok:
    removals = append(removals, slug)   // no longer in the corpus
case current != idx:
    updates = append(updates, change{slug, current})
}

Rows whose slug has left the corpus are dropped from the index but their files are deliberately not deleted. A mistaken corpus edit should cost you a database row, not hours of GPU time. Orphaned files are cheap; regenerating them is not.

Two processes will find every race you left in

We started a second generator by accident — a shell backgrounding subtlety, the kind of mistake that takes four seconds to make. Both processes walked the same corpus in the same order, wrote to the same temporary paths, and each one's atomic rename pulled the file out from under the other's stat. Twenty-two spurious failures before anyone noticed, all with a confusing error message about a file that plainly existed.

The failure was ours twice over: once for starting two, and once for building something that could be started twice. Both generators now take an exclusive advisory lock on their library directory at start-up and refuse to run if they cannot get it:

if err := syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
    return nil, fmt.Errorf("another objectgen is already writing %s", root)
}

The lock is released by the kernel when the process exits, however it exits — which is the property you want for something that will be killed, OOMed and rebooted many times over a multi-week run.

Storage decisions that compound

WebP quality 85, re-encoded

The inference server returns WebP at quality 90. Re-encoding to 85 is a lossy-to-lossy step, which is normally a bad idea, but at this scale it buys about twenty percent on every file and the difference is invisible on a product shot against white. At 158,592 images that is the difference between roughly seven gigabytes and roughly nine. More importantly it makes the whole library one consistent format, so the browse grid has predictable decode cost.

Sharded directories

Files live at data/object-library/<index/1000>/<slug>.webp. A thousand files per directory is small enough for ls to stay usable and for the filesystem to keep directory lookups cheap. Content-addressed sharding by hash would spread more evenly, but index sharding has a property hashing does not: the directory listing is in corpus order, so you can see at a glance which part of the run produced what.

Atomic writes, always

Write to target.tmp, then rename. Rename is atomic within a filesystem, so a killed process leaves either a complete file or no file — never a truncated one that the browse grid will try to decode and fail on three weeks later.

Guard against the failure modes that look like success

A saturated GPU occasionally returns a valid, well-formed, completely blank image. It has the right dimensions, decodes cleanly, and is useless. The cheapest reliable detector we found is file size: a real 1024px WebP of a subject on white is tens of kilobytes; a blank one is under two. So anything under 2 KB is rejected, the file is removed and the entry is marked failed for retry.

This is a deliberately crude check. A smarter one — variance, edge density, a CLIP score against the prompt — would catch more, but every one of those costs GPU or CPU time per image and adds a component that can itself be wrong. At three failures in 1,300 images, the crude check is doing its job.

What it actually costs

Three weeks is the honest number and it is worth stating plainly rather than discovering later. It also reframes the design: at that timescale, throughput optimisations matter far less than never losing work. A change that makes generation ten percent faster saves two days. A crash that loses the index costs everything.

The part that makes it worth doing

A library only earns its keep if it is reachable. Every image gets its own page with the exact prompt that produced it, structured data, and links to its nearest neighbours in the same category; every category gets a landing page; and both are fed into chunked sitemaps that grow as the run progresses. The browse grid is the fast path for a person, and the item pages are the durable path for everyone and everything else.

Start at the library, or go straight to a category — tools, kitchen & dining, furniture, instruments, creatures.

Related reading

• Why a white background makes better 3D
• Choosing AI 3D models by representation
• From mesh to manufactured object