The server

foldkit-media/server answers the route Provider.hosted writes, for originals no CDN can fetch. MediaApi is its Effect HttpApi, one route; the handler decodes the path with the space's Schema, the one the provider encodes with. A URL no release could write (a leading zero, a malformed hash) is a 400; one well formed but outside this space (a width off the ladder) is a 404, since another release, the next one in a rollout, may offer it.

It needs three services, each a Layer you choose per host:

A request is served from Variants when it can; otherwise the original is read, transformed, kept, and served. A failure is its own response, never the original in the variant's place, naming its kind ({ "error": "NotFound" }) and nothing of the server's internals.

Every answer's headers come from one function, headersFor:

Every image response also says X-Content-Type-Options: nosniff, Access-Control-Allow-Origin: * (a canvas may read it), Cross-Origin-Resource-Policy: cross-origin and Timing-Allow-Origin: *, and a variant says where it came from in Server-Timing (edge, store or made). Nothing sends Vary: the format is in the URL.

• serveFetch({ space?, maxBytes?, layer: env => Layer }) is a Worker's fetch handler. The services are built from the request's bindings, once per environment. It 404s any other path, so route /_image/ to it.
• layer(options) is the route as an Effect HttpRouter layer, for a Node server beside other routes.
• variantsNone keeps nothing, for a host with a cache of its own in front.

Stores#

Originals and Variants are kept in any Effect KeyValueStore (effect/persistence), the store the application already has:

const stores = Layer.mergeAll(originalsKeyValueStore(), variantsKeyValueStore()).pipe(
  Layer.provide(KeyValueStore.layerFileSystem('./media')),
  Layer.provide(NodeServices.layer),
)

• KeyValueStore.layerMemory for tests, layerFileSystem on a disk (it needs the platform's FileSystem and Path, Node's on a server), layerSql over SQLite or D1, keyValueStoreR2 (below) over R2.
• variantsKeyValueStore({ prefix }) keeps a variant's bytes under variants/<hash>/<spelling>; its media type is the key's extension, so nothing else is stored. A store that fails a read or a write is logged and the request is answered by making the variant: variants are a cache.
• originalsKeyValueStore({ prefix }) reads an original under its hash. A store that fails is a 500, not a 404: the original is the data.

Rate limits#

Only a miss costs work, so only a miss is limited: Admission is asked after the store and before the original is read, and a variant already made is served to anyone. It is optional; without it every miss is admitted.

• admissionRateLimiter({ limit, window, algorithm? }) is Effect's RateLimiter (effect/persistence): limit misses a client per window, over RateLimiter.layerStoreMemory in one process or Redis across several.
• admissionRateLimit(binding) (foldkit-media/cloudflare) is a Workers rate-limit binding ([[ratelimits]]), whose limit and period are its configuration.
• A refused client gets a 429 with Retry-After. A limiter that fails admits the request, logged: a limiter that is down should not take the images down.
• The client is CF-Connecting-IP, else the connection's address. Pass clientOf to read your own proxy's header; X-Forwarded-For is the client's to write, so it is not read by default.

On a Node server#

foldkit-media/sharp gives the Transformer (sharp is an optional peer):

const media = layer().pipe(
  Layer.provide(
    Layer.mergeAll(
      transformerSharp({ concurrency: 4, maxPixels: 50_000_000 }),
      originalsKeyValueStore(),
      variantsKeyValueStore(),
    ),
  ),
  Layer.provide(KeyValueStore.layerFileSystem('./media')),
  Layer.provide(NodeServices.layer),
)

• transformerSharp reads an original's size from its header before decoding it, and refuses one above maxPixels (50 megapixels by default) with a 413, so a small file that inflates to a huge picture never reaches the heap. It never enlarges, and encodes at the URL's quality.
• At most concurrency transforms run at once (the machine's cores by default), through limitTransforms(n), which wraps any Transformer.
• layer coalesces: concurrent misses for one variant share one transform and one write (coalesce, on by default here and off for serveFetch, since a Worker request cannot wait on another's).

Facts#

An ImageAsset's facts are derived from its original once, where it enters, and stored beside it:

• describe(key, bytes) reads them through the Inspector: size, alpha, the SHA-256, and a ThumbHash where the inspector makes a thumbnail.
• inspectorSharp({ maxPixels? }) (Node) reads all of them; alpha is whether any pixel is transparent, so an opaque PNG still gets a placeholder.
• inspectorBrowser({ maxPixels? }) (foldkit-media/browser) reads them in the browser or a web worker, with its own decoder: an editor describes a file a reader dropped, ThumbHash and hash included, before anything is uploaded, and stores its facts beside the image (a rich-text Image's asset). What the browser cannot decode is refused.
• inspectorImages(binding) (Cloudflare) reads the size from info, which costs no transformation. It makes no thumbnail, since the binding's raw pixels are not offered offline, and takes every format but JPEG to be transparent; both cost a placeholder, never show a wrong one.
• factsOf(url, { allow, maxBytes?, cache?, fetch? }) describes an image a page names only by URL (a rich-text image), so it gets its box like any other. Only https on a host in allow (matched exactly) is read, each redirect is checked again, and the body is limited as it streams. Given a KeyValueStore as cache, the facts are kept by URL and the original is asked for again with its ETag, so an unchanged one is not read twice.

Uploads#

An original that no CDN or build holds, such as an editor's upload, enters through POST /_media/upload: the bytes are the body, with an image/* Content-Type. The route reads at most maxBytes (20 MB by default) as the body streams, reads its facts through the Inspector, keeps the bytes under their SHA-256 in Uploads (where Originals reads them), and answers the facts as an ImageAsset, keyed by that hash, with a 201. The variant server serves it at once. Only the facts go on to the application's data (a CMS's asset row); the bytes never travel through RPC.

Who may upload is the application's to say, and the route has no default: UploadPolicy is asked before a byte is read.

const editors = Layer.succeed(UploadPolicy, {
  admit: request =>
    request.headers['authorization'] === `Bearer ${token}`
      ? Effect.void
      : Effect.fail(new UploadForbidden({})),
})

const uploads = uploadLayer({ maxBytes: 10 * 1024 * 1024 }).pipe(
  Layer.provide(Layer.mergeAll(uploadsKeyValueStore(), inspectorSharp(), editors)),
)

uploadsKeyValueStore() keeps them where originalsKeyValueStore() reads, over the same KeyValueStore; on Cloudflare, uploadsR2(bucket) beside originalsR2(bucket), and serveUploadFetch({ layer }) hosts the route in a Worker. A refusal answers its kind and nothing more, never kept by a cache: 403 from the policy, 415 for a body that is not image/* or not an image the inspector reads, 413 past maxBytes or the inspector's pixel limit, and 400 for a body that broke off. Because the body must be image/*, which is not a type a browser sends to another origin without a preflight, and the route answers no preflight, a cookie alone cannot upload from another site.

Written out at build#

A static site needs no server for its hosted images: materialize makes every variant the built pages name, through the server's own handler, and returns them as files at their own paths, for foldkit-ssr's staticSite emit:

const media = serveFetch<object>({
  layer: () =>
    Layer.mergeAll(transformerSharp(), originalsKeyValueStore(), variantsKeyValueStore()).pipe(
      Layer.provide(KeyValueStore.layerFileSystem('.media')),
      // The disk itself: Node's file system and paths.
      Layer.provide(NodeServices.layer),
    ),
})
const env = {}

export const site = {
  // …
  emit: pages =>
    materialize({
      pages,
      fetch: request => media(request, env),
      headersFile: readFileSync('public/_headers', 'utf8'),
    }),
}

• A variant that cannot be made fails the build, naming each one: a page that names it would show a broken image.
• More than maxFiles (20,000 by default, Cloudflare Pages' free limit) fails the build.
• A static host sends what its _headers says. Add headersRule() (what the server sends a variant) to public/_headers; given that file's text as headersFile, the build fails without it, since Pages would otherwise serve every variant with max-age=0.
• With Variants on disk, a second build writes without encoding again.

On Cloudflare#

foldkit-media/cloudflare gives the three services from a Worker's bindings:

const answerImages = serveFetch<WorkerEnv>({
  layer: env =>
    Layer.mergeAll(
      originalsR2(env.MEDIA),
      transformerImages(env.IMAGES),
      variantsR2(env.MEDIA, { cache: caches.default }),
    ),
})

• transformerImages(images) is the Images binding: it scales down, never up, and names the format. Cloudflare's code for a spent allowance (9422) is the 503; any other refusal is the 502, its reason logged.
• keyValueStoreR2(bucket) is an R2 bucket as a KeyValueStore, for the store adapters above or the application's own (clear and size list the bucket).
• variantsR2(bucket, { prefix?, cache? }) is variantsKeyValueStore over it, so a variant made last month is not made, and billed, again; with cache, the Cache API is read first and filled from R2.
• originalsR2(bucket) streams an original rather than reading it into memory, unlike the general adapter.

Cloudflare bills each new width of each original once a month (format and quality do not count), so the ladder is the cost: eight widths are at most eight transformations an original. examples/cloudflare routes /_image/ to this server.