Media / foldkit-media

Running the resizing server

foldkit-media/server serves the /_image/ route that Provider.hosted writes, for originals no CDN can fetch. MediaApi defines the Effect HttpApi endpoint; the handler decodes each path with the space's Schema. URLs outside the grammar (leading zeros, malformed hashes) return 400; well-formed URLs outside the configured space return 404. Variants cache for a year; error responses cache for one minute at the edge and are marked no-store for browsers so rolling deploys recover quickly.

Variant URLs carry unguessable SHA-256 hashes, though every page rendering an image publishes its hash; keep sensitive files out of public variant routes.

The server needs three Effect services, each a Layer you choose per host:

ServiceJobKind
OriginalsOriginal bytes by content hashAuthoritative; read-only
TransformerResize and encode without enlargingStateless compute
VariantsCached variants by encoded requestRebuildable cache

A request is served from Variants when cached; otherwise the original is read, transformed, stored, and served. Failures return structured error JSON ({ "error": "NotFound" }) without server internals.

Every response gets its headers from headersFor. Unhandled defects (such as failing stores or missing Worker layers) produce a framework 500 response.

AnswerStatusCache-Control
Variant (ETag: "<hash>/<spelling>")200public, max-age=31536000, immutable
Conditional hit (If-None-Match)304Same as 200; skips reading bytes
Malformed URL400no-store (CDN-Cache-Control: public, max-age=60)
Width/format outside space404Same as 400; next release may add it
Retired revision not in store302Redirect to current revision; 60s CDN cache
Missing original404Same as 400; upload may arrive later
Original over maxBytes (20 MB)413Same as 400
Transformer failure502no-store
Rate limited429no-store, Retry-After
Quota spent503no-store, Retry-After: 3600

Every image response also sends X-Content-Type-Options: nosniff, Access-Control-Allow-Origin: *, Cross-Origin-Resource-Policy: cross-origin, Timing-Allow-Origin: *, and Server-Timing (edge, store, or made). Because the format is encoded in the URL path, Vary is left unset.

  • serveFetch({ images?, uploads?, otherwise? }) builds a Worker fetch handler owning /_image/ and /_media/. Unmatched subpaths and OPTIONS preflights under those prefixes return 404; other paths route to otherwise (404 by default).
  • layer({ provider?, maxBytes?, coalesce? }) mounts the route as an Effect HttpRouter layer for Node servers.
  • variantsNone skips variant caching for deployments fronted by an external cache.

Stores

Originals and Variants read from any Effect KeyValueStore (effect/persistence):

ts
import { NodeServices } from '@effect/platform-node'
import { Layer } from 'effect'
import { KeyValueStore } from 'effect/persistence'
import { originalsKeyValueStore, variantsKeyValueStore } from 'foldkit-media/server'

const stores = Layer.mergeAll(originalsKeyValueStore(), variantsKeyValueStore()).pipe(
  Layer.provide(KeyValueStore.layerFileSystem('./media')),
  Layer.provide(NodeServices.layer),
)
  • KeyValueStore.layerMemory for tests, layerFileSystem on disk (requiring FileSystem and Path), layerSql over SQLite or D1, or keyValueStoreR2 over R2.
  • variantsKeyValueStore({ prefix }) stores variants under variants/<hash>/<spelling>, deriving media type from the key extension. Read or write errors emit a StoreFailed diagnostic and fall back to transforming from the original.
  • originalsKeyValueStore({ prefix }) reads originals by content hash. Store failures return 500 rather than 404 so missing storage fails loudly.

Rate limits

Only cache misses trigger transforms, so Admission runs after a cache miss and before reading the original. Cached variants serve all clients without counting against rate limits:

  • admissionRateLimiter({ limit, window, algorithm? }) wraps Effect's RateLimiter (effect/persistence): limit misses per client per window, backed by memory or Redis.
  • admissionRateLimit(binding) (foldkit-media/cloudflare) wraps a Cloudflare Workers rate-limit binding ([[ratelimits]]).
  • Exceeding the limit returns 429 with Retry-After. Limiter failures emit AdmissionFailed and admit the request so a down limiter keeps images online.
  • Clients are identified by CF-Connecting-IP, then connection address. Pass clientOf to read another proxy header (X-Forwarded-For is ignored by default since clients can spoof it).

On a Node server

foldkit-media/sharp provides the Transformer (with sharp as an optional peer):

ts
import { NodeServices } from '@effect/platform-node'
import { Layer } from 'effect'
import { KeyValueStore } from 'effect/persistence'
import { layer, originalsKeyValueStore, variantsKeyValueStore } from 'foldkit-media/server'
import { transformerSharp } from 'foldkit-media/sharp'

const media = layer().pipe(
  Layer.provide(
    Layer.mergeAll(
      transformerSharp({ concurrency: 4, maxPixels: 50_000_000 }),
      originalsKeyValueStore(),
      variantsKeyValueStore(),
    ),
  ),
  Layer.provide(KeyValueStore.layerFileSystem('./media')),
  Layer.provide(NodeServices.layer),
)
  • transformerSharp and inspectorSharp normalize EXIF orientation so portrait phone photos are measured, resized, and placeheld upright.
  • transformerSharp checks header dimensions before decoding and rejects images above maxPixels (50 megapixels by default) with 413 before allocating pixel buffers. It caps output width at the original's width and encodes at the URL's quality.
  • limitTransforms(n) caps concurrent transforms on any Transformer (defaults to CPU core count).
  • layer coalesces concurrent misses for the same variant into one transform and store write (coalesce defaults to true on Node and false in serveFetch, where Worker requests run in separate request contexts).

Uploads

Uploaded images enter through POST /_media/upload as raw image/* request bodies. The route streams up to maxBytes (20 MB by default), extracts facts with Inspector, stores the bytes under their SHA-256 hash alongside StoredFacts, and returns a 201 JSON HashedAsset. The variant server can serve that hash immediately.

Authorization is required. UploadPolicy.admit(request, route) runs before reading any body bytes, receiving 'Upload' or 'DescribeUrl'. It fails with UploadForbidden (403) or UploadUnauthenticated (401) when identity is missing so clients know to refresh a session.

ts
import { Layer } from 'effect'
import { GuardMedia } from 'foldkit-guard/media'
import {
  uploadLayer,
  uploadPolicyAny,
  uploadPolicyBearer,
  uploadsKeyValueStore,
} from 'foldkit-media/server'
import { inspectorSharp } from 'foldkit-media/sharp'

const uploads = uploadLayer({ maxBytes: 10 * 1024 * 1024 }).pipe(
  Layer.provide(
    Layer.mergeAll(
      uploadsKeyValueStore(),
      inspectorSharp(),
      uploadPolicyAny(
        uploadPolicyBearer(process.env['UPLOAD_TOKEN']),
        GuardMedia.uploadPolicy(Policy, { resolve, Upload: [G.isEditor] }),
      ),
    ),
  ),
)
  • uploadPolicyBearer(secret) accepts Authorization: Bearer <secret> using constant-time digest comparison and rejects requests while the secret is unset or empty.
  • GuardMedia.uploadPolicy (foldkit-guard/media) authorizes each route using your guard set's principal guards (G, Policy) and session resolve (such as AuthServer.resolver from foldkit-auth/server). Routes omitted from the map reject all requests; here editors can upload while URL fact lookups stay closed.
  • uploadPolicyAny(a, b, …) tries policies in order, admitting on the first match or returning the last refusal.
  • Custom policies use Layer.succeed(UploadPolicy, { admit }).

Registering an upload. Mutations receive only the submitted hash and look up server-recorded facts via StoredFacts.get(hash) (backed by storedFactsKeyValueStore() or storedFactsR2(bucket)), returning none if no upload recorded that hash. Client-submitted editor metadata such as a focal point travels alongside the hash.

uploadsKeyValueStore() and uploadsR2(bucket) share backing stores with originals, and serveFetch's uploads mounts the routes in a Worker. Refusals return minimal status codes with no-store (401/403 from policy, 415 for unrecognized image bodies, 413 above byte/pixel limits, 400 for truncated streams). Requiring image/* without CORS OPTIONS prevents cross-origin form or cookie uploads from other sites.

In the browser, upload(file, { url?, headers? }) (foldkit-media/browser) returns a Stream emitting Progressed { loaded, total } and then Uploaded { asset }. Failures emit UploadFailed { reason, status } (Forbidden, Unauthenticated, TooLarge, Unsupported, Malformed, Network, or Unexpected). It uses XMLHttpRequest for upload progress and cancels when the Effect stream is interrupted:

ts
import { Effect, Match, Stream } from 'effect'
import { upload } from 'foldkit-media/browser'

const send = (file: File) =>
  upload(file).pipe(
    Stream.runForEach(event =>
      Effect.sync(() =>
        Match.valueTags(event, {
          Progressed: ({ loaded, total }) => console.log(`${loaded} of ${total} bytes`),
          Uploaded: ({ asset }) => console.log('kept as', asset.key),
        }),
      ),
    ),
  )

Same-origin browser uploads send the page's session cookie automatically; reserve bearer tokens for deploy scripts so secrets stay off client pages.

Remote image URLs get their facts from GET /_media/facts?url=, protected by the same UploadPolicy with 'DescribeUrl'. Configure allowed hosts in uploadLayer({ facts: { allow } }) (empty by default). Unreachable or invalid images return 422.

Files

Forms often need arbitrary attachments (PDFs, spreadsheets). fileLayer adds a separate route set at POST /_media/files and GET /_media/files/:key, storing bytes in FileStore under random 32-hex keys distinct from the 64-hex hashes used by /_image/.

Ownership. The upload route owns the bytes and recorded metadata (key, name, type, size). Your application owns which database row holds a FileAsset and who may download it:

  1. The browser runs uploadFile(file) to POST /_media/files, receiving a minted FileAsset.
  2. Your form submits the FileAsset; your server mutation authorizes the write and calls keepOrClaim to bind the file to the row.
  3. GET /_media/files/:key checks FileReadPolicy for the requester and streams the file as a download attachment.

Each upload receives a fresh 128-bit random key and stores its own copy so duplicate uploads reveal nothing about existing files.

ts
import { Effect, Layer, Option } from 'effect'
import { SecureRandom } from 'effect/crypto'
import { KeyValueStore } from 'effect/persistence'
import type { FileAsset } from 'foldkit-media'
import {
  FileUploadPolicy,
  UploadForbidden,
  fileLayer,
  fileReadPolicy,
  filesKeyValueStore,
  keepOrClaim,
} from 'foldkit-media/server'

// Your own: who a request is, and which rows hold which file.
declare const userOf: (headers: Headers) => Option.Option<string>
declare const rowHolding: (key: string) => Effect.Effect<Option.Option<{ readonly owner: string }>>

const files = fileLayer({ accept: ['application/pdf', 'image/*'] }).pipe(
  Layer.provide(
    Layer.mergeAll(
      filesKeyValueStore(),
      Layer.succeed(FileUploadPolicy, {
        admit: request =>
          Option.match(userOf(new Headers(request.headers)), {
            onNone: () => Effect.fail(new UploadForbidden({})),
            onSome: Effect.succeed,
          }),
      }),
      fileReadPolicy({
        resolve: request => Effect.succeed(userOf(new Headers(request.headers))),
        read: (user, key) =>
          rowHolding(key).pipe(
            Effect.map(row =>
              Option.exists(row, ({ owner }) => Option.contains(user, owner)),
            ),
          ),
      }),
    ),
  ),
  Layer.provide(Layer.merge(KeyValueStore.layerMemory, SecureRandom.layerWebCrypto)),
)

// In the mutation that saves a row, once the write is authorized:
const attached = (held: Option.Option<FileAsset>, submitted: FileAsset, user: string) =>
  keepOrClaim(held, submitted, user)
  • FileUploadPolicy.admit(request) resolves the uploader's stable owner ID before reading bytes. Blank IDs fail with 403 and emit OwnerMissing.
  • fileReadPolicy({ resolve, read }) authenticates the reader first (returning 401 FileUnauthenticated when unknown), then checks read(user, key). Refused reads return 404 identically to missing keys so key existence stays hidden.
  • keepOrClaim(held, submitted, owner) keeps an unchanged key in place when re-saving a row, or claims a newly uploaded key from FileStore within claimWithin (24 hours by default, failing FileUnclaimed for expired, claimed, or foreign keys). Call it after authorizing the mutation so refused writes leave keys unclaimed.

Downloads execute safely. Responses use application/octet-stream, X-Content-Type-Options: nosniff, Content-Security-Policy: sandbox; default-src 'none', Cross-Origin-Resource-Policy: same-origin, Cache-Control: private, no-store, and an attachment Content-Disposition filename from the stored record.

Upload validation. Bodies arrive as application/octet-stream with headers x-foldkit-file-name (percent-encoded UTF-8, up to 255 bytes, free of control chars or slashes) and x-foldkit-file-type (parsed media type matched against required accept, or '*'). Because clients declare the type, inspect file contents directly with inspect: (bytes, declared) => Effect<void, FileRefused> when you need magic-number checks.

AnswerWhy
201Kept; body is the FileAsset
400Invalid/missing name or truncated body
401 / 403Unauthenticated or policy rejected
408Slower than minBytesPerSecond (32 KB/s over 10s)
413Exceeded maxBytes (20 MB)
415Wrong content type, outside accept, or inspect failed
429Admission rate limit reached (Retry-After)
503Concurrency slots full (maxConcurrent: 2)

Memory. Uploads hold roughly twice their size in memory while streaming (80 MB peak at defaults, fitting inside a 128 MB Worker). Downloads also buffer in memory, so serve high-traffic large files via signed R2 URLs instead.

In the browser, uploadFile(file, { url?, headers? }) (foldkit-media/browser) emits Progressed and FileUploaded { asset }, adding TooSlow, RateLimited, and Busy failure reasons. Mount in a Worker via serveFetch's files option.

Written out at build

Static sites without an image server can emit every referenced variant during build using materialize inside foldkit-ssr's staticSite emit:

ts
import { NodeServices } from '@effect/platform-node'
import { readFileSync } from 'node:fs'
import { Layer } from 'effect'
import { KeyValueStore } from 'effect/persistence'
import {
  materialize,
  originalsKeyValueStore,
  serveFetch,
  variantsKeyValueStore,
} from 'foldkit-media/server'
import { transformerSharp } from 'foldkit-media/sharp'
import type { BuiltPage } from 'foldkit-ssr/vite'

const staticMedia = serveFetch<object>({
  images: {
    layer: () =>
      Layer.mergeAll(transformerSharp(), originalsKeyValueStore(), variantsKeyValueStore()).pipe(
        Layer.provide(KeyValueStore.layerFileSystem('.media')),
        // The disk itself: Node's file system and paths.
        Layer.provide(NodeServices.layer),
      ),
  },
})
const env = {}

export const site = {
  // …
  emit: (pages: ReadonlyArray<BuiltPage>) =>
    materialize({
      pages,
      fetch: request => staticMedia(request, env),
      headersFile: readFileSync('public/_headers', 'utf8'),
    }),
}
  • Missing or broken variants fail the build by name.
  • Exceeding maxFiles (20,000 by default) fails the build.
  • Include headersRule() in public/_headers so static hosts serve variants with immutable caching; materialize verifies headersFile contains it.
  • Keeping Variants on disk lets repeat builds skip re-encoding.

On Cloudflare

foldkit-media/cloudflare wires R2, Cloudflare Images, and Workers rate limiting into serveFetch:

ts
import type { ImagesBinding, R2Bucket, RateLimit } from '@cloudflare/workers-types'
import { Layer } from 'effect'
import {
  admissionRateLimit,
  inspectorImages,
  originalsR2,
  transformerImages,
  uploadsR2,
  variantsR2,
} from 'foldkit-media/cloudflare'
import { serveFetch, uploadPolicyBearer } from 'foldkit-media/server'

interface WorkerEnv {
  readonly MEDIA: R2Bucket
  readonly IMAGES: ImagesBinding
  readonly LIMITER: RateLimit
  readonly UPLOAD_TOKEN: string
}

export default {
  fetch: serveFetch<WorkerEnv>({
    images: {
      layer: env =>
        Layer.mergeAll(
          originalsR2(env.MEDIA),
          transformerImages(env.IMAGES),
          variantsR2(env.MEDIA, { cache: true }),
          admissionRateLimit(env.LIMITER),
        ),
    },
    uploads: {
      layer: env =>
        Layer.mergeAll(
          uploadsR2(env.MEDIA),
          inspectorImages(env.IMAGES),
          uploadPolicyBearer(env.UPLOAD_TOKEN),
        ),
    },
    // The application's own routes.
    otherwise: async () => new Response('Not found', { status: 404 }),
  }),
}
  • transformerImages(images) scales down without enlarging and returns 503 on quota exhaustion (code 9422) or 502 with an ImageFailed diagnostic.
  • keyValueStoreR2(bucket) exposes an R2 bucket as an Effect KeyValueStore.
  • variantsR2(bucket, { prefix?, cache? }) stores variants in R2 so each size is generated once, optionally checking the Worker Cache API (cache: true or a custom Cache instance) first.
  • originalsR2(bucket) streams originals directly to avoid buffering full files in Worker memory.

Cloudflare bills each unique width per original once per month, so a smaller width ladder caps monthly transformation cost. See examples/cloudflare for a complete Worker setup.

Diagnostics

When the server recovers from a recoverable failure, it emits a typed MediaDiagnostic through foldkit-surface's Diagnostics.Reporter (which defaults to Effect logging and wrangler tail):

_taglevelWhen
StoreFailederrorA variant, edge, or facts cache read/write failed and fell back
AdmissionFailederrorRate limiter failed; request admitted without limiting
ImageFailederror / warningTransform failed (error), or upload inspection rejected a body (warning)
FactsUnansweredwarningFacts route returned Unavailable for url with reason
RequestDefecterrorUnhandled defect answered 500 by serveFetch

Provide a custom reporter as an Effect Layer alongside your media layers:

ts
import { Effect, Layer } from 'effect'
import { Diagnostics } from 'foldkit-surface'

serveFetch<WorkerEnv>({
  images: {
    layer: env =>
      Layer.mergeAll(
        originalsR2(env.MEDIA),
        transformerImages(env.IMAGES),
        variantsR2(env.MEDIA, { cache: true }),
        Layer.succeed(Diagnostics.Reporter, diagnostic =>
          Effect.logError(`media: ${diagnostic._tag}: ${diagnostic.message}`),
        ),
      ),
  },
})

In tests, Diagnostics.collect() gives a reporter and the array it records.