Saying something about a row: Expr

An Entity says what a domain has. An Expr says something about one row of it, as a value:

Why query semantics live in this package. They describe and interpret nothing, which is the line this package already draws: an Expr says which rows and fetches none of them, and the interpreters that fetch live elsewhere — foldkit-remote-drizzle compiles a body to SQL against a database.

evaluate is here rather than there because it is the reference semantics of the IR: what the operator definitions mean, operationally, over rows already in hand. A specification's reference implementation belongs with the specification, and it reaches nothing — no database, no transport, no Remote concept. The rule that keeps it honest is that it may depend on the IR and nothing else; the moment it wants one of those it has moved to the wrong place. Nothing that imports only Entity pays for them: they are ordinary consts in a sideEffects: false package, so a bundler drops them, and a package like foldkit-form imports only types from here in any case.

import { Expr, Order } from 'foldkit-entity'

const byTitle = Expr.eq(Blog.Post.fields.title, Expr.input('title', Schema.String))
const published = Expr.eq(Blog.Post.fields.published, true)
const newest = [Order.desc(Blog.Post.fields.title), Order.asc(Blog.Post.fields.id)]

Building one performs no work: it reads nothing, names no database, and runs no query. An interpreter compiles it — foldkit-remote-drizzle to SQL, an in-memory evaluator to a predicate over rows — which is what lets one query mean the same thing in more than one place.

A comparison coerces what it is given, so the common forms read as they mean: a field becomes a reference, a plain value becomes a literal, and an Expr is already one. What it will not do is compare a field to the wrong kind of value — Expr.eq(Blog.Post.fields.title, 42) is an error where it is written, rather than a row that never matches. The value's type comes from the left, so eq takes the field on the left; Expr.eq('Hello', Blog.Post.fields.title) is refused with that sentence. The comparisons (lt, gte, …) also take a field second, wrapped in Expr.field(…).

An input is a placeholder, not a value. A query's body is built once, so Expr.input('title', …) stands for whatever the query is given when it runs — there is nothing there yet to branch on. An InputExpr is an object, so a condition ? a : b over one is always truthy and decides itself once, forever. A query that depends on what it was passed says so with a comparison over the placeholder instead of a branch around it.

dependenciesOf(...) says which fields and inputs an expression reads and which operations it uses, so a planner knows what it needs and an interpreter can refuse a query it cannot run. Each field names its owner, the Entity's identity, as well as the Entity's name, so two Entities defined with the same name are two entries.

Every node is frozen as it is built, so a query is the value it was when Query.where checked it: nothing can swap a field or a literal in afterwards.

A node can be shared (Expr.eq(n, n)), and the analysis visits each one once. An interpreter does not: evaluate reads a predicate as a tree on every row, and the SQL compiler writes a shared operand out wherever it appears. So Query.where refuses predicates that expand past maxQueryNodes (1,000) nodes, counting a shared node on each path that reaches it. A node may be shared, Expr.eq(n, n), and every walk over one visits each node once, so a deeply shared predicate costs its size, not its paths.

Asking a question that depends on an input, without branching on it#

Since an input is a placeholder, a query cannot choose between two shapes based on what it was given. It does not need to: a question that looks like a branch is usually a comparison waiting to be written.

// A list that shows archived entries or unarchived ones, as the reader asks:
Expr.eq(Expr.isNotNull(Entry.fields.archivedAt), input.archived)

// A search box that filters when something is typed and not when nothing is:
Expr.contains(Entry.fields.label, input.search)

The first is "is-archived equals what you asked for". The second relies on everything containing the empty string. Both are one static body, and both ask exactly what archived ? … : … and search === '' ? … : … asked.

isNull and isNotNull are the same node with the answer absence gives flipped, so nothing has to negate a predicate to get the other.

contains is case-insensitive, which is what a search means — and which has to be said, not left to the interpreter: SQLite's like ignores case and Postgres's does not, so a body that left it open would mean two things. Folding is ASCII-only, since that is what lower does in SQLite without ICU: É does not match é, though each matches itself. Text holding a NUL character is refused by every interpreter here, in contains, eq and in alike, since Postgres text cannot hold one and SQLite compares only up to it.

in asks whether a value is one of a list, written out or given as an input: Expr.in(Todo.fields.projectId, input.projectIds). Each element is compared as eq compares, and it reads as SQL's in: a null value is unknown, so is no match against a list holding a null, and an empty list matches no row (on Postgres too, which refuses in (); the compiler says false instead).

It searches text, and the operand is constrained to text. contains over a number would compile to lower(rank) like …, which SQLite coerces into an answer and Postgres rejects at runtime — so a non-text operand is a compile error where it is written rather than a surprise where it runs. Nullable text is allowed, because it is a real case with a real meaning:

Over a column that can be null it is not the same as no filter. A null contains nothing, not even the empty string, so its rows drop out. The column the CMS searches is declared not-null, which is what makes an empty search exactly everything there.

Which rows: Query#

A Query is an Entity to read, the predicates every row must hold, and the order to read them in — composed with pipe, one immutable value per step:

const published = Query.where(Expr.eq(Blog.Post.fields.published, true))
const newest = Query.orderBy(Order.desc(Blog.Post.fields.title))

const recent = Query.from(Blog.Post).pipe(published, newest)
const oneOf = Query.from(Blog.Post).pipe(published, Query.where(byTitle))

published and newest are fragments: written once, piped into any query over the same Entity. Composing performs no work — no table is named, no connection opened, nothing read. A fragment over another Entity is refused where it is piped: a predicate's type names the Entities its fields belong to, so Query.from(Blog.Post).pipe(Query.where(Expr.eq(Blog.Comment.fields.body, 'x'))) reads a predicate reads Comment, but the query is from Post, and an ordering term over another Entity is refused by Query.orderBy the same way. The same check runs again when the query is built, for what a type cannot see: a predicate typed plainly as Predicate, or two Entities that share a name.

Two wheres conjoin and two orderBys append. Neither replaces what came before, so piping a fragment can only ever narrow a query, never silently undo part of it. An earlier ordering term stays the more significant one, which is what makes a later Query.orderBy(Order.asc(id)) a tie-breaker. Neither ever replaces what a fragment added.

A sort the reader picks is part of the body too. Order.chosen maps each name the input's sort can hold to a field, so the server and the client read one declaration of what title means, and the client can tell which edits may move a row:

const PostSort = Schema.NullOr(
  Schema.Struct({ by: Schema.Literals(['title', 'id']), direction: Schema.Literals(['asc', 'desc']) }),
)
const sorted = (sort: InputExpr<typeof PostSort.Type>) =>
  Query.from(Blog.Post).pipe(
    Query.orderBy(Order.chosen(sort, { title: Blog.Post.fields.title, id: Blog.Post.fields.id })),
  )

Every name the sort can hold needs a field, or it does not compile. A null sort chooses none, and the terms after it decide. Query.orderFor(query, input) is the fixed terms for one input, which is what every interpreter sorts by.

Every order is total. orderFor ends it on the Entity's id, ascending, unless the terms already read the id, so rows with equal keys come out in one order on every interpreter rather than each breaking ties its own way.

How text orders is declared, or left to the server. Left unsaid, SQLite compares bytes, Postgres follows the database's locale and a JavaScript sort follows UTF-16 code units. A field declares its collation, and every interpreter that can honour it orders the same; one that cannot refuses:

const Tagged = Entity.define('Tagged', Schema.Struct({ id: Schema.String, name: Schema.String })).pipe(
  Entity.annotateMembers({ name: Collation.of(Collation.asciiFold) }),
)
const byName = Query.from(Tagged).pipe(Query.orderBy(Order.asc(Tagged.fields.name)))
Query.placement(byName, {}) // { _tag: 'Placeable' }

• Collation.binary is code point order: SQLite BINARY, Postgres COLLATE "C". An id is binary unless it says otherwise.
• Collation.asciiFold folds ASCII letters, then compares by code point: SQLite NOCASE, Postgres lower(x COLLATE "C").
• Collation.locale('en-US') is the backend's own, ordered on the server and never by a client.

A term overrides its field's: Order.asc(field, { collation }). Query.placement(query, input) says whether a client can order the rows exactly as the server does: every term over a number or a boolean (a row may lack it), or over text with a portable collation. A body with no order, text with none declared, or a locale is NotPlaceable, with the reason. foldkit-crud's Sort.make(['title', 'id']).Schema is such a sort.

The list of predicates is the conjunction: a query wanting three conditions writes three wheres. Expr.and is for a conjunction inside something else, and Expr.or and Expr.not are the other two connectives:

// A search over two fields: the registry finds a product by description or UPC.
Expr.or(
  Expr.contains(Product.fields.description, input.search),
  Expr.contains(Product.fields.upc, input.search),
)

// A filter that applies only when its input is given (`input.status` may be null).
Expr.whenGiven(input.status, status => Expr.eq(Product.fields.status, status))

They mean what SQL's do, three-valued: a comparison of a null is unknown, a row is kept only when its predicate is true, or takes true over unknown and and takes false over it, and not of an unknown is unknown. So Expr.not(Expr.eq(field, 'draft')) leaves out a row whose field is null; "everything but drafts", nulls included, is Expr.or(Expr.isNull(field), Expr.not(Expr.eq(field, 'draft'))). and and or need an operand (an empty one would match every row, or none), a nested one of the same kind is flattened into its parent, and one operand is that operand. An interpreter that runs per request decides an operand that reads no field before it writes anything: whenGiven compiles to true, which filters nothing, when its input is null, and to the predicate alone when it is given.

Comparisons order two values: Expr.lt, lte, gt and gte.

Expr.lt(Product.fields.cents, 1000)
// A range of dates, stored as ISO strings, which order by time under binary:
Expr.and(
  Expr.gte(Shipment.fields.shippedAt, input.from, { collation: Collation.binary }),
  Expr.lt(Shipment.fields.shippedAt, input.until, { collation: Collation.binary }),
)

Numbers compare as numbers, and nothing compares across kinds or for a value stored as neither a number nor text. Text orders by a collation, and backends disagree about the default one (SQLite's BINARY, the Postgres database's locale, JavaScript's UTF-16 units), so text is compared only under Collation.binary or Collation.asciiFold, declared on the field or passed, and only against a field: anything else is refused where the comparison is written. A value is compared in its encoded form, which is why a date stored as an ISO string is text here. A null on either side is unknown, as in SQL. There is no between; it is the and above.

A query reads one Entity, so a predicate or ordering term naming a different one is refused where it is piped: an interpreter would otherwise be asked for a column of a table it was never told to read. Entities are compared by identity, so two declared with the same name are two Entities here as everywhere else.

Saying what an interpreter can run#

An Expr is only portable if the thing running it says what it runs. Query.unsupported(query, supported) names the operations a query needs that an interpreter does not have:

const missing = Query.unsupported(body, ['eq', 'isNull', 'isNotNull'])
if (missing.length > 0) throw new Error(`cannot run ${missing.join(', ')}`)

The refusal belongs to the interpreter, not here: one that compiles at registration and one that runs a body directly fail at different moments. What matters is that it refuses rather than skipping the operation — an interpreter that quietly drops a contains it cannot compile answers a different question in full confidence, and every test it does support still passes.

Reading one back#

Query.show(query) renders a body as text, one clause per line:

Query.show(body)
// FROM Post
// WHERE Post.slug = $slug
//   AND Post.published = true
// ORDER BY Post.updatedAt DESC

For a person — an explanation, a diagnostic, a test asserting on a whole predicate at once — and never for an interpreter. It is close enough to SQL to read at a glance and unlike it everywhere that matters: an input is $slug rather than a bound parameter, contains is named rather than rendered as somebody's like, and no dialect's escaping or collation is implied. What actually ran is whatever that backend compiled, which is not this. Expr.show(node) does the same for one expression.

A Query says which rows. It does not say which fields — that is a Selection — and it does not say how many, whether absence is an error, or whether to watch for changes: those belong to the consumer doing the reading, not to the relation.

Only the operations a real query in this repository needs exist. The set grows from queries, not from what a database could express.