Queries and pagination

Entities answer "which fields of this known thing?" A Query answers "which entities belong in this server-owned list?"

Declare the query and register it with the domain:

import { Query } from 'foldkit-remote'

const ProjectsByOwner = Query.make('ProjectsByOwner', {
  Input: { ownerId: Schema.String },
  Result: Project,
})

const Data = Remote.make({
  model: App.model.remote,
  entities: [User, Project],
  queries: [ProjectsByOwner],
})

Query.make names a query and says what it returns. Where it means something the client should be able to state, declare it with a body instead:

import { Entity, Expr, Order, Query } from 'foldkit-entity'

const Task = Entity.define(
  'Task',
  Schema.Struct({ id: Schema.String, ownerId: Schema.String, updatedAt: Schema.Number }),
)

const TasksByOwner = Query.define(
  'TasksByOwner',
  { ownerId: Schema.String },
  ({ input }) =>
    Query.from(Task).pipe(
      Query.where(Expr.eq(Task.fields.ownerId, input.ownerId)),
      Query.orderBy(Order.desc(Task.fields.updatedAt), Order.asc(Task.fields.id)),
    ),
)

A body is written over a foldkit-entity Entity, because that is what has addressable fields: Task.fields.ownerId is a reference an Expr can be built from, where this package's own Entity.make describes a field as the schema of its value and has nothing to point at. An entity declared either way still reads, selects and normalizes the same; only a query body needs the richer declaration.

What comes back is an ordinary descriptor — the same name, Input, ref and connection identity — carrying its body besides, and the result is a connection over the Entity the body reads, so it is not named twice. A server can compile that body rather than be told the same thing again in its own dialect; one that would rather answer the query its own way still can, and a descriptor from Query.make has no body at all.

The body is built once, when the query is declared. Inside it input.ownerId is a placeholder for the value the query will be given, not the value — there is nothing there yet to branch on. A condition that depends on what was passed is a comparison over the placeholder, never a ?: around it.

Input must be fields or a plain Schema.Struct, because that is what the placeholders are made from. A codec that exposes no keys is refused where the query is declared rather than handing the body an empty object, which would compare a column to nothing while every type agreed.

Then a query is still just a Projection:

const projects = Data.query(
  ProjectsByOwner,
  { ownerId },
  { select: ProjectSummary, first: 25 },
)

projects.read(model)
// RemoteData<Page<ProjectSummary>>: at most 25 items

Data.more(model, projects)
// Option<Model>: "load more", the window grown to 50; none when all are shown

The connection stores entity references and explicit boundaries rather than one flat array. That matters when page 1 and page 3 are loaded but page 2 is not: Remote represents an honest gap instead of pretending the visible rows are adjacent.

A query Projection is Initial until the page and every selected field of its visible items are present. The select travels with the query: the server returns the selected fields of the page's items with the edges, and one ConnectionMerged merges the page and writes them, so a fresh list usually lands Ready in one response. What the server leaves out (withheld fields, nested targets it does not expand) becomes ordinary entity requirements and is fetched next, as before; a server that predates payloads sends edges only and the list reads the old two-round-trip way.

A read shows at most its window. Remote keeps one connection per query and input, whatever window asks for it, so a picker's first: 50 and a card's first: 3 of the same query share the rows, and each read is cut to its own window: first from the start, last from the end. hasNext / hasPrevious are true when the read cut rows or the connection's boundary says there are more; they are never guessed from a row count.

Two Projections of the same connection plan one query, for the wider window, and select the union of their fields. When a connection holds fewer rows than a window asks for, the read entry asks only for the rows it lacks, after the loaded end (or before the loaded start).

"Load more" grows the window. Data.more(model, projection) is the Model with that read's window one page larger, called from update; the read entry then fetches what the connection lacks. It is kept in Remote's store, under the query, input and window first asked for, so a list holds no state of its own, and it is forgotten when retention drops the connection. Another window of the same query does not grow with it.

Filtering a loaded list without asking the server#

A search box over a page you already have should not be a round trip. Data.filtered narrows a loaded list by a query body:

const found = Data.filtered(model, projects, ActiveProjects, {})

found.items     // decoded exactly as the list decodes them
found.complete  // whether the answer was about the whole list

It filters a list; it does not run a query, and the distinction is the design. "Which rows match" would need to know this list holds every row the body could match — predicate containment, which Remote deliberately does not do. "Which rows of this list match" is decidable from what is already here.

complete is false unless three things hold: every edge could be judged (no row missing a field the body reads), every match could be shown (no row missing a field the Selection reads), and the list is terminal at both ends. So empty-and-complete and empty-and-partial stay different answers — one means "none", the other means "none that I can see yet". An input no interpreter may answer (a search holding NUL, which the server refuses too) gives no items and complete: false, rather than throwing in update.

Nothing is created: no connection, nothing new to retain, nothing new to fetch. The server stays authoritative for which rows exist.

Judging the rows you already hold#

A connection's rows are edges the server delivered, and a query body describes which rows — so until something evaluates the body on the client, nothing here can say whether a row belongs to a query. Remote.matching says:

const judged = Remote.matching(Data.storeOf(model), ProjectsByOwner, { ownerId: 'u1' })

judged.matched  // keys satisfying the body, in the order it asks for
judged.skipped  // keys held, but missing a field the body reads

It is pure, and it runs the same interpreter the server checks itself against — the reference implementation in foldkit-entity — so a body means one thing in both places. The conformance suite is run over a store as well as over rows and a table.

Three things about it are deliberate.

It answers "which of the rows I hold match", never "which rows match." Those are different questions, and the difference is not recoverable from a list of keys — so it is in the shape. A caller that knows it holds the whole population (a connection terminal at both ends) is the one that can turn this into a complete answer.

skipped is the honest half. A row missing a field the body reads cannot be judged: it is neither a match nor a non-match. Dropping it silently would turn "I could not tell" into "no", so it is named instead, and the caller decides whether to fetch the field, ask the server, or say the answer is partial.

It takes the input decoded and encodes it itself. A store holds wire values, so a comparison has to happen in that space — and handing an interpreter a decoded value is a mistake no runtime error catches: the answer is simply empty, which reads exactly like a correct one. Doing the encoding here means a caller cannot get it wrong.

It is not authorization. On the server a compiled where is conjoined with the binding's visible rule; there is no visible here. This is safe only because the client holds only rows the server already released to it, and a local filter is not an access decision.

Local execution is not a second authority. matching and filtered derive conclusions from facts the server gave; they never promote those conclusions into knowledge of the server's whole dataset. Knowing p1.ownerId = u7 says that p1 matches ProjectsByOwner(u7); it does not say that ProjectsByOwner(u7) is [p1], and nothing here will say so unless a connection's boundaries, terminal at both ends, independently prove it. That line is what keeps Remote a cache of what the server owns rather than a local database with opinions of its own.

An input that changes as fast as someone types#

A connection's identity is its query plus its input, which is exactly right for caching and exactly wrong for a search box. Bound naively, every keystroke is a different connection, a different request, and a different thing to retain:

// Don't: `postSearch` changes per keystroke, so the query's input does too.
h.OnInput(text => Message.Searched({ text }))

Remote has no debounce, and should not: its job is to be a faithful function of the Model, so if the Model says the search is Engi then Engi is what the query asks. The debounce belongs between the input Message and the Model field the query reads — which is ordinary application state, and foldkit-primitives already has the piece:

import { debounce } from 'foldkit-primitives/time'

const SearchInput = debounce({ name: 'PostSearch', value: Schema.String })

// `latest` is what the box shows, so typing stays immediate.
// The settled OutMessage is what moves the field the query reads.
// A step of the page's Bundle.compose(...).pipe(...):
Bundle.withChild('search', SearchInput, {
  args: { delayMs: 250 },
  onOut: out => model => ({ model: modifyFields(model, { postSearch: () => out.value }) }),
})

Two fields, deliberately: the one the box draws changes on every keystroke, and the one the query reads changes a quarter second after the last one. examples/entity does exactly this.

The same applies to any high-frequency query input — a slider, a map viewport, a date scrubber. What makes a search box the usual case is that the query is often written for it: Expr.contains over an empty string matches everything, so an empty box and a filled one are one query rather than two.

A page that overruns its window is refused#

The client asks for first: 25, and a page carrying more than that is a protocol disagreement rather than a windfall: the client cannot tell which of the edges the window meant, and the connection's boundaries stop describing what it holds. Such a page becomes a QueryFailed with a named protocol error, and none of its edges reach the store — a connection already loaded is left exactly as it was, so a refresh that overruns cannot replace good rows with a rejected page.

A window with neither first nor last bounds nothing: after/before says where to start, not how much to take.

When a read fails#

A failure is kept on the Model until something settles it, and the read says so. Queries fail per connection, and entity reads per field. A read with nothing to show reads Failed with the error. One that had a value (a failed refresh, or a failed "load more") reads Failed with that value as previous, which RemoteData.render draws as data with a Stale freshness, so it stays on screen. A failure reaches every read that needs it: a list fails when one of its rows' fields failed, and a project read fails when its owner's did.

It is not retried on its own. A persistent error would otherwise be asked again every time some unrelated read restarted the entry. What failed is left out of the plan, and everything else is still fetched: the rows a failed list holds, the other fields of an entity. What asks again:

• Data.refresh(model, projection) — the retry button. It works on something that never loaded, too.
• The value arriving anyway: a page for a list, or a read, live patch or mutation result that writes the field.
• The server invalidating a list over a live stream, or deleting the entity.
• Retention dropping it: something nothing reads any more forgets its failure, so coming back to it later asks the server again.

Remote.inspect(model.remote).failures lists what failed, by connection identity and by entity\0id\0field mark.

A broken live stream is not a failed read. Its ReadFailed carries the stream, and records a gap in RemoteModel.gaps rather than failing any field: nothing was being read, and the values on screen are still what the server last said. The stream restarts on its own; see When a stream breaks.

When a field is withheld#

A server may answer a read without a field it was asked for, and mean it: the Source's authorize withheld it from this principal, or the record it returned had no such field. The read result says so, in settled: the fields it asked for that this answer does not carry and a later one would not either. Why is not on the wire. The client records each as unavailable, one more kind of knowledge beside a value and a tombstone:

missing      not established; the planner asks
present      known
stale        known, and being asked again
unavailable  asked, and answered without: the planner does not ask again
tombstone    the whole entity is known absent

A Selection that names an unavailable field reads Failed with an Unavailable error naming the field, not Loading (nothing is fetching it) and not NotFound (the entity is there). What it is not is a seventh state: the remedy is the one a failed read has. Select without the field, in a Selection of its own if some principals may read it, or show the error. A Selection that does not name it is unaffected: name reads Ready while privateNotes is unavailable beside it.

The mark is forgotten by what forgets a failure: Data.refresh asks again, and a read, live patch or mutation result that writes the field clears it. It survives persistence and a server render's Remote.resume, so the browser does not ask once more for what the server already declined.

An id the server answers with nothing about, neither values nor settled fields, is a tombstone as before. A server settles a withheld field without reading its Source for an id whose every field is withheld, so an absent id and a hidden one answer alike, and existence is not told.