Data and sync

A canvas is a folder of JSON files. The CLI and the server write it under a shared lock, and every write produces a new revision. Reads go through a SQLite index built from the files. Around that core are backups, a merge for saves that lose a race, and undo in the app.

A folder of files per canvas, under a lock

Each canvas has its own folder (#475). canvas.json holds its settings, and changes only when they do. Each node is nodes/<id>.json and each link is links/<id>.json, so editing a link never rewrites a node file. New nodes and links get random UUIDs, and older ids keep working. Each file keeps a fractional order key, so adding or moving an entry rewrites only its file, and a changedAt time for each field, data fields one by one, so two versions of a file merge by keeping the newer value of each field (#479). Two branches that edit separate cards merge in git without a conflict. Keys are sorted and formatting is stable. An id keeps lowercase letters, digits, hyphens, and underscores in its file name and escapes every other byte as % and two hex digits, uppercase letters included, so ids that differ only in case don't share a file on a case-insensitive disk.

Fields that record one machine's view of the canvas, the GitHub refresh's records, the directory of GitHub repositories by id, and the stored suggestions, go in local.json instead, along with the revision, which counts this machine's writes. The lock, backups, focus report, and log sit beside the files for now, so a canvas can still be copied or backed up as a unit.

A write splits the whole canvas into files and rewrites only those whose text changed, removes node and link files the canvas no longer has, and replaces local.json last, even when it's unchanged. Each file is written to a temporary file and moved into place. While a write runs, an orion.writing marker sits in the folder. A reader that sees the marker, or finds local.json changed by the time it finishes, reads again, so it never returns a write half done. A writer that's killed partway leaves the marker behind, and the next reader or writer finishes its write before going on, as described under the index below. The server's watcher looks at local.json, since a change to it marks a finished write.

The first command a newer Orion runs splits every canvas's orion.json into files, keeping the original as orion.backup.json. It refuses while an older orion server start still serves the folder, since that server would keep writing orion.json; a server that reads folders says so in its serve.json.

Every writer takes the same lock. The lock is an orion.lock directory, created with mkdir, which either succeeds for one process or fails for the rest. A writer waits up to 5 seconds for it. Writes of view state skip the lock. The app's report of its open group and selection never changes the revision (#33).

A lock can outlive its writer if the writer crashes. Reclaiming those locks depends on whether the writer is still running, not only on the lock's age:

• A lock with no readable owner stamp uses the lock directory's modified time in its place.
• A stale lock is reclaimed once its owner process is gone, or when no owner process is recorded. After ten minutes it is reclaimed even if its process is still running, in case the process id was reused.

Before this change, any lock older than 30 seconds was reclaimed, even while its writer was still running. A second writer could then overwrite the first one's result.

Callers that read a revision can refuse a write when the document has moved on. --if-revision does this for the batch commands, and the same check is what lets an agent preview a change and then apply it safely (CLI and agents).

A SQLite index for reads

Opening every file of a canvas gets slow once it holds thousands of ingested records, so reads go through a SQLite index beside the files, index.sqlite (#476). The files stay the source of truth. Deleting the index loses nothing, and a damaged one, or one made for an older layout, is deleted and built again.

Lute has no SQLite binding, so the CLI runs the sqlite3 command with -json and WAL mode on. A batch goes in as one JSON string, through a file since Linux caps one command-line argument at 128 KB, and json_each() unpacks it. The canvas itself comes back as one JSON value printed as it is, so it's parsed once rather than as a string inside -json's rows. Without sqlite3, Orion reads every file as before and logs a warning.

Each read first brings the index up to date. sqlite3's fsdir() lists the folder's file times, and only files that are new, have a different time, or changed in the same second they were last read are read again. Times come in whole seconds, so that last rule catches a change the time can't show. Files that are gone are dropped. The index then joins the canvas the way the files do: entries in the order of their keys, with ties broken by id, then any without a key. A file that doesn't parse no longer fails the read (#478). A file with git's conflict markers keeps one side, ours when it parses. When the two sides disagree on a note, the note holds both versions between the markers, so it shows as a conflicted note until someone resolves it. Any other node file that doesn't parse shows as a note holding its text. Each problem is logged, and the read joins that canvas from every file's text rather than in SQL. Only a canvas.json that doesn't parse on either side still fails, since nothing else holds the canvas's settings.

The index also journals writes (#477). A writer puts the marker down, records the files it will write and remove in the index's operations table, moves them into place with local.json last, then marks the operation finished and records the files it wrote, so the next read doesn't open them again. A marker that's still down when someone takes the lock belongs to a writer that's gone. They put every unfinished operation in place again from the start, so a write killed partway leaves the canvas as it is after that write. A marker without an unfinished operation means no file had moved yet, so only the marker goes. An unfinished operation whose marker someone removed is marked abandoned by the next write, so it's never replayed over later writes. Readers that find the marker take the lock to do the same, which also waits out a live writer.

The server's watcher also watches the nodes and links folders, so files changed outside Orion, such as by a git pull, reach the app without a restart. Every read returns a fingerprint of the index's file hashes. When it changes without a new revision, the watcher announces the change with reindexed set, and the app reloads despite seeing its own revision.

On a generated canvas of 5,000 cards, opening it and answering orion next took about 140 ms from the index against about 590 ms from every file, and about 350 ms when building the index from nothing. lute run bench-index in workspace/orion-cli/ measures it.

Rolling backups

Before a write, the writer saves the whole canvas it replaces as one JSON file in the canvas's backups folder, named with the revision it replaces and a UTC timestamp. Backups are kept for the last 24 hours, plus the newest backup from each UTC day for 30 days. Pruning only removes files that match that name pattern. A backup that fails doesn't stop the write.

orion backup list shows the saved revisions, newest first. orion backup restore <revision> brings one back as a new revision, after backing up the current file first (#18). A restore is never destructive, so a mistaken restore can be undone the same way.

Merging saves that lost a race

When the app's save is rejected because another writer saved first, the app no longer drops its edit. It fetches the newer file, merges its own changes onto it, and saves again, up to two retries (#21). The same merge runs when another writer's change arrives while the app has an unsaved edit.

The merge is three-way. It compares the last copy known to match the file, the app's current state, and the newer file:

• Nodes and links match by id. Additions from either side are kept.
• A deletion wins only if the other side didn't edit the item.
• Position, parent, size, and each data field merge separately.
• If both sides changed the same field, the app's value wins, and a short notice says how many fields that touched.
• Links whose ends no longer exist are dropped.

Workspaces became canvases

A workspace started as its own data folder (#23). The CLI renamed the commands to canvas in #346, and #409 renamed the data, routes, and MCP tools to match.

The data folder now lists canvases in canvases.json, which also records the active one. Canvas folders live under canvases/<name>/. The first command a newer Orion runs moves an older layout into place:

• Folders move whole, so each canvas keeps its data and backups.
• The old list stays beside the new one as workspaces.backup.json.
• A move that stops partway picks up where it left off.
• The move refuses while an older orion server start is still serving the folder.

The migrations live in workspace/orion-cli/src/migrations/, and that folder's README dates each one. They're removed at the 0.1.0 release, since nobody installing that release has an older folder. The decisions explain why data moves on its own instead of by hand.

Refs and document version 2

Cards used to carry source-specific fields such as pr, branch, and external. Version 2 of the document replaces them with data.refs, a list of refs that each record their source, kind, key, URL, status, and source-specific details. A ref's kind and case-insensitive key identify it across ingests. Older documents migrate when they are read, so nothing needs a manual step.

Undo and redo

The app records undo steps in memory (#166). A step is recorded once the document has been still for 300 milliseconds, so a drag, a tidy's animation, or a burst of typing counts as one step. Nothing is recorded while a card is being dragged.

Undo doesn't restore a copy of the old document. It merges the step's inverse onto the current one. Writes from the CLI, a sync, or another window stay in place, and where one of them touched the same field as the step, its value wins. History holds the last 100 steps and starts over when the canvas changes. The decisions say undo history should survive a canvas switch, which this version doesn't do yet.

Edits as operations

Each canvas edit is becoming one operation with an inverse (charter, principle 8). Part one is in review. It adds workspace/orion-cli/src/operations/, where a list of operations applies in order under the lock. If one no longer applies, the earlier ones are undone and the whole list is refused, with the failing position named. Each operation returns the nodes and links it changed as they were before, so applying those inverses in reverse restores the document exactly. Nothing stores those inverses yet. Storing them is the next step.

The operations go out through POST /api/operations. A list lands as one revision. An optional base revision must still match, and a refused list or a stale base leaves the file's bytes alone.

Part two, also in review, changes the app to send each committed edit to that endpoint instead of saving the whole document. The plan then removes the whole-document save and the app's merge, once the operations path covers every edit.

Store hardening

Several changes exist to protect the data:

• Concurrent writers are tested. A spec, in review, starts six orion add processes at once and checks that every card lands and the final revision is right. It also checks that an unreadable or truncated data file is left byte for byte unchanged. Creating a fresh data folder had a race that lost one writer, and the same change fixes it.
• Undo is tested against random edits. A property test, in review, checks that undo and redo never leave dangling links or duplicate ids, even when another writer changed the document in between.
• A mutation testing script for Luau modules checks whether the specs catch deliberate changes to the code.

Where it's headed

The charter's first phase has a storage bet: a canvas becomes a folder of text files that can live in a repository, like an Obsidian vault, and SQLite becomes a rebuildable index for reads (#246, #121). The files and the index came first. A journal for writes that touch several files, re-indexing after a git pull, and merge-friendly ids and ordering follow. The decision keeps simple read and write commands in the CLI and keeps one sqlite3 process open in the server. Files stay easy to inspect and diff, and the index keeps large canvases fast.