Skip to content

feat(appkit): add appkit db introspect - #670

Draft
ditadi wants to merge 28 commits into
stack/database-dev-flow/03-db-migratefrom
stack/database-dev-flow/04-db-introspect
Draft

ditadi wants to merge 28 commits into
stack/database-dev-flow/03-db-migratefrom
stack/database-dev-flow/04-db-introspect

Conversation

@ditadi

@ditadi ditadi commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR (generated by Stack)

Use this link to review incremental changes.


Stacked on #666. The design is #677.

What did you change, and why?

Apps whose tables already exist in Lakebase need a schema.ts and a migration history that match them. appkit db introspect reads a Postgres schema (app by default, or --schema-name) from pg_catalog and writes config/database/schema.ts, keeping the previous file as schema.ts.bak. It also writes the migration that brings history up to the database, and records it as applied. Without history that migration is a first 0000_baseline, which the deployed app adopts on start (#654). With history it is the next NNNN_introspect. Recording is only true on the branch introspect read, so introspect runs only on your dev or session branch.

What the builder cannot express is left out with an // appkit: skipped … comment, and what introspect does not read at all is named in a warning. Introspect refuses when the branch does not record exactly the committed migrations, when its migration would drop something schema.ts left out, and when it would replace schema.ts with an empty one. If anything fails after the write, it puts back schema.ts, its backup, database.d.ts and the migrations.

This revision keeps what the replaced schema.ts declares:

  • Introspect loads the schema.ts it replaces first. A declared column keeps its .private() as it was. Every other column, table and foreign key is written as the catalog has it, foreign keys with fk().
  • A schema.ts that exists but does not load refuses the run before anything is read or written.
  • When two foreign keys point at one table, the one schema.ts declares keeps the relation.
  • Introspect holds the history lock from feat(appkit): add appkit db migrate #666 for its whole run.

Review round, 2026-11-09:

  • Introspect refuses a branch that recorded a migration whose file changed since; keeps one foreign key per column instead of silently overwriting the first; and reads the schema an existing schema.ts declares when --schema-name is absent.
  • Fidelity: an enum with an empty label skips the column; literal defaults cast to quoted type names keep their value; deferrable unique constraints and key sequences with non-default options are reported as approximations.
  • schema.ts, its .bak and the migration journal are written through tooling/files.ts (refuse a symlinked destination, temp file + rename).

Fix round for fjakobs's review of #677, 2026-11-15:

How do you know it works?

  • render.test.ts, introspect.test.ts and roundtrip.test.ts cover declared privacy, new columns and tables written without .private(), kept, dropped and undeclared foreign keys, a schema.ts that does not load, a run while migrate holds the lock, and each drift and drop refusal. The round trip loads both files through the real loader. A refresh keeps what the old file declares, and adds the new column, table and foreign key as the catalog has them.
  • catalog.pg.test.ts and apply.pg.test.ts run against Postgres. They read a real schema, and record pending migrations without running them.
  • A smoke against a local PostgreSQL 14 kept a private column private. It ran before this revision, which writes new columns and tables without .private().
  • At this PR's head, the database, plugin and CLI db tests passed against a local PostgreSQL 14, typecheck passed, and a clean build stayed within the size gate. The full vitest suite passed at an earlier revision of this PR and at the top of the stack (feat(appkit): add appkit db check #678). The packed tarballs export the same root and ./beta names as main, and ./beta/database-tooling adds only runIntrospect. CI runs the full suite on this PR.
  • This revision was not run against a real Lakebase workspace.

Review round, 2026-11-09: each fix has a failing-then-passing test in its PR, and every layer's appkit and shared tests ran against Postgres 14 after the sync. At the top of the stack, pnpm build, pnpm test (6502 passed, 1 skipped), pnpm -r typecheck, pnpm check, pnpm knip and pnpm docs:build pass.

Fix round for fjakobs's review, 2026-11-15: each fix has a failing-then-passing test in its PR, and every layer's database, plugin and CLI tests ran against Postgres 17 after the sync. A throwaway Lakebase fork confirmed that the developer role can write the service principal's rows but cannot alter its tables. At the top of the stack, pnpm test (6602 passed, 1 skipped), pnpm -r typecheck, pnpm build and pnpm docs:build pass, and oxlint and oxfmt are clean on every changed file. After the AutoCRUD removal, the introspect tests failed first and then passed, and at the top of the stack pnpm test (6601 passed, 1 skipped), pnpm -r typecheck, pnpm build and pnpm docs:build pass.

`appkit db introspect` reads an existing Postgres schema (default: app) from
pg_catalog and writes config/database/schema.ts from it, keeping the previous
file as schema.ts.bak. It also writes the migration that brings history up to
the database, 0000_baseline first and NNNN_introspect after, and records it
as applied. That is only true on the branch it read, so it runs only on the
developer's dev or session branch, which `appkit db init` creates. The
deployed app adopts 0000_baseline on start when its tables already exist.

What the schema builder cannot express is skipped and reported, and what it
declares differently from the database (a timestamp precision, an index) is
listed. Objects it does not read (views, triggers, policies, functions,
standalone sequences) are named in a warning, since a database the
migrations create will not have them.

It refuses when the branch's journal and the checkout's migrations differ,
and it refuses a migration that drops something schema.ts left out, since
the deployed app runs that migration on start. Recording takes the
migration lock and checks the journal again, and a run interrupted before
recording is finished by the next one. If anything fails after schema.ts is
written, schema.ts, its backup, database.d.ts and the migrations are put
back.

Signed-off-by: ditadi <victordperd@gmail.com>
@ditadi
ditadi force-pushed the stack/database-dev-flow/04-db-introspect branch from c3e1172 to a45e7df Compare October 20, 2026 12:23
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 41109406030 -R databricks/appkit -n appkit-template-0.89.0-pr.8d862d7-stack-database-dev-flow-04-db-introspect-670 -D appkit-pr-670 \
  && unzip -o "appkit-pr-670/appkit-template-0.89.0-pr.8d862d7-stack-database-dev-flow-04-db-introspect-670.zip" -d "appkit-pr-670" \
  && databricks apps init --template "appkit-pr-670"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+41 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.4 MB (+48 KB) 499 KB (+15 KB)
Type declarations 470 KB (+2.8 KB) 171 KB (+1.3 KB)
Source maps 2.7 MB (+93 KB) 930 KB (+27 KB)
Other 11 KB 3.7 KB
Total 4.6 MB (+143 KB) 1.6 MB (+43 KB)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB 2.5 KB 108 KB external 343 KB
./beta 108 KB (+40 B) 485 B 108 KB (+40 B) external 323 KB (+81 B)
./beta/database-tooling 37 KB (+9.2 KB) 0 B 37 KB (+9.2 KB) external 110 KB (+28 KB)
./testing 45 KB 32 KB (+2 B) 77 KB (+2 B) external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB (+1 B) 0 B 26 KB (+1 B) external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 88 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 178 B
./beta index.js lazy 115 B
./beta/database-tooling index.js initial 37 KB
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (+1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (+1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

ditadi and others added 14 commits October 30, 2026 16:08
Signed-off-by: ditadi <victordperd@gmail.com>

# Conflicts:
#	packages/appkit/src/database/tooling/migrate.ts
Introspect writes and discards migration files too, so it now holds
config/database/.appkit-db.lock for its whole run, like migrate. A second
command refuses before the catalog is read and changes nothing. The shared CLI
test mocks the beta/database-tooling path the CLI loads.

Signed-off-by: ditadi <victordperd@gmail.com>
Introspect replaced schema.ts without reading it, so every .private() column
came back public and every catalog foreign key became an API include. It now
loads the schema.ts it replaces first. A declared column keeps its privacy, and
a column the file does not declare is written .private() and named in a
warning. A foreign key the file does not declare refuses the run before
anything is written, and so does a schema.ts that does not load. Without a
schema.ts the first render is unchanged.

Signed-off-by: ditadi <victordperd@gmail.com>
With two catalog foreign keys to one table, the one first in constraint name
order took the relation, even when schema.ts declared the other. The refusal
then asked for a declaration the builder rejects as ambiguous. Foreign keys
schema.ts declares now go first.

Signed-off-by: ditadi <victordperd@gmail.com>
The refusals told developers to move schema.ts out of config/database, which
discards its .private() declarations and makes the next introspect write every
column public. A schema.ts that does not load now says to fix it and rerun. An
undeclared foreign key says to declare it with fk() in schema.ts, and names the
tables schema.ts does not declare yet.

Signed-off-by: ditadi <victordperd@gmail.com>
…ct reads

A schema.ts of another schema declares nothing in the one introspect reads, so
every column renders private and every foreign key is refused. The privacy
warning and the refusal now name the schema schema.ts declares, and say that
--schema-name with it refreshes the file.

Signed-off-by: ditadi <victordperd@gmail.com>
…it replaces

The docstring claimed a refresh never widens the API of the replaced
schema.ts. What render does is narrower: columns that file does not declare
render private, and foreign keys it does not declare are listed for
introspect to refuse.

Signed-off-by: ditadi <victordperd@gmail.com>
…file changed

The drift introspect reads before rendering, and the check markMigrationsApplied
repeats under the lock, both leave out migrations the branch recorded but whose
committed file changed since. The introspect test mock now records the real file
hash, so an edited file shows up the way it does on a branch.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
A migration the branch recorded, whose committed file changed since, ran in a
version the snapshot does not describe, so the migration introspect diffs from
that snapshot would be recorded against history the branch never had. The drift
now carries these migrations: introspect refuses before reading the catalog, and
markMigrationsApplied refuses again under the lock when the edit lands between
the two reads.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…keeps

Two foreign keys on the same column reach the render loop one after the other,
and the second writes its fk() over the first with no skip. With identity keys the
first rewrites the column's base type, so the second is skipped for a type reason
that names the wrong cause.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
A column with two foreign keys keeps the first one the render loop reaches, and
the other is skipped with that as its reason. Declared keys are sorted first, so a
refresh keeps the one schema.ts declares.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Without --schema-name, introspect reads the default app schema even when the
schema.ts it refreshes declares another one, so an app on public has to pass the
flag on every run. The explicit flag keeps overriding the declaration.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
… absent

An app whose schema.ts declares another schema, public say, is refreshed where
its tables live instead of in the default app schema. The explicit flag still
overrides the declaration, and a schema.ts that does not load is still refused.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
An enum label that is the empty string renders into schema.ts, which the builder
then rejects, so the whole run fails instead of skipping the column. A literal
default cast to a quoted type name, 'open'::app."order-status" as Postgres prints
it, is dropped. The catalog test shows the shape pg_get_expr returns.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
ditadi and others added 13 commits November 9, 2026 13:19
…trospect

An enum whose label is the empty string is skipped with that reason, since the
builder rejects it and the whole run failed on the generated schema.ts. The
literal-default parser takes the cast as pg_get_expr prints it, with quoted
names such as app."order-status", so the default is kept.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ce options

A DEFERRABLE unique constraint renders as a plain .unique() with nothing in the
report, unlike a deferrable foreign key. An identity or serial key whose sequence
has other options (START WITH 100, say) renders as an exact id()/bigid(), which
drizzle-kit writes with the defaults. The catalog test reads both from Postgres.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…s approximations

The catalog reads DEFERRABLE on unique constraints and the options of the
sequence a serial or identity column owns, as text so bigint bounds survive
JSON. The render lists a deferrable unique key as declared without DEFERRABLE,
as it already does for foreign keys, and a key whose sequence differs from the
defaults id()/bigid() declare with the clauses on each side.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
introspect writes schema.ts and schema.ts.bak with writeFileSync, which follows
a symbolic link at either path and overwrites whatever it points at.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ally

A new tooling/files.ts holds refuseSymlink and writeFileAtomic: lstat the
destination and refuse a link, then write a sibling temporary file and rename it
over, so a crash leaves the old file or the new one. introspect refuses a
symlinked schema.ts or schema.ts.bak before it reads anything, and writes and
restores both through the helper. migrations.ts drops its own copy of the rename
and writes the journal through the same helper.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
The introspect test's journal fake now answers the catalog lookup the way
#654's journal check reads it (relkind, owner, trust).

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
AutoCRUD is replaced by named operations, so introspect no longer guards the
generated HTTP surface. A foreign key the replaced schema.ts does not declare
renders with fk() instead of refusing the run. A column it does not declare
is written as the catalog has it, not .private(). A .private() the old file
declares is kept. Warnings about NOT NULL columns and dropped defaults name
app.database.<table>, which still inserts through schema.ts. The round-trip
test asserts the schema itself survives a refresh, with no dependency on the
crud capabilities.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…rivate() in introspect

Both policies existed for the generated HTTP routes, which named operations
replace. A foreign key the replaced schema.ts does not declare is rendered
with fk(), like a declared one; the refusal, the newRelations result and the
other-schema hint go. A column the old file does not declare is written as
the catalog has it. A .private() the old file declares is kept, so a
schema.ts that does not load still refuses the run. The NOT NULL and default
warnings name app.database.<table>, which still inserts through schema.ts.
The render tests ignore the constraint-name warning the fixture's
<column>_fkey names raise.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…it replaces

The help text described the private-by-default columns and the foreign key
refusal that introspect no longer has.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant