Skip to content

feat(appkit): add appkit db migrate - #666

Draft
ditadi wants to merge 37 commits into
stack/database-dev-flow/02-default-app-schemafrom
stack/database-dev-flow/03-db-migrate
Draft

ditadi wants to merge 37 commits into
stack/database-dev-flow/02-default-app-schemafrom
stack/database-dev-flow/03-db-migrate

Conversation

@ditadi

@ditadi ditadi commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR (generated by Stack)

Use this link to review incremental changes.


Stacked on #655. The design is #677.

What did you change, and why?

#654 applies committed migrations when the deployed app starts, so something has to write those migrations and try them first. A developer or an agent changes schema.ts, runs one command, reads the error and tries again, with no hand edits to drizzle's files.

appkit db migrate generates the next migration from config/database/schema.ts with drizzle-kit, regenerates shared/appkit-types/database.d.ts, and applies pending migrations to the branch .env points at. It applies only to your dev branch or this checkout's session branch. On any other branch it still generates, and says why nothing was applied. It warns about drops, renames, type changes and new NOT NULL columns, which break instances of the previous release. When the migration it wrote fails on the branch, it removes that file and restores the journal. Before connecting, it checks that PGHOST is a host of LAKEBASE_ENDPOINT. drizzle-kit is an optional peer dependency pinned to 0.31.10, and an ambiguous rename outside a TTY throws and writes nothing.

This revision changes these behaviors:

  • The tooling moves to @databricks/appkit/beta/database-tooling. The old @databricks/appkit/database-tooling subpath is gone, with no alias. oxlint still stops runtime code from importing the tooling or drizzle-kit.
  • Session branches are session-<user hash>-<id>, so a .env copied from another developer does not make their session branch yours.
  • The deploy boundary reads every target, every postgres resource of every app, the postgres_branch variable, the files include: names, and each target's .databricks/bundle/<target>/variable-overrides.json. A branch any of them names takes no writes.
  • A branch or database reference those files cannot resolve, such as an unset postgres_branch, makes migrate apply to no branch, and the message names the reference. So does an include: pattern with [, ] or \, or an overrides file that is not a JSON object.
  • migrate holds config/database/.appkit-db.lock for its whole run, including the discard of a failed migration. The lock file is created exclusively with mode 0600. A second appkit db command in the checkout fails and names the holder. AppKit never deletes a lock it does not hold, and the message gives the manual recovery step for a stale one.
  • bundle-size-baseline.json is regenerated. The packed tarball grows from 1,297,373 to 1,364,290 bytes (+5.16%), over the 5% budget, because of the CLI-only tooling entry.

appkit db stays hidden from appkit --help, and nothing a deployed app runs changes. Some messages tell you to run appkit db init, which #672 adds. The guard reads only this checkout's files. It does not see --var, BUNDLE_VAR_* or resources defined in Python, and it does not replace Lakebase permissions.

Review round, 2026-11-09:

  • The appkit lint overrides keep the base ban on direct @databricks/sdk-* imports; they had replaced it.
  • appkit db migrate flags ADD COLUMN … NOT NULL without a default as breaking the previous release, reports a kept migration's breaking statements when applying fails, restores the journal before deleting a discarded migration's files, and refuses to apply when the implicit app schema would shadow tables left in public.
  • A ${var…} standing in for an app's resources, one of them, or a postgres block is an unresolved binding, so the deploy-branch guard refuses instead of assuming the app binds no branch.

Fix round for fjakobs's review of #677, 2026-11-15:

  • appkit db migrate adds a second warning class: statements that may fail on rows the deploy branch has and a new branch does not, such as SET NOT NULL, a required ADD COLUMN, a new constraint, a unique index or a type change (L345).
  • The breaking-change warning says the previous release can still serve during a migrate on start, not that it always does (L350).

How do you know it works?

  • targets.test.ts, guard.test.ts, naming.test.ts and history-lock.test.ts cover every binding kind, overrides and included files, unresolved references, user-bound session names, and how the lock is taken, released and reported when stale. migrate.test.ts checks that a second concurrent run is refused and writes nothing.
  • migrate.pg.test.ts runs against Postgres. A NOT NULL column on a table with a row fails with 23502 and is removed. The fixed schema.ts then applies, and the row keeps its data.
  • At this PR's head, a clean build matched the new size baseline, typecheck passed, and the tooling and CLI db tests passed against a local PostgreSQL 14. The full vitest suite passed at an earlier revision of this PR and at the top of the stack (feat(appkit): add appkit db check #678). The packed tarballs export the same root and ./beta names as main, and ./beta/database-tooling is the only new subpath. The built CLI leaves db out of appkit --help. CI runs the full suite on this PR.
  • An earlier revision checked currentUser.me, an endpoint lookup and a branch lookup live and read-only. This revision was not run against a real Lakebase workspace.

Review round, 2026-11-09: each fix has a failing-then-passing test in its PR, and every layer's appkit and shared tests ran against Postgres 14 after the sync. At the top of the stack, pnpm build, pnpm test (6502 passed, 1 skipped), pnpm -r typecheck, pnpm check, pnpm knip and pnpm docs:build pass.

Fix round for fjakobs's review, 2026-11-15: each fix has a failing-then-passing test in its PR, and every layer's database, plugin and CLI tests ran against Postgres 17 after the sync. A throwaway Lakebase fork confirmed that the developer role can write the service principal's rows but cannot alter its tables. At the top of the stack, pnpm test (6602 passed, 1 skipped), pnpm -r typecheck, pnpm build and pnpm docs:build pass, and oxlint and oxfmt are clean on every changed file.

@ditadi
ditadi added this pull request to stack #667 October 13, 2026 13:34
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

@databricks/appkit

npm tarball (packed): 1.3 MB (+7 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 1.3 MB 485 KB
Type declarations 468 KB 170 KB (+1 B)
Source maps 2.6 MB 904 KB
Other 11 KB 3.7 KB
Total 4.4 MB 1.5 MB (+1 B)
Per-entry composition (own code — deps external (as shipped))
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
. 106 KB 2.5 KB 108 KB external 343 KB
./beta 108 KB 485 B 108 KB external 323 KB
./beta/database-tooling 28 KB 0 B 28 KB external 82 KB
./testing 45 KB 32 KB 77 KB external 223 KB
./tsdown 520 B 0 B 520 B external 813 B
./type-generator 26 KB 0 B 26 KB external 74 KB

Chunks:

Entry Chunk Load Size (gz)
. index.js initial 101 KB
. utils.js initial 4.6 KB
. remote-tunnel-manager.js lazy 2.5 KB
./beta beta.js initial 88 KB
./beta modular.js initial 6.0 KB
./beta stream-manager.js initial 5.9 KB
./beta service-context.js initial 4.2 KB
./beta databricks.js initial 3.3 KB
./beta client.js initial 647 B
./beta index.js initial 20 B
./beta supervisor-api.js lazy 192 B
./beta databricks.js lazy 178 B
./beta index.js lazy 115 B
./beta/database-tooling index.js initial 28 KB
./testing manifest.js initial 32 KB
./testing index.js initial 10 KB
./testing wide-event-emitter.js initial 2.9 KB
./testing index.js lazy 28 KB
./testing remote-tunnel-manager.js lazy 2.5 KB
./testing utils.js lazy 1.8 KB
./tsdown index.js initial 520 B
./type-generator index.js initial 26 KB

@databricks/appkit-ui

npm tarball (packed): 397 KB (+4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist raw gzip
JS (runtime) 429 KB 147 KB
Type declarations 267 KB 98 KB (+1 B)
Source maps 853 KB 286 KB
CSS 16 KB 3.2 KB
Total 1.5 MB 534 KB (+1 B)
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
Entry Initial (gz) Lazy (gz) Total (gz) node_modules (min) Own code (min)
./js 5.3 KB 49 KB 55 KB 208 KB 14 KB
./js/beta 2.1 KB 0 B 2.1 KB 0 B 5.0 KB
./react 433 KB 49 KB 481 KB 1.3 MB 177 KB
./react/beta 4.9 KB 0 B 4.9 KB 0 B 12 KB

Chunks:

Entry Chunk Load Size (gz)
./js index.js initial 5.2 KB
./js chunk initial 120 B
./js apache-arrow lazy 49 KB
./js/beta beta.js initial 2.1 KB
./react index.js initial 431 KB
./react tslib initial 2.1 KB
./react apache-arrow lazy 49 KB
./react/beta beta.js initial 4.9 KB

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 41109403208 -R databricks/appkit -n appkit-template-0.89.0-pr.7490e85-stack-database-dev-flow-03-db-migrate-666 -D appkit-pr-666 \
  && unzip -o "appkit-pr-666/appkit-template-0.89.0-pr.7490e85-stack-database-dev-flow-03-db-migrate-666.zip" -d "appkit-pr-666" \
  && databricks apps init --template "appkit-pr-666"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

`appkit db migrate` generates a migration from config/database/schema.ts
with drizzle-kit, regenerates shared/appkit-types/database.d.ts, and
applies pending migrations to the branch .env points at. It applies only
to a branch created for this developer (dev-…) or this checkout
(session-…); on any other branch, including the project default and any
branch databricks.yml deploys to, it generates and says why nothing was
applied. It connects only after the Lakebase API confirms that PGHOST is
a host of LAKEBASE_ENDPOINT.

The logic lives in a new @databricks/appkit/database-tooling entry point
that the shared CLI loads lazily, so the runtime never imports drizzle-kit.
drizzle-kit is an optional peer dependency: apps that generate migrations
install it as a dev dependency, and the command says how when it is
missing. `appkit db` stays hidden from --help until the database plugin
is offered by `databricks apps init`.

Signed-off-by: ditadi <victordperd@gmail.com>
A generated migration that fails to apply is removed and the journal
restored, so a fix in schema.ts replaces it instead of stacking behind it.

Also: refuse a drizzle-kit other than the pinned one or one loading another
drizzle-orm, parse the branch and endpoint responses the guard decides on,
type the CLI against a contract appkit's typecheck enforces, and enforce
the drizzle-kit and tooling import boundary with oxlint.

Signed-off-by: ditadi <victordperd@gmail.com>
…nch pool

resolveEnvBranch returns the branch kind (default, protected, deploy
target, dev, session, other), and takesWrites is the one rule for which
kinds take writes. It needs PGHOST, PGDATABASE and LAKEBASE_ENDPOINT,
like the pool. withBranchPool opens and always closes the pool. The
drizzle-kit v7 file names live only in migrations/paths.ts.

Signed-off-by: ditadi <victordperd@gmail.com>
…aks the previous release

Apps keeps instances of the previous release serving while the new one
migrates on start, and a code rollback cannot undo a migration. When the
generated migration drops, renames, retypes or makes a column NOT NULL,
the report lists those statements and the CLI says to ship code that
works with both schemas first.

Signed-off-by: ditadi <victordperd@gmail.com>
…lient

Signed-off-by: ditadi <victordperd@gmail.com>
ditadi added 10 commits October 30, 2026 15:19
Signed-off-by: ditadi <victordperd@gmail.com>

# Conflicts:
#	pnpm-lock.yaml
…tabase-tooling

The CLI-only tooling was exported as @databricks/appkit/database-tooling, outside the beta
boundary. Export it only as @databricks/appkit/beta/database-tooling and drop the old subpath.
Migrate the CLI loader, its ambient declaration, the runtime import restriction, the test
mock and the comments that name the path.

Signed-off-by: ditadi <victordperd@gmail.com>
A session branch was session-<id>, and the guard accepted it for anyone whose .env carried
that id, so a copied .env let another developer write to it. Name session branches
session-<user hash>-<id>, the hash dev branch names already use, so only the user who
created the branch passes the guard.

Signed-off-by: ditadi <victordperd@gmail.com>
… not name

The guard read only each target's postgres_branch variable and dropped references it could
not resolve, so a branch bound through an app's postgres resource, an included file or an
unresolved reference could pass as a dev branch. Read every postgres binding of every app
and target, including included files, into one deploy boundary that keeps unresolved branch
and database references. While any reference is unresolved, no branch takes writes.

Signed-off-by: ditadi <victordperd@gmail.com>
Generation read the journal, waited for drizzle-kit, then wrote the next SQL, snapshot and
journal, and a failed apply restored the journal read before. Two commands in one checkout
could overwrite or remove each other's migration. appkit db migrate now holds
config/database/.appkit-db.lock, created exclusively, from before it reads the history until
it finishes. Generation and discard take the held lock, and discard restores the journal
that generation itself replaced. A lock left by a killed process is reported as stale with
the manual recovery step, never taken over.

Signed-off-by: ditadi <victordperd@gmail.com>
Document where the beta tooling lives, which branches migrate writes to, the refusal while
databricks.yml has an unresolved branch or database reference, the file-only limits of that
check, and how to recover from a stale config/database/.appkit-db.lock.

Signed-off-by: ditadi <victordperd@gmail.com>
The Databricks CLI applies .databricks/bundle/<target>/variable-overrides.json ahead of a
target's variables and their defaults, so an override could deploy the app to a developer's
dev branch while the guard still classified it as dev. Read each target's override file and
add its bindings to the deploy boundary next to databricks.yml's own. A value that is not a
resolvable string is unresolved, and a file that is not a JSON object throws.

Signed-off-by: ditadi <victordperd@gmail.com>
includedFiles matched only * and ?, so an include: pattern with [, ] or \\ matched no file
and the bindings in the files it named were left out of the deploy boundary. Throw
"Cannot read include pattern ..." for such a segment instead.

Signed-off-by: ditadi <victordperd@gmail.com>
A lock whose pid runs on this host was reported only as busy, so a pid reused after a reboot
left no way forward. End the message with the stale-lock recovery step, and say in the docs
when each message appears.

Signed-off-by: ditadi <victordperd@gmail.com>
Regenerated with pnpm clean && pnpm build && pnpm size:baseline. Against main's baseline the
@databricks/appkit packed tarball grows from 1297373 to 1364290 bytes (+66917, +5.16%), over
the 5% budget. The growth is the CLI-only @databricks/appkit/beta/database-tooling entry
(28 KB gzipped) and the database tooling it ships. @databricks/appkit-ui is unchanged at
406741 bytes.

Signed-off-by: ditadi <victordperd@gmail.com>
ditadi and others added 22 commits November 9, 2026 13:06
The two no-restricted-imports overrides for packages/appkit/src and its
database tooling replaced the base rule's options, which allowed direct
@databricks/sdk-* imports anywhere in appkit. Each override now carries the
base group and message alongside its own patterns.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…evious release

Instances still running the old code insert rows without the column, and a
populated table rejects the migration. A default, a generated expression or
an identity fills the column, so those additions stay safe.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
BREAKS_PREVIOUS_RELEASE matched only drops, renames, type changes and SET
NOT NULL. ADD COLUMN ... NOT NULL with no DEFAULT, generated expression or
identity breaks the previous release the same way, so the rules are now
predicates and that case is one of them.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…missing files

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…on's files

discardMigrationFiles deleted the SQL and snapshot first and restored the
journal last, so a process killed in between left a journal naming files
that no longer existed. The journal is restored (or removed) first, through
the same atomic rename, and the unreferenced files go after it.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…at it breaks

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ting it

runMigrate computed the breaking statements only after a successful apply.
When branch resolution or the connection failed (a 403, say) the migration
stayed on disk, the warning was never printed, and the next run generated
nothing to warn about. The statements are now computed right after
generation; the failure throws UnappliedMigrationError carrying the kept
migration and them, and the CLI prints the same warning it prints on
success.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…unresolved

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…s as unresolved

bundleApps read a `${var…}` string where an app's resources list, one of its
entries, or a postgres block should be as "no postgres binding", so the
deploy-branch guard assumed the app bound no branch. Such a reference is now
an unresolved binding, and the guard refuses to apply until it has a value.
The docs name the case.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…ling

knip flags it as an unused export: the CLI reads its shape, not the class.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…n public

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…in public

With the implicit app schema, applying would create empty copies of tables
that still live in public; the deployed app already refuses this on start.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…e reader

The fail-closed fix flattened each resource into separate branch and
database values; keep one PostgresBinding per resource (with a whole
marker for a ${…} standing in for the structure), which later readers
that count an app's resources need.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…xisting rows

A new dev branch has no rows, so SET NOT NULL, a required column, a new
constraint, a unique index and a type change pass there and can fail on the
deploy branch. The report and the CLI must carry them as their own class,
apart from what breaks the previous release.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
…sting rows

Postgres checks SET NOT NULL, a NOT NULL column without a default, a new
constraint, a unique index and a type cast against the rows already in the
table. A new dev branch has none, so the migration passes there and can fail
on deploy. Classify them as a second table of predicates next to
BREAKS_PREVIOUS_RELEASE, carry them in the report and the kept-migration
error under dataDependent, and print one warning per class.

An identity column added inline with PRIMARY KEY is exempt: the identity
fills every row, and it is what id() generates for an existing table.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>
# Conflicts:
#	packages/appkit/src/plugins/database/schema-check.ts
Eight comments this round added or changed in appkit db migrate, its
statement classes and the shared CLI output restate the name, type or
body next to them, so they go. The note on addsConstrainedColumn stays.
It records the Postgres semantics behind the AS IDENTITY exclusion,
which the code cannot show.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: ditadi <victordperd@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant