Repository navigation
Open console query results in Positron's Data Explorer - #595
samclark2015 wants to merge 8 commits into
Conversation
|
Most of the logic of this PR is implemented in the Jupyter kernel and I'm not super fond of that if it can be avoided at all. For the kernel to contain a module so closely tied to positron feels like a code smell. Can you take another round at this with the goal of keeping it as confined as possible to the VS code extension? Also, we plan to spin out the VS Code extension to its own repo that you can have more control over once this is in (just an FYI :-) ) |
|
Sweet! Thanks for this feedback. I churned on it with the 🤖 and came up with this proposal, which moves Positron-specific code into the extension. Kernel additions are minimal and client-agnostic. Let me know how this sounds, and I'll spin up a revision. Proposal: the kernel offers a general results API, and the extension serves the Data ExplorerKernel: a general results API, with no Positron logic.
Extension: everything specific to Positron's Data Explorer.
Why this shape
Tradeoffs
ScopeBoth directives go. The extension calls |
|
I think this is a better shape in general. If possible, it would be nice to reuse the cache mechanism already in place in ggsql, however, care should be taken to not enforce a cached reader on the user (because that change the dialect for them). So, the cache reader should only be used for the results API |
Console query results get a link that opens them in Positron's Data Explorer. The kernel keeps each result in a dedicated in-memory DuckDB results cache (sharing `CacheTables` with `CachingReader`, with pinning so open results are not evicted; the user's reader is never wrapped) and exposes a frontend-agnostic results API over `call_method`. The VS Code extension registers the Data Explorer RPC handler, adds the console link, and pages, sorts, filters and profiles results through that API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
307d520 to
6711156
Compare
|
The shape of this refactor is great - basically how I envisioned it. Below are some correctness issues found and summarised together with Kimi for you to look through: High
Medium
Low
Test gaps
|
- A failed sort or filter leaves its view as it was: the new order table is built under a new name and only then replaces the old one. - Pages are bounded (MAX_PAGE_ROWS), scattered indices fetch just those rows, and an index past the end is an error rather than a shifted column. - One byte budget (--results-max-bytes) covers unopened results, cached ones including pinned, and order tables. Keeping results is opt-in (--keep-results / GGSQL_KEEP_RESULTS), which the extension sets for console sessions. - result_sql quotes for the session dialect, escapes LIKE with '!', keeps a cell's earlier statements, and leaves out repeated column names. Regex support comes from SqlDialect::sql_regex_match. - Views close with the UI comm. The extension releases a view when the Data Explorer fails to open, ignores requests after close, and never calls a session that is down. - CacheTables takes a byte budget, keeps pins in memory, and has a narrower API; the kernel builds its own DuckDB backend for it. - kind_of matches DuckDB type names exactly. - The DuckDB Arrow appender change moves to its own PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte budget counts the results kept, not the working memory DuckDB needs to sort or profile them. With no memory limit of its own, the cache's DuckDB could use most of the machine's memory before failing, so the process could be killed first. It now gets a hard limit of twice the budget (at least 64 MB), so a sort that would need more fails with an error and leaves the view as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@thomasp85 Thanks for this!! Everything is addressed, and the appender is split out into #617. Some differences from your suggestions:
|
|
Thanks! Two notes:
Also, but less blocking I'd be really happy if you leaned more into the dialect system in |
- Make room only once a new result is known to fit beside open views, so an oversized insert drops nothing (`CacheTables::pinned_bytes`). - Treat a cell that does not parse cleanly as one statement in `result_sql`, rather than splitting around ERROR nodes. - Write `result_sql` through the dialect: string literals (`sql_string_literal`), LIKE (`sql_like`), NULL ordering (`sql_order_term`) and typed date/time literals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Close every view when a new UI comm opens, so a frontend that reloaded without closing its old comm releases its pins. - Cap exports at 100,000 rows, separately from the page limit. - Write timezone-aware timestamps as timestamps in UTC. - Reserve budget for the cached copy's row column, and for every order-table rebuild, including clearing a sort or filters. - Leave an already open Data Explorer's view alone when showing it fails, and forget closed datasets with their session. - Regex search for ten more dialects; leave a non-query last statement unwrapped in `result_sql`; create the results store on first use. - Expect `NULLS LAST` in the integration test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Making room can drop older unopened results from the front of the queue, so putting the result back at its old index could pass the end and panic. It goes back at its index or, past the end, last. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Validate export columns before anything else, so a request with no rows and a bad column is an error rather than a kernel panic. - Cut a console table short only when its result was kept, since only then does a link to the rest follow it. - Keep Positron out of the kernel's results wording; budget errors now say how much open views hold. - Case-sensitive LIKE for SQLite (GLOB), MySQL (binary) and SQL Server (case-sensitive collation); `sql_is_empty_string`, NULL on Oracle. - CacheTables: `put`/`put_pinned`, `contains`, `evict_to` returns the evicted keys, only an existing entry can be pinned; `MemoEntry` and `lookup` are crate-private. - Format timezone-aware timestamps in `array_util::value_to_display`. - Share `kind_of` with the Connections-pane explorer, fixing its substring type matching. - Internal names through `ggsql::naming`; numbers read as numbers in profiles; bounded profile requests. - `ggsql.resultsMemory` setting; overflow-safe `parse_human_bytes`. - Drop the results benchmark example from this PR. - Warm the renderer in a timing-sensitive plot test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- `kind_of` unwraps ClickHouse's `Nullable(…)` and `LowCardinality(…)` and knows ClickHouse, Oracle, DataFusion and SQLite names the shared exact match had dropped. - Oracle "is empty" stays `= ''`, as the view on DuckDB shows it; only "not empty" becomes `IS NOT NULL`. - `ggsql.resultsMemory` drops a byte count past `u64`, which would stop the kernel starting. - Tighten a cache test, document MySQL's byte-wise `_`, and note the time-zone timestamp display in the CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses posit-dev/positron#16137 ("ggsql: View result of SQL query in data explorer"). No Positron changes are needed.
Following review, the Positron-specific code lives in the extension. The kernel gains a small results API that any frontend can use, and it keeps results in a cache of its own, so the session's reader and dialect are never changed.
How it works
Kernel: results API (
ggsql-jupyter/src/results/). Results are kept only when the frontend asks, with--keep-results/GGSQL_KEEP_RESULTS; the extension sets it for console sessions. A kept result's key goes in theexecute_resultmetadata as{"ggsql": {"result": key}}. The calls are documented inmethods.rs; nothing in the module knows about Positron. The only Positron-specific part of the kernel is the route: acall_methodrequest on the UI comm whose method starts withresult_. Closing the UI comm closes every view, so a frontend that reloads or crashes doesn't leave results pinned.Cache reuse.
CachingReader's table bookkeeping is pulled out intoCacheTables(src/reader/cache.rs), which takes a byte budget and gains pinning (kept in memory, not in__ggsql_cache_meta__). Its public surface isput/put_pinned,contains,set_pinned(an existing entry only), andevict_to/total_bytes/pinned_bytesfor a caller with a wider budget,evict_toreturning the keys it evicted. The results API uses one on its own in-memory DuckDB, so the user's reader is never wrapped, and keeps its own budget across the cache and what lives outside it (unopened results, order tables). A result is copied in on first open and stays pinned while a view of it is open.Sorting, filtering and paging. Sorting and filtering build an order table of row ids, so scrolling never sorts again. A new order table replaces the old one only once it's built, so a sort or filter that fails leaves the view as it was. A page is at most 10,000 rows, and a row index outside the view is an error rather than a shifted column.
One memory budget.
--results-max-bytes/GGSQL_RESULTS_MAX_BYTES(default 1 GB; the extension'sggsql.resultsMemorysetting) covers unopened results, cached results (pinned ones included) and order tables. Closed results go first, then the oldest unopened ones, and only once the new result is known to fit beside open views, so one that can't fit drops nothing. Opening a result and every order-table rebuild reserve their room first. The cache's DuckDB has a hard memory limit of twice the budget and never spills to disk; running out of room is an error rather than a crash.Extension: Data Explorer backend (
ggsql-vscode/src/dataExplorer/). Registers withpositron.dataExplorer.registerRpcHandlerand translates Positron's protocol to the results API: paging, sorting, row filters, column search, profiles (summary stats, histograms, frequency tables), export (CSV/TSV/HTML) and Convert to Code. Copying or exporting is capped at 100,000 rows. Convert to Code writes SQL for the session's own database, not the cache, through its dialect: identifiers, string literals,LIKEescapes, regular expressions (now for ten more backends) and typed date/time literals are written for that backend, and sort keys put NULLs last as the view does (newSqlDialectmethodssql_string_literal,sql_like,sql_order_termandsql_is_empty_string; a case-sensitiveLIKEis case-sensitive on SQLite, MySQL and SQL Server too). A multi-statement cell keeps its earlier statements (or is kept whole when it doesn't parse cleanly), a last statement that isn't a query is left unwrapped, and a filter the database can't run is left out with a comment. A session that has ended or restarted isn't called, so an open tab shows as disconnected instead of hanging, and a kernel without the results API is reported as too old.Extension: the link (
ggsql-vscode/src/resultLinks.ts). Each ggsql session is wrapped. After a marked result in a console session, the extension adds an OSC 8 "Open in Data Explorer" hyperlink tox-ggsql-result:<session>/<key>, and the wrapper'sopenResourcehandles the click. Clicking a link again focuses the open tab, or reopens one that was closed.Shorter console tables. Like R's tibble, a kept result of up to 20 rows shows in full and a longer one shows its first 10, so the link stays in view. A result that couldn't be kept (budget full, or a frontend that didn't ask) has no link and still shows up to 100 rows. The
ggsql.consoleRowssetting changes the number; it reaches the kernel asGGSQL_CONSOLE_ROWS. Notebooks still show up to 100 rows.Timestamps with a time zone are written in UTC (
…Z) by core'sarray_util::value_to_display, wherever ggsql shows a value as text, rather than as their Arrow type name.Why DuckDB for the results cache
A benchmark (kept out of this PR, available as a follow-up), release build, Apple Silicon (14 cores), 10M rows (617 MB of Arrow data). The alternative was to sort and filter the Arrow data directly with row indices.
DuckDB uses about twice the memory but sorts 4–8× faster, and a multi-second wait on every sort click isn't acceptable for large results. Memory levels off near 2 GB across 30 open/close cycles rather than growing. #617 speeds up the copy with DuckDB's Arrow appender; it's a separate PR because it affects every DuckDB user.
Not in this PR
format_options, Positron's number-formatting settings.$$-quoted body containing;(e.g.CREATE FUNCTION … AS $$ …; $$) is still split into statements by Convert to Code, because the grammar has no dollar-quoted strings.data_explorer.rs) now shares the results API's column typing (kind_of), but still has its own profile SQL; moving it onto the results API's views is a follow-up.Testing
cargo test -p ggsql-jupyter: 136 unit tests pass, including the results store, views, filters, profiles, eviction and pinning, the memory budget, failed sorts and filters, page bounds, and Convert to Code SQL per dialect (quoting, string literals,LIKE, NULL ordering and date/time literals, several run against DuckDB and SQLite).src/reader/odbc/mod.rshas a one-line clippy fix (borrow_deref_ref) thatcargo clippyflags onmaintoo.buildjob's SIGSEGV in the hephaestus render tests also fails onmain's recent runs; it is not from this PR.cargo test -p ggsql --lib: passes except two parser comment tests (test_extract_sql_from_first_skips_line_comment,test_validate_visualise_in_comment_is_valid) that also fail onmain.ggsql-jupyter/tests/test_integration.py:call_methodround trips for open, filter, sort, page, profile, export, SQL and close; an unknown key or method is a JSON-RPC error; results are kept only when asked for; closing the UI comm, or opening a new one, closes its views; and the session's database gains no tables.ggsql-vscode: extension tests pass in CI on macOS, Linux and Windows, including the Data Explorer backend against a recordedcallMethodstand-in, the session wrapper, opening a result, session end and restart, and concurrent requests sharing one view.Demo
ggsql-data-explorer.mp4
🤖 Generated with Claude Code