Skip to content

perf(semantics): cut analysis instructions by a quarter without changing output - #271

Merged
kvlonge merged 24 commits into
mainfrom
perf/analysis-hot-paths-v2
Nov 19, 2026
Merged

kvlonge merged 24 commits into
mainfrom
perf/analysis-hot-paths-v2

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

Callgrind showed the analysis workload spending much of its time on allocation, SipHash, repeated scope walks and linear name scans that produce nothing the findings need. Each change removes one of those costs and leaves every output identical.

This branch carries #270's twelve commits: on current main they take analysis 3.38% over its recorded count, which failed #270 in the merge queue. With these changes the combined count is 24% lower.

Changes

  • fix(semantics): bind join conditions, merged keys and named windows as each engine does #270's join and window binding fixes, unchanged (its twelve commits).
  • Normalise a single-spaced ASCII type spelling in one step, and read a bare type word's family without allocating.
  • Upper-case a Snowflake identifier in place.
  • Order findings without a position map when none needs one, and use FxHash otherwise.
  • Walk a node's children through the syntax visitor instead of collecting a vector per node.
  • Hash the scope rules' name sets with FxHash; none is iterated into an output.
  • Collect a SELECT's scope once for its structure rules.
  • Look an aggregate's name up in a set, upper-cased on the stack.
  • Append node and level findings instead of extending with them.
  • Record the lower count with make instruction-baseline.
  • docs: complete the 0.22.0 changelog's lint breaking-change entry.

Verification

  • Analysis instructions: 239,698,431 recorded, 247,824,122 for main with fix(semantics): bind join conditions, merged keys and named windows as each engine does #270, 182,168,067 here (−24.0%); parse unchanged (−0.04%).
  • Output identity: a scratch harness (not committed) printed statement_lineage_and_findings and statement_name_findings for all 144,772 statements of the repository's corpora that parse in Snowflake, DuckDB, Postgres and SQLite. Base and head output was byte-identical (97,512 findings in 22 codes, lineage included, no panics).
  • Read-only reviews of every perf commit found no input whose output, order or panic behaviour changes.
  • make check-ci passes on head 56952db.

A raw string in a semi-structured path key and in a resource gave the
same tree as the plain string: SELECT c1 : [ r'k' ] FROM t1 and
SELECT c1 : [ 'k' ] FROM t1, and ADD FILE r'v1' and ADD FILE 'v1' (the
same for ADD JAR and LIST FILE).

A path key now keeps r'k' as a raw string literal. A resource unquotes
only a plain string; a raw one is kept as written, as Spark's resource
commands unquote only text that begins with a quote.
…cing

A function's RETURNS TABLE ( ... ) kept its column list as the source
text, whitespace and comments included, so reformatting the statement
changed its tree and the formatter round trip failed:
RETURNS TABLE ( c1 INT , c2 STRING ) and RETURNS TABLE(c1 INT, c2 STRING)
gave two trees.

returns_table_body now holds the list's tokens separated by single
spaces, so spacing and comments no longer reach the tree while every
name and type still does.
An OVER window name was looked up among the WINDOW clause's names by its
exact key, so SELECT sum(b) OVER W AS x FROM t WINDOW w AS (...) gave a
false E232 "Unknown named window 'W'" in DuckDB and SQLite, which
resolve W to w. The check for which windows an OVER reaches already
compared names the profile's way.

The lookup now compares names with the profile's identifier rule too. A
DuckDB case covers a window named in another case.
A named window can extend another, as WINDOW w2 AS (zz ORDER BY b). When
an OVER read w2 and zz named no window, no finding was reported, while
PostgreSQL, DuckDB and SQLite refuse the query. The window an OVER
reads is now followed along its bases, and a base no WINDOW clause names
gives E232 "Unknown named window" at the base's name. As for a window's
keys, a window no OVER reaches is not checked.

DuckDB cases cover a direct and a two-step unknown base and an unused
window.
The shared parser built window names without their quoting: the name
after OVER, a base named inside OVER ( ... ) or a WINDOW definition, and
the WINDOW clause's own names all came out unquoted, and a quoted name
after WINDOW did not parse, because WINDOW followed by a quoted name read
as an alias. Semantics parses every dialect with this parser, so in
PostgreSQL SELECT sum(x) OVER "W" ... WINDOW w AS (...) resolved "W" to
w instead of reporting E232, and WINDOW "w" AS (...) was a parse error.

Each of these names now keeps whether it was quoted, and WINDOW followed
by a quoted name starts the clause. Formatting reads tokens and is
unchanged. A parser case covers each name position, and PostgreSQL cases
cover a quoted name that does not match, one that does, and an
unquoted name that folds to a quoted lower-case window.
…nces

A named window's PARTITION BY and ORDER BY keys were resolved, but not
the offsets of its frame, so WINDOW w AS (ORDER BY id ROWS BETWEEN nope
PRECEDING AND CURRENT ROW) reported nothing while the same frame written
inline in OVER gave E201.

The offsets of a used window's frame bounds now resolve with its keys,
in the same scope. DuckDB cases cover an unknown column in a start
offset, the inline form, and an unknown qualifier in an end offset.
…t reads

Where a SELECT's outputs can be read anywhere, an inline OVER key that
reads its own projection's alias gave E201, but the same key in a named
window did not: SELECT sum(amount) OVER w AS x FROM orders WINDOW w AS
(ORDER BY x) reported nothing, while DuckDB refuses both forms.

A used window's keys are now read once for each projection whose OVER
reaches it, with that projection's own alias hidden, as for an inline
window; a window only other clauses reach is read as before. A finding a
window shared by several projections gives is reported once. DuckDB
cases cover the own alias, a shared window ordered by one reader's
alias, and an unknown column in a shared window.
…put aliases

SQLite and BigQuery let a SELECT's clauses read its output aliases, and
the window keys of the select list were read the same way, so
SELECT quantity AS q, sum(quantity) OVER (PARTITION BY q) AS x FROM orders
reported nothing. SQLite 3.40 refuses it with "no such column: q",
inline or through a named window, and BigQuery's OVER cannot read an
output alias either.

A dialect now says whether the keys of a window the select list reads
may read output aliases. For SQLite and BigQuery they read only the
input columns; a window in ORDER BY still reads the aliases, as SQLite
accepts. Other dialects are unchanged. SQLite cases cover an inline and
a named window that refuse the alias and two ORDER BY windows that
accept it; a BigQuery case covers an inline window.
SQLite resolves output aliases in WHERE, GROUP BY, HAVING and ORDER BY,
and in subqueries read there, but not in the select list itself: both
`SELECT quantity AS q, q + 1 FROM orders` and a scalar subquery in the
select list reading `q` fail with "no such column". A new dialect rule,
select_list_reads_outputs, is false for SQLite only, so its select list
reads input columns and reports E201 for an output alias.
DuckDB refuses a NATURAL join whose sides have no column name in common
("No columns found to join on in NATURAL JOIN"), where PostgreSQL and
SQLite treat it as a cross join. On DuckDB, a NATURAL join of closed
sides that share no name now reports E201 at the joined item. Only the
join's own left side counts, not a comma neighbour, matching DuckDB 2.0.
…column on PostgreSQL and SQLite

PostgreSQL and SQLite read an unqualified USING or NATURAL key as the
merged column only when no FROM item outside that join offers the name:
`a JOIN b USING (id), c` and `a JOIN b USING (id) JOIN c ON ...` with a
`c.id` are "ambiguous" there, while DuckDB binds the merged key. A new
dialect rule, merged_keys_hide_other_items, is false for PostgreSQL and
SQLite, which now report E221 when a merged key and an uncovered item
both offer the name. A later USING or NATURAL join merging the key again
still reads it once.
… it see

An ON condition was checked against every FROM item of its level. In
PostgreSQL it sees only its own comma group, up to the join: `a, b JOIN
c ON a.id = c.id` is an "invalid reference to FROM-clause entry", also
from a subquery or a parenthesised join, and an ON condition reading a
later comma group or a later item of its own join chain is a "missing
FROM-clause entry". DuckDB lets it read every earlier item and rejects
only later ones. A new dialect rule, join_conditions_read, gives each
join condition the items it reaches: its own group on PostgreSQL, every
earlier item on DuckDB, and every item elsewhere, as before. The
existing name checks then report E222 or E201.
… step

type_family normalised every spelling by splitting on whitespace, collecting, joining and
lowercasing, four allocations per call, though spellings such as NUMBER(38,0) are almost always
printable ASCII with single spaces already. Such a spelling is now lowered in place; any other
takes the full path, so the result is unchanged. The analysis workload runs 4.3% fewer
instructions.
Snowflake's resolved_identifier built each unquoted name's upper-case copy byte by byte through
an iterator. It now copies the name once and upper-cases it in place, with the same result. The
analysis workload runs 0.65% fewer instructions.
…s one

ordered_findings mapped every node of the statement to its depth-first position on each call,
with SipHash, to sort findings by where they occur. When every finding is a name finding, all
sort at position zero in the order they were emitted, so the map went unread; a statement with
no finding built it too. Such a list is now returned as it is, and the map that remains uses
FxHash, as the crate's other position maps do. The analysis workload runs 4.5% fewer
instructions.
Most types that lineage carries are a single word such as BIGINT or VARCHAR.
For a spelling made only of ASCII letters, digits and underscores, normalising,
looking for arguments or a composite form, and stripping signedness all reduce
to the base family of the lower-cased word, so type_family now lowers such a
word on the stack and looks it up directly. Every other spelling takes the
existing path unchanged.
The scope walks behind type locations, aggregate placement, clause findings and
held-node checks asked each node for a fresh vector of its children, which
allocated once per visited node. They now visit the children in place through
the syntax visitor that children() itself uses, pushing them onto the walk's
own stack and reversing that slice where the walk reversed the vector, so every
walk sees the same nodes in the same order. The held-node check still stops
judging children once one holds.
The scope rules and the level findings kept their alias, name and table-index
sets in std maps, which hash every key with SipHash. They now use the FxHash
maps the rest of the crate already uses. Each set is only probed by key or
searched with any(), never iterated in an order that reaches a finding, so the
hasher cannot change what is reported.
The structure rules walked a SELECT's own scope for named windows and then
walked it again to find window functions written without OVER. They now
collect the scope once and read the same nodes, in the same order, for both.
Whether a named call is an aggregate was answered by comparing its name with
each of the 63 aggregate names in turn, ignoring ASCII case. Every listed name
is upper-case ASCII, so the name is now upper-cased on the stack and looked up
in a set of them, which gives the same answer for every spelling. A name longer
than the longest listed one is refused without a lookup.
Each node's checks return a vector that is usually empty, and node_findings
added each one to the node's findings with extend, which builds an iterator
for every vector, empty or not. It now appends them, which moves the same
findings in the same order without the iterator.
…them

select_findings gathered each node's, clause's and rule's findings with
extend, building an iterator for every vector it received, most of them empty.
It now appends them, which moves the same findings in the same order.
The entry stopped mid-sentence after "600". It now names the 600 `::`
casts that no longer lint in those dialects, matching the release notes.
The analysis workload now runs 182,168,067 instructions against the
recorded 239,698,431 (24.0% fewer); parse is unchanged within noise.
@kvlonge
kvlonge enabled auto-merge November 19, 2026 05:55
@kvlonge
kvlonge added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 3ce6c39 Nov 19, 2026
10 checks passed
@kvlonge
kvlonge deleted the perf/analysis-hot-paths-v2 branch November 19, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant