Repository navigation
perf(semantics): cut analysis instructions by a quarter without changing output - #271
Merged
Merged
Conversation
A raw string in a semi-structured path key and in a resource gave the same tree as the plain string: SELECT c1 : [ r'k' ] FROM t1 and SELECT c1 : [ 'k' ] FROM t1, and ADD FILE r'v1' and ADD FILE 'v1' (the same for ADD JAR and LIST FILE). A path key now keeps r'k' as a raw string literal. A resource unquotes only a plain string; a raw one is kept as written, as Spark's resource commands unquote only text that begins with a quote.
…cing A function's RETURNS TABLE ( ... ) kept its column list as the source text, whitespace and comments included, so reformatting the statement changed its tree and the formatter round trip failed: RETURNS TABLE ( c1 INT , c2 STRING ) and RETURNS TABLE(c1 INT, c2 STRING) gave two trees. returns_table_body now holds the list's tokens separated by single spaces, so spacing and comments no longer reach the tree while every name and type still does.
An OVER window name was looked up among the WINDOW clause's names by its exact key, so SELECT sum(b) OVER W AS x FROM t WINDOW w AS (...) gave a false E232 "Unknown named window 'W'" in DuckDB and SQLite, which resolve W to w. The check for which windows an OVER reaches already compared names the profile's way. The lookup now compares names with the profile's identifier rule too. A DuckDB case covers a window named in another case.
A named window can extend another, as WINDOW w2 AS (zz ORDER BY b). When an OVER read w2 and zz named no window, no finding was reported, while PostgreSQL, DuckDB and SQLite refuse the query. The window an OVER reads is now followed along its bases, and a base no WINDOW clause names gives E232 "Unknown named window" at the base's name. As for a window's keys, a window no OVER reaches is not checked. DuckDB cases cover a direct and a two-step unknown base and an unused window.
The shared parser built window names without their quoting: the name after OVER, a base named inside OVER ( ... ) or a WINDOW definition, and the WINDOW clause's own names all came out unquoted, and a quoted name after WINDOW did not parse, because WINDOW followed by a quoted name read as an alias. Semantics parses every dialect with this parser, so in PostgreSQL SELECT sum(x) OVER "W" ... WINDOW w AS (...) resolved "W" to w instead of reporting E232, and WINDOW "w" AS (...) was a parse error. Each of these names now keeps whether it was quoted, and WINDOW followed by a quoted name starts the clause. Formatting reads tokens and is unchanged. A parser case covers each name position, and PostgreSQL cases cover a quoted name that does not match, one that does, and an unquoted name that folds to a quoted lower-case window.
…nces A named window's PARTITION BY and ORDER BY keys were resolved, but not the offsets of its frame, so WINDOW w AS (ORDER BY id ROWS BETWEEN nope PRECEDING AND CURRENT ROW) reported nothing while the same frame written inline in OVER gave E201. The offsets of a used window's frame bounds now resolve with its keys, in the same scope. DuckDB cases cover an unknown column in a start offset, the inline form, and an unknown qualifier in an end offset.
…t reads Where a SELECT's outputs can be read anywhere, an inline OVER key that reads its own projection's alias gave E201, but the same key in a named window did not: SELECT sum(amount) OVER w AS x FROM orders WINDOW w AS (ORDER BY x) reported nothing, while DuckDB refuses both forms. A used window's keys are now read once for each projection whose OVER reaches it, with that projection's own alias hidden, as for an inline window; a window only other clauses reach is read as before. A finding a window shared by several projections gives is reported once. DuckDB cases cover the own alias, a shared window ordered by one reader's alias, and an unknown column in a shared window.
…put aliases SQLite and BigQuery let a SELECT's clauses read its output aliases, and the window keys of the select list were read the same way, so SELECT quantity AS q, sum(quantity) OVER (PARTITION BY q) AS x FROM orders reported nothing. SQLite 3.40 refuses it with "no such column: q", inline or through a named window, and BigQuery's OVER cannot read an output alias either. A dialect now says whether the keys of a window the select list reads may read output aliases. For SQLite and BigQuery they read only the input columns; a window in ORDER BY still reads the aliases, as SQLite accepts. Other dialects are unchanged. SQLite cases cover an inline and a named window that refuse the alias and two ORDER BY windows that accept it; a BigQuery case covers an inline window.
SQLite resolves output aliases in WHERE, GROUP BY, HAVING and ORDER BY, and in subqueries read there, but not in the select list itself: both `SELECT quantity AS q, q + 1 FROM orders` and a scalar subquery in the select list reading `q` fail with "no such column". A new dialect rule, select_list_reads_outputs, is false for SQLite only, so its select list reads input columns and reports E201 for an output alias.
DuckDB refuses a NATURAL join whose sides have no column name in common
("No columns found to join on in NATURAL JOIN"), where PostgreSQL and
SQLite treat it as a cross join. On DuckDB, a NATURAL join of closed
sides that share no name now reports E201 at the joined item. Only the
join's own left side counts, not a comma neighbour, matching DuckDB 2.0.
…column on PostgreSQL and SQLite PostgreSQL and SQLite read an unqualified USING or NATURAL key as the merged column only when no FROM item outside that join offers the name: `a JOIN b USING (id), c` and `a JOIN b USING (id) JOIN c ON ...` with a `c.id` are "ambiguous" there, while DuckDB binds the merged key. A new dialect rule, merged_keys_hide_other_items, is false for PostgreSQL and SQLite, which now report E221 when a merged key and an uncovered item both offer the name. A later USING or NATURAL join merging the key again still reads it once.
… it see An ON condition was checked against every FROM item of its level. In PostgreSQL it sees only its own comma group, up to the join: `a, b JOIN c ON a.id = c.id` is an "invalid reference to FROM-clause entry", also from a subquery or a parenthesised join, and an ON condition reading a later comma group or a later item of its own join chain is a "missing FROM-clause entry". DuckDB lets it read every earlier item and rejects only later ones. A new dialect rule, join_conditions_read, gives each join condition the items it reaches: its own group on PostgreSQL, every earlier item on DuckDB, and every item elsewhere, as before. The existing name checks then report E222 or E201.
… step type_family normalised every spelling by splitting on whitespace, collecting, joining and lowercasing, four allocations per call, though spellings such as NUMBER(38,0) are almost always printable ASCII with single spaces already. Such a spelling is now lowered in place; any other takes the full path, so the result is unchanged. The analysis workload runs 4.3% fewer instructions.
Snowflake's resolved_identifier built each unquoted name's upper-case copy byte by byte through an iterator. It now copies the name once and upper-cases it in place, with the same result. The analysis workload runs 0.65% fewer instructions.
…s one ordered_findings mapped every node of the statement to its depth-first position on each call, with SipHash, to sort findings by where they occur. When every finding is a name finding, all sort at position zero in the order they were emitted, so the map went unread; a statement with no finding built it too. Such a list is now returned as it is, and the map that remains uses FxHash, as the crate's other position maps do. The analysis workload runs 4.5% fewer instructions.
Most types that lineage carries are a single word such as BIGINT or VARCHAR. For a spelling made only of ASCII letters, digits and underscores, normalising, looking for arguments or a composite form, and stripping signedness all reduce to the base family of the lower-cased word, so type_family now lowers such a word on the stack and looks it up directly. Every other spelling takes the existing path unchanged.
The scope walks behind type locations, aggregate placement, clause findings and held-node checks asked each node for a fresh vector of its children, which allocated once per visited node. They now visit the children in place through the syntax visitor that children() itself uses, pushing them onto the walk's own stack and reversing that slice where the walk reversed the vector, so every walk sees the same nodes in the same order. The held-node check still stops judging children once one holds.
The scope rules and the level findings kept their alias, name and table-index sets in std maps, which hash every key with SipHash. They now use the FxHash maps the rest of the crate already uses. Each set is only probed by key or searched with any(), never iterated in an order that reaches a finding, so the hasher cannot change what is reported.
The structure rules walked a SELECT's own scope for named windows and then walked it again to find window functions written without OVER. They now collect the scope once and read the same nodes, in the same order, for both.
Whether a named call is an aggregate was answered by comparing its name with each of the 63 aggregate names in turn, ignoring ASCII case. Every listed name is upper-case ASCII, so the name is now upper-cased on the stack and looked up in a set of them, which gives the same answer for every spelling. A name longer than the longest listed one is refused without a lookup.
Each node's checks return a vector that is usually empty, and node_findings added each one to the node's findings with extend, which builds an iterator for every vector, empty or not. It now appends them, which moves the same findings in the same order without the iterator.
…them select_findings gathered each node's, clause's and rule's findings with extend, building an iterator for every vector it received, most of them empty. It now appends them, which moves the same findings in the same order.
The entry stopped mid-sentence after "600". It now names the 600 `::` casts that no longer lint in those dialects, matching the release notes.
The analysis workload now runs 182,168,067 instructions against the recorded 239,698,431 (24.0% fewer); parse is unchanged within noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Callgrind showed the analysis workload spending much of its time on allocation, SipHash, repeated scope walks and linear name scans that produce nothing the findings need. Each change removes one of those costs and leaves every output identical.
This branch carries #270's twelve commits: on current main they take analysis 3.38% over its recorded count, which failed #270 in the merge queue. With these changes the combined count is 24% lower.
Changes
make instruction-baseline.docs:complete the 0.22.0 changelog's lint breaking-change entry.Verification
statement_lineage_and_findingsandstatement_name_findingsfor all 144,772 statements of the repository's corpora that parse in Snowflake, DuckDB, Postgres and SQLite. Base and head output was byte-identical (97,512 findings in 22 codes, lineage included, no panics).make check-cipasses on head 56952db.