Skip to content

feat(kernel): honor _port and _connection_uri on use_kernel connections - #1071

Open
cathleeny wants to merge 15 commits into
mainfrom
cathleeny/feat/kernel-honor-port-uri-parity
Open

cathleeny wants to merge 15 commits into
mainfrom
cathleeny/feat/kernel-honor-port-uri-parity

Conversation

@cathleeny

@cathleeny cathleeny commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

On the kernel backend (use_kernel=True), _port and _connection_uri are now honored the way Thrift honors them. Before this PR the kernel path ignored both. Thrift is unchanged.

Behavior

  • Port:
    • A non-default _port is added to a bare server_hostname.
    • A port already in server_hostname wins. If _port is set to a different non-443 value, the driver logs a warning.
    • A falsy _port means the default.
    • An invalid _port raises ValueError; Thrift fails on it too.
  • Connection URI: _connection_uri (http(s)://host[:port]/path?query) sets the host, port and path, and wins over _port. A URI-only connection works.
  • Rejected when a port has to be added: a host that has a path, query or fragment, credentials, a malformed port, or an unbracketed IPv6 address.
  • OAuth guard: with OAuth, a _connection_uri whose origin differs from server_hostname raises before any request. OAuth here includes an access_token that is a JWT from another issuer, or any token sent with identity_federation_client_id. PAT is unaffected.
  • Implementation: host parsing uses urllib.parse, and error messages redact credentials.

Testing

  • Unit: test_session.py (200) and test_kernel_client.py (282) pass, plus the real-kernel loopback tests. mypy and black are clean.
  • Live, DBSQL and Reyden on the kernel path: explicit port 443, a URI that overrides _port, and URI-only all pass.

Note

This overlaps databricks-sql-python#915, another engineer's PR that changes the same function. Only one of the two should merge.

Related PRs: databricks/databricks-sql-nodejs#636, databricks/databricks-sql-go#581, https://github.com/databricks/databricks-driver-test/pull/2188

This pull request and its description were written by Isaac.

The kernel backend (use_kernel=True) ignored the Thrift-style _port and
_connection_uri overrides. Resolve both connector-side onto the kernel's single
host and http_path, with the Thrift rules:

- _port (1..65535; 0 or None mean the default) folds into server_hostname as
  host:port. The default 443 adds nothing, IPv6 literals are bracketed, and an
  embedded port equal to _port is accepted. An empty server_hostname with _port
  raises ValueError.
- _connection_uri (http(s)://host[:port]/path?query) replaces host, port, path
  and query, and wins over _port. Scheme-less, non-http(s), path-less and
  credential-bearing URIs raise ValueError. URI-only use works.
- With OAuth (U2M, Azure service principal, JWT key file, or M2M client id and
  secret), a _connection_uri whose origin differs from https://server_hostname
  raises before a kernel session opens. PAT is not affected. A non-default _port
  with OAuth is not guarded: the token exchange goes to the same host:port, which
  is documented.
- The SPOG x-databricks-org-id header follows the resolved path.

Tests cover the resolver tables, the OAuth origin guard, the SPOG re-derivation,
and a real-wheel loopback dial proving the kernel reaches the requested port
over TLS for _port and for _connection_uri.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
The helper returned Optional[str] for the resolved path, but KernelDatabricksClient takes a str, so mypy fails in check-types on every supported Python. Session.__init__ already types http_path as str, so this is an annotation-only change with no runtime effect.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 Low

Looks good: two low concerns. Resolver, OAuth origin guard and SPOG re-derivation match Thrift's precedence and have thorough table tests. Nit: a few new lines (the Conflicting-ports f-string, the _split_host and _uri_host_and_path docstrings) exceed the 100-char limit in CONTRIBUTING.md.

Comment thread src/databricks/sql/session.py Resolved
Comment thread src/databricks/sql/session.py Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 2 Low

Looks good, with 1 low concern. The resolver, the OAuth origin guard and the SPOG re-derivation match the auth bridge's routing and are well covered by tests. Nit: a few new lines go past the 100-char limit (the Conflicting ports f-string and the _split_host/_uri_host_and_path docstrings), and the AC-P4 ticket reference in the docstring.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread src/databricks/sql/session.py Resolved
Comment thread src/databricks/sql/session.py Resolved
A kernel connection without _port or _connection_uri again passes server_hostname through unchanged, so odd hosts no longer raise on the default path. The OAuth origin guard now skips connections the kernel auth bridge routes to PAT: an access_token with no JWT or client-secret M2M credentials, even when auth_type is databricks-oauth. azure-sp-m2m and M2M/JWT stay guarded.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, with 1 low concern. The resolver, the OAuth predicate (which matches the auth bridge's routing) and the SPOG re-derivation all check out. Nits: session.py:135 and the _split_host/_uri_host_and_path docstrings go past the 100-char limit, and test_session.py imports databricks.sql twice.

Comment thread src/databricks/sql/session.py Resolved
Wraps the origin-guard condition to the repo's black line length (check-linting). The guard no longer treats a custom credentials_provider as OAuth: the kernel auth bridge rejects that provider with NotSupportedError, so no OAuth origin is involved.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
…ernel port rules

Check the host for a path, query or fragment before the port is parsed, so 'h.io/path:8443' with _port cannot read as port 8443. Only explicit _port is affected; a host without _port passes through unchanged, as on main. Add Thrift tests pinning that host and _port reach ThriftDatabricksClient unchanged. Update CONNECTION_PARAMETERS.md: _port and _connection_uri are now kernel options, with the embedded-port and falsy rules and the OAuth origin rule. Co-authored-by: Isaac <no-reply@databricks.com>

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

cathleeny and others added 2 commits November 20, 2026 00:44
Replace the _HOSTPORT_RE regex, the _port regex and the manual bracketing in the kernel host folding with urllib.parse.urlsplit, int() and an ipaddress pre-check for bare IPv6 literals. The original host string still passes through unchanged unless a port is folded in; folded IPv6 hosts are bracketed.

_check_oauth_origin treats an access_token that decodes as a JWT whose issuer is not server_hostname as OAuth. The kernel wraps every provider with the connection host, so such a token would be exchanged at the _connection_uri origin, while Thrift exchanges it at server_hostname. A mismatched origin now raises before any request. PATs and same-host issuers are unaffected.

The port-conflict warning is skipped when _port is the default 443, matching Node. Non-default ports still warn on a mismatch.

Behavior-table tests in test_session.py pass unchanged. One warning test changed: test_embedded_port_wins_over_differing_explicit_port lost its ('h.io:8443', 443) row, which asserted the 443 warning. That case moved to test_embedded_port_with_default_443_does_not_warn under the new rule.

Known differences from bc7c09e: the OAuth origin check now compares the parsed hostname, so a server_hostname such as 'h.io/x' or 'user@h' with a matching _connection_uri origin passes where it was previously refused. The error text for 'h:70000', 'h:abc' and '[::1' is now one generic malformed-or-out-of-range message. 'user:pass@h' with _port is still refused.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
_is_bare_ipv6 now returns early unless the text has at least two colons and no leading bracket. ip_address raised a ValueError for every host name and host:port string, which made the folding path about 2x slower than bc7c09e. A bare IPv6 literal always has two or more colons, so behavior is unchanged; the session tests and the realkernel loopback tests pass.

Microbenchmark, median us per call (20000 calls x 7 repeats):
- bare host, _port 8443: 1.305 base, 1.261 new
- host:port, _port 8443: 1.364 base, 1.393 new (+2%)
- IPv6 bracketed, _port 8443: 1.340 base, 1.355 new (+1%)
- IPv6 bare, _port 8443: 1.413 base, 2.642 new (+87%, the successful ip_address parse)

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Remove _is_bare_ipv6 and the ipaddress import. Bare IPv6 hosts such as ::1 never worked on the kernel path: the kernel prepends https:// and its URL parser rejects an unbracketed IPv6 literal. IPv6 literals must be bracketed ([::1]).

With _port set, urlsplit reads everything after the first colon of a bare IPv6 literal as the port. That text always contains another colon, so the integer parse fails and the existing "malformed IPv6 literal or port" ValueError is raised before any request. This includes _port=443. Without _port the host still passes through unchanged, as on main.

Tests: the ("::1", 8443) -> "[::1]:8443" row is removed. ::1 with _port 8443 or 443, and 2001:db8::1 with _port 8443, are now in the reject table.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

🔀 1 finding considered and dismissed by the judge
  • 🟡 Medium src/databricks/sql/session.py:240 — An access_token combined with identity_federation_client_id bypasses the origin guard when its issuer matches server_hostname, although the bridge forwards that
    Judge's evidence: if routed_to_pat or not oauth:

mypy (check-types) rejected passing the Optional access_token to
_is_foreign_jwt, because bool(access_token) does not narrow its type. Use
`access_token and _is_foreign_jwt(...)` inside bool(), which narrows it.
The behavior is the same; the session tests pass.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Low

Looks good, no blocking issues. The host/port/URI resolution, the OAuth-origin guard (which follows the auth_bridge.py routing order) and the SPOG re-derivation all check out. Nit: no test covers a caller-set x-databricks-org-id combined with _connection_uri, and test_resolved_kernel_host_and_path_reach_kernel_client never closes its connection.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Comment thread src/databricks/sql/session.py Resolved
_redact kept everything before the first "://" as the scheme. In a
scheme-less _connection_uri such as
"h.io/sql/x?token=secret&redirect=https://example.com", that prefix is the
path and query, so the error message echoed token=secret. The prefix is now
kept only when it is http or https; otherwise it is shown as <redacted>.
Two regression rows in test_invalid_connection_uri_message_omits_credentials
fail without this change.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 Medium · 1 Nit

Looks good, no blocking issues. I checked the host/port/URI resolution, redaction, the SPOG header re-derivation, and the OAuth origin guard's PAT-routing logic against auth_bridge.kernel_auth_kwargs (azure-sp-m2m, then JWT, then M2M, then PAT, then U2M), and they are consistent. Thrift passthrough is unchanged and covered by tests.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — An access_token paired with identity_federation_client_id bypasses the origin guard as PAT, although the bridge forwards that selector for mandatory token exchange, allowing _connection_uri to redirect that exchange to another origin. Treat an explicit federation client ID as exchange-bearing auth, exclude it from the PAT exemption, and add a mismatched-origin regression test. (Found by GPT-6.1 Sol · the judge couldn't confirm it)
  • ⚪ Nit — The test opens a kernel connection and never closes it, unlike test_spog_org_id_follows_resolved_path. Append .close() to the connect() call. (Found by Claude Opus 5.5 · confirmed by the judge)

…is set

With identity_federation_client_id, the kernel bridge forwards the
access_token for mandatory workload-identity token exchange at the
connection host. _check_oauth_origin routed any access_token that wasn't a
foreign-issuer JWT to the PAT exemption, so a mismatched _connection_uri
origin could receive that exchange, where Thrift exchanges at
server_hostname. Such a token now counts as OAuth for the origin check, so a
mismatched origin raises before any request. A plain PAT without a
federation client id is unaffected.

Also close the connection in
test_resolved_kernel_host_and_path_reach_kernel_client, and describe the
exchanged-token rule in CONNECTION_PARAMETERS.md.

Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
@cathleeny

Copy link
Copy Markdown
Collaborator Author

Both findings from the latest review are addressed in 973b4e2.

  • Medium (identity_federation_client_id bypassing the origin guard): fixed. With identity_federation_client_id, the kernel bridge forwards the access_token for mandatory workload-identity exchange at the connection host. _check_oauth_origin now treats such a token as OAuth, the same as a foreign-issuer JWT, so it is no longer exempt as a PAT. A mismatched _connection_uri origin now raises before any request. test_federation_client_id_access_token_origin_guard covers a mismatched origin (raises) and a matching one (passes). A PAT without a federation client id is unaffected. CONNECTION_PARAMETERS.md and the PR description describe the rule.
  • Nit (unclosed connection): test_resolved_kernel_host_and_path_reach_kernel_client now closes its connection, like test_spog_org_id_follows_resolved_path.

The session tests (200), kernel client tests (282), mypy on session.py and black all pass.

Automated reply from the driver feature factory (Isaac). Not reviewed by a human.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

🔀 1 finding considered and dismissed by the judge
  • 🔵 Low src/databricks/sql/session.py:259 — _check_oauth_origin reuses _split_host, which raises when _port is used with a malformed host. So OAuth plus a valid _connection_uri fails with "malform
    Judge's evidence: f"Invalid server_hostname {_redact(host)!r}: malformed IPv6 literal or port"

This branch was previously deployed

1 inactive deployment
azure-prod — 973b4e28 Deployed Nov 20, 2026 by cathleeny via test-with-coverage #1345
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants