Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ regex = "1"
# Edits a TOML config file and keeps its comments and layout.
toml_edit = "0.25.15"
sqlparser = { version = "0.63", default-features = false, features = ["std"] }
# The core's SQL formatter, checked statement by statement against the script lexer.
sqlformat = "0.5"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "time", "sync", "process"] }
tokio-util = { version = "0.7", default-features = false, features = ["codec", "rt"] }
secrecy = { version = "0.10", features = ["serde"] }
Expand Down
4 changes: 2 additions & 2 deletions core/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,8 @@ Each boundary error has a stable `code()`, such as `driver.timeout` or `ssh.host

`ffi` is the library an app outside Rust loads: the Windows app as a DLL, the Apple apps as a static library in an XCFramework, built without the `engines` feature by `apple/scripts/build-core.sh`. Core ADRs 0002, on the bindings, 0003, on result windows, and 0009, on the macOS slices, hold the reasons.

- One UniFFI API. `Core` lists the engines with the family each is listed under, opens a file, connects a saved connection or tests one from a form, and parses a connection URL with `tablepro_core::connection_url`, refusing one that needs a setting its record has no field for. Its `ConnectionLibrary` reads and changes the saved connections, groups, favourites and tags in `connections.json` and stores their secrets, and gives the Apple apps the library's CloudKit records and merges a pull into it (`tablepro_storage::cloudkit`). A `DatabaseSession` reads the catalog one level at a time, writes a table's sorted page at an offset and counts the table, writes a query re-run with a sort (`docs/decisions/0007-sorting-a-result.md`), picks the statement at a caret with the session's grammar (`statement_at`), says what Safe Mode lets an editor run do at the session's level and takes a level saved while it is open (`gate_run`, over `SafeModeLevel::gate`, and `set_safe_mode`), reports and waits on the session's health, opens a `QueryEditor`, and opens a `TableEditor`, which plans a table tab's typed edits into a `RowSave` and runs it (`docs/decisions/0019-row-edits-through-the-ffi.md`). `QueryEditor::run` returns a `QueryRun` at once; the app drains its events after a `RunListener` call, which comes only when no earlier one is waiting to be drained, and the last says which transaction the editor is left in. Each result set is a `ResultSet`, whose `sorted` gives the rows it holds in a column's order.
- `Core::open_account` opens the TablePro account as an `AccountService`, whose `AccountListener` only signals that `status()` changed. It also shares links: `create_share` builds a result's snapshot from a `ResultSet` in Rust, and a table page's `view_sql` is the query its view stands for ([ADR 0027](docs/decisions/0027-share-links.md)). `create_share` takes the share sheet's Keep, Mask or Remove for each column, and `ResultSet::share_masking`, `share_preview` and `share_summary` start the sheet's columns from the detectors and show and count what a share sends ([ADR 0028](docs/decisions/0028-data-masking.md)). `scan_sql_for_secrets`, `scan_columns_for_secrets`, `scan_sql_for_personal_data` and `share_token_from_link` are functions of their own.
- One UniFFI API. `Core` lists the engines with the family each is listed under, opens a file, connects a saved connection or tests one from a form, and parses a connection URL with `tablepro_core::connection_url`, refusing one that needs a setting its record has no field for. Its `ConnectionLibrary` reads and changes the saved connections, groups, favourites and tags in `connections.json` and stores their secrets, and gives the Apple apps the library's CloudKit records and merges a pull into it (`tablepro_storage::cloudkit`). A `DatabaseSession` reads the catalog one level at a time, writes a table's sorted page at an offset and counts the table, writes a query re-run with a sort (`docs/decisions/0007-sorting-a-result.md`), picks the statement at a caret with the session's grammar (`statement_at`), says what completes at a caret for an editor in one of the server's databases as a `CompletionList` that ranks again for each word typed, read from the catalog the other views share (`complete`, over `tablepro_core::sql_syntax::completion` and `tablepro_session::completion`), says what Safe Mode lets an editor run do at the session's level and takes a level saved while it is open (`gate_run`, over `SafeModeLevel::gate`, and `set_safe_mode`), reports and waits on the session's health, opens a `QueryEditor`, and opens a `TableEditor`, which plans a table tab's typed edits into a `RowSave` and runs it (`docs/decisions/0019-row-edits-through-the-ffi.md`). `QueryEditor::run` returns a `QueryRun` at once; the app drains its events after a `RunListener` call, which comes only when no earlier one is waiting to be drained, and the last says which transaction the editor is left in. Each result set is a `ResultSet`, whose `sorted` gives the rows it holds in a column's order.
- `Core::open_account` opens the TablePro account as an `AccountService`, whose `AccountListener` only signals that `status()` changed. It also shares links: `create_share` builds a result's snapshot from a `ResultSet` in Rust, and a table page's `view_sql` is the query its view stands for ([ADR 0027](docs/decisions/0027-share-links.md)). `create_share` takes the share sheet's Keep, Mask or Remove for each column, and `ResultSet::share_masking`, `share_preview` and `share_summary` start the sheet's columns from the detectors and show and count what a share sends ([ADR 0028](docs/decisions/0028-data-masking.md)). `scan_sql_for_secrets`, `scan_columns_for_secrets`, `scan_sql_for_personal_data` and `share_token_from_link` are functions of their own. So are `format_sql` and `format_sql_selection`, which reformat a script or a selection through `tablepro_core::sql_syntax::format` and keep a statement as typed where the formatter would change its tokens.
- `Core::ai_assistant` gives an `AiAssistant` over the core's one AI client and its secret store: it says whether a provider's key is kept and keeps or removes one (the key never crosses back), lists the provider's models, reads a session's catalog into the system prompt of a conversation (`start_conversation`), and streams a reply as an `AiReply`, drained after an `AiReplyListener` call as a run is. Both refuse a saved connection whose AI policy is Never with `ai.blocked`; asking under Ask is the app's ([ADR 0032](docs/decisions/0032-ai-through-the-ffi.md)).
- `Core::mcp_server` gives an `McpService` that runs the MCP server on the sessions `connect_saved` opened, keeps its token in the Core's secret store, and asks the app to open a connection or approve a write through `McpAppHost`; `McpListener` only signals that `status()` or `seen_clients()` changed. The package also builds `tablepro-cli`, the Windows package's command ([0036](docs/decisions/0036-mcp-and-the-command-line-through-the-ffi.md)).
- The grid reads a `ResultSet` through the C ABI in `src/window_abi.rs`: `tablepro_cursor_fetch` packs the display text of one visible range, and `tablepro_cursor_free` ends the cursor. The editor reads SQL there from its own UTF-16 text: `tablepro_syntax_tokens` gives the highlight tokens, and `tablepro_script_statements` every statement as `ScriptPlan` splits the text for `statement_at` and the drivers.
Expand Down
2 changes: 2 additions & 0 deletions core/crates/core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@ secrecy.workspace = true
serde.workspace = true
serde_json.workspace = true
sha2.workspace = true
sqlformat.workspace = true
sqlparser.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["io-util", "rt"] }
tokio-util.workspace = true
unicode-normalization.workspace = true
unicode-segmentation.workspace = true
unicode-width.workspace = true
url.workspace = true
Expand Down
24 changes: 24 additions & 0 deletions core/crates/core/src/sql_syntax/completion/catalog.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
use std::sync::Arc;

use crate::ColumnInfo;
use crate::meta::{ObjectInfo, SchemaInfo, TableRef};

// What a session's catalog holds for one completion; a list it could not read is None or left out.
#[derive(Debug, Default)]
pub struct CatalogAnswer {
pub schemas: Option<Arc<[SchemaInfo]>>,
pub objects: Option<Arc<[ObjectInfo]>>,
pub columns: Vec<(TableRef, Arc<[ColumnInfo]>)>,
}

// Where an unqualified name resolves for an editor that may run in another of the server's databases.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub enum CatalogHome {
// The connection's default schema.
#[default]
Connection,
// Another of the server's databases, which the catalog lists as a schema (MySQL).
Schema(String),
// A database the connection's catalog does not describe, so nothing read from it would be right.
Elsewhere(String),
}
156 changes: 156 additions & 0 deletions core/crates/core/src/sql_syntax/completion/fuzzy.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
// GtkSourceView's fuzzy ranking over characters rather than bytes: its strchr lookup never matches a needle outside
// ASCII, so "đơn" would not find "đơn_hàng".

use std::ops::Range;

// Each character lowercased on its own, so a letter the matcher compares one to one never becomes two, as GLib's
// casefold turns ß into ss.
pub fn casefold(needle: &str) -> String {
needle
.chars()
.map(|c| {
let mut lower = c.to_lowercase();
let first = lower.next().unwrap_or(c);
if lower.next().is_some() { c } else { first }
})
.collect()
}

// How far `haystack` is from `casefold_needle`, lower being closer; None when it does not match.
pub fn fuzzy_match(haystack: &str, casefold_needle: &str) -> Option<u32> {
matched(haystack, casefold_needle).map(|found| found.score)
}

// Everything matches while nothing is typed.
pub fn matches_word(haystack: &str, word: &str) -> bool {
word.is_empty() || fuzzy_match(haystack, &casefold(word)).is_some()
}

// Byte ranges of the matched characters, adjacent ones joined, for a list to mark; empty without a match.
pub fn matched_runs(haystack: &str, casefold_needle: &str) -> Vec<Range<usize>> {
matched(haystack, casefold_needle).map_or_else(Vec::new, |found| found.runs)
}

struct Match {
score: u32,
runs: Vec<Range<usize>>,
}

// A skipped character costs two, a match only by case one, and every character left at the end one, so a prefix
// ranks first.
fn matched(haystack: &str, casefold_needle: &str) -> Option<Match> {
if haystack.is_empty() {
return None;
}
let mut chars = haystack.char_indices();
let mut score: u32 = 0;
let mut runs: Vec<Range<usize>> = Vec::new();
for wanted in casefold_needle.chars() {
let mut skipped: u32 = 0;
let (at, found) = loop {
let (at, found) = chars.next()?;
if same_letter(found, wanted) {
break (at, found);
}
skipped += 1;
};
score = score.saturating_add(skipped.saturating_mul(2));
if found != wanted {
score = score.saturating_add(1);
}
let end = at + found.len_utf8();
match runs.last_mut() {
Some(run) if run.end == at => run.end = end,
_ => runs.push(at..end),
}
}
let rest = u32::try_from(chars.count()).unwrap_or(u32::MAX);
Some(Match {
score: score.saturating_add(rest),
runs,
})
}

fn same_letter(found: char, wanted: char) -> bool {
// The general case builds two iterators a character, which ranking a catalog of 50,000 tables feels.
if found.is_ascii() && wanted.is_ascii() {
return found.eq_ignore_ascii_case(&wanted);
}
found == wanted || found.to_lowercase().eq(wanted.to_lowercase())
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn fuzzy_match_ranks_prefix_higher() {
let needle = casefold("us");

let prefix = fuzzy_match("users", &needle).expect("a match");
let scattered = fuzzy_match("product_status", &needle).expect("a match");

assert!(prefix < scattered, "{prefix} is not closer than {scattered}");
assert_eq!(fuzzy_match("orders", &needle), None);
assert_eq!(fuzzy_match("su", &needle), None, "the order of the letters was ignored");
}

#[test]
fn an_exact_case_match_ranks_above_one_by_case() {
let needle = casefold("id");

assert!(fuzzy_match("id", &needle) < fuzzy_match("ID", &needle));
}

#[test]
fn non_ascii_needle_is_casefolded() {
let needle = casefold("ĐƠN");

assert_eq!(needle, "đơn");
assert!(fuzzy_match("đơn_hàng", &needle).is_some());
assert!(fuzzy_match("ĐƠN_HÀNG", &needle).is_some());
assert_eq!(fuzzy_match("don_hang", &needle), None);
}

#[test]
fn a_proposal_is_offered_only_for_a_word_it_matches() {
assert!(matches_word("orders", ""));
assert!(matches_word("orders", "ORD"));
assert!(!matches_word("orders", "ordu"), "a proposal ranked for the last word");
}

#[test]
fn an_empty_needle_matches_everything() {
assert!(fuzzy_match("users", "").is_some());
assert_eq!(fuzzy_match("", ""), None);
}

#[test]
fn matched_runs_join_adjacent_characters() {
assert_eq!(matched_runs("users", &casefold("US")), [Range { start: 0, end: 2 }]);
assert_eq!(matched_runs("users", ""), Vec::<Range<usize>>::new());
assert_eq!(matched_runs("orders", &casefold("us")), Vec::<Range<usize>>::new());
}

#[test]
fn matched_runs_mark_whole_multibyte_characters() {
let text = "bảng_đơn";
let d = text.find('đ').expect("đ");
let n = text.rfind('n').expect("n");

assert_eq!(matched_runs(text, &casefold("đn")), [d..d + 2, n..n + 1]);
}

#[test]
fn a_dotted_i_keeps_one_source_character() {
assert_eq!(casefold("İ"), "İ");
assert!(matches_word("İstanbul", "İ"));
assert_eq!(matched_runs("İstanbul", &casefold("İst")), [Range { start: 0, end: 4 }]);
}

#[test]
fn a_sharp_s_typed_finds_the_name_it_is_in() {
assert_eq!(casefold("STRAẞE"), "straße");
assert!(matches_word("straße_nr", "straß"));
}
}
14 changes: 14 additions & 0 deletions core/crates/core/src/sql_syntax/completion/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
// What the text before a caret asks for, how closely a name matches the word typed, and the order a list shows them
// in. An app reads the catalog (tablepro_session::completion), builds its own rows and inserts the choice.

mod catalog;
mod fuzzy;
mod ranking;
mod scope;
mod word;

pub use catalog::{CatalogAnswer, CatalogHome};
pub use fuzzy::{casefold, fuzzy_match, matched_runs, matches_word};
pub use ranking::{Ranked, Ranking};
pub use scope::{CompletionScope, analyze, tables_named};
pub use word::{is_word_char, word_start};
102 changes: 102 additions & 0 deletions core/crates/core/src/sql_syntax/completion/ranking.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
use std::sync::Arc;

use super::fuzzy;

// What a list ranks. The proposal itself is each app's, so it only names its label and its order among equal matches.
pub trait Ranked {
fn label(&self) -> &str;

// Lower comes first among equally close matches: what the statement already names before what it might.
fn kind_rank(&self) -> u8;
}

// The closest match first, then by kind, ties in the catalog's order, which lists a table's columns as it defines them.
#[derive(Debug)]
pub struct Ranking<C> {
candidates: Arc<[C]>,
needle: String,
ranked: Vec<u32>,
}

impl<C: Ranked> Ranking<C> {
// `needle` is casefolded. With nothing typed every candidate matches equally: the matcher would rank by length
// alone and put every short keyword above the columns.
pub fn new(candidates: Arc<[C]>, needle: &str) -> Self {
let mut scored: Vec<(u32, u8, u32)> = candidates
.iter()
.enumerate()
.filter_map(|(at, candidate)| {
let score = if needle.is_empty() {
0
} else {
fuzzy::fuzzy_match(candidate.label(), needle)?
};
Some((score, candidate.kind_rank(), u32::try_from(at).ok()?))
})
.collect();
scored.sort_unstable();
Self {
ranked: scored.into_iter().map(|(_, _, at)| at).collect(),
candidates,
needle: needle.to_owned(),
}
}
}

impl<C> Ranking<C> {
pub fn candidates(&self) -> &Arc<[C]> {
&self.candidates
}

pub fn needle(&self) -> &str {
&self.needle
}

// Indexes into the candidates the word matches, closest first.
pub fn ranked(&self) -> &[u32] {
&self.ranked
}
}

#[cfg(test)]
mod tests {
use super::*;

struct Named(&'static str, u8);

impl Ranked for Named {
fn label(&self) -> &str {
self.0
}

fn kind_rank(&self) -> u8 {
self.1
}
}

const COLUMN: u8 = 0;
const TABLE: u8 = 1;
const KEYWORD: u8 = 5;

#[test]
fn closer_matches_rank_first_then_by_kind_then_in_catalog_order() {
let candidates: Arc<[Named]> = vec![
Named("USING", KEYWORD),
Named("product_status", TABLE),
Named("users", TABLE),
Named("user_id", COLUMN),
Named("status", COLUMN),
]
.into();

let order = |needle: &str| Ranking::new(candidates.clone(), &fuzzy::casefold(needle)).ranked;

assert_eq!(order("us"), [2, 3, 0, 4, 1]);
assert_eq!(
order(""),
[3, 4, 1, 2, 0],
"nothing typed ranks by kind, then catalog order"
);
assert_eq!(order("zz"), Vec::<u32>::new());
}
}
Loading
Loading