Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 77 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,10 @@ jobs:
- name: Run unit tests
env:
VIRTUAL_ENV: ${{ github.workspace }}/.venv
# The ML demos run in their own job, below.
run: uv run --no-project pytest -v . -m "not integration" --ignore=tests/test_ml_demos.py
# The ML demos run in their own job, below. --benchmark-disable runs
# each benchmark once, as a test, so a PR that breaks one fails here
# rather than in the benchmarks job on main.
run: uv run --no-project pytest -v . -m "not integration" --benchmark-disable --ignore=tests/test_ml_demos.py

ml-demos:
# Each ML demo in benchmarks/ml, at its smallest size, checked against
Expand Down Expand Up @@ -121,3 +123,76 @@ jobs:
env:
VIRTUAL_ENV: ${{ github.workspace }}/.venv
run: uv run --no-project pytest -v tests/test_ml_demos.py

benchmarks:
# tests/test_benchmarks.py, timed with pytest-benchmark (a pytest
# plugin that runs a function repeatedly and reports its timings) and
# stored on gh-pages by github-action-benchmark (an action that keeps
# each run's results as a history, charts it, and comments on a commit
# that looks like a regression).
name: "benchmarks"
runs-on: ubuntu-latest
# Only on main: a PR from a fork has no permission to push to
# gh-pages, and a dispatch from another branch shouldn't record a
# point into history that doesn't belong to main's timeline.
if: github.ref == 'refs/heads/main' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
permissions:
contents: write
deployments: write
steps:
- uses: actions/checkout@v4

- uses: dtolnay/rust-toolchain@stable
id: rust-toolchain

- name: Setup sccache
uses: mozilla-actions/sccache-action@v0.0.9

- name: Configure sccache
run: |
echo "SCCACHE_GHA_ENABLED=true" >> $GITHUB_ENV
echo "RUSTC_WRAPPER=sccache" >> $GITHUB_ENV

- name: Cache Cargo
uses: actions/cache@v4
with:
path: ~/.cargo
key: cargo-cache-${{ steps.rust-toolchain.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
Comment thread
d33bs marked this conversation as resolved.
Resolved

- uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"
enable-cache: true

- name: Install xarray_sql
run: uv sync --locked --dev --extra benchmark --no-install-package xarray-sql
- name: build rust
run: uv run --no-project maturin develop --uv
- name: Run benchmarks
env:
VIRTUAL_ENV: ${{ github.workspace }}/.venv
run: uv run --no-project pytest tests/test_benchmarks.py -m benchmark --benchmark-only --benchmark-json=benchmark.json
- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
with:
name: "Python benchmarks (pytest-benchmark)"
tool: "pytest"
output-file-path: benchmark.json
github-token: ${{ secrets.GITHUB_TOKEN }}
# The action retries a push that gh-pages rejects (up to 10 times,
# re-cloning gh-pages each time), so it can share the branch with
# mike's pushes from docs.yml. A new point goes live when docs.yml
# next publishes gh-pages to Pages, which may be one push to main
# later if that publish checked out gh-pages before this push.
auto-push: true
# mike (docs.yml) owns dev/, 0.x/, latest/, index.html and
# versions.json on gh-pages, and replaces dev/ wholesale on
# every docs deploy; the default dev/bench would collide
# with that and get wiped out. Use a root-level path mike
# never touches.
benchmark-data-dir-path: benchmarks
# Hosted runners are noisy; keep this loose until there's
# enough history on gh-pages to pick a tighter threshold.
alert-threshold: "200%"
comment-on-alert: true
fail-on-alert: false
20 changes: 20 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,26 @@ MATURIN_PEP517_ARGS="--no-default-features --profile wasm" pyodide build -o dist
On NixOS, run these commands in `nix run .#wasm-shell`. This shell is an
FHS environment, which lets the prebuilt Emscripten and rustup binaries run.

### Benchmarks

`tests/test_benchmarks.py` holds benchmarks written with
[pytest-benchmark](https://github.com/ionelmc/pytest-benchmark), a pytest
plugin that runs a function repeatedly and reports its timings. Each one
also checks its result, so it's a test too. To time them locally:

```shell
uv run pytest tests/test_benchmarks.py --benchmark-only
```

CI's unit run runs each benchmark once, untimed, with `--benchmark-disable`.
On every push to main, the `benchmarks` job times them and stores the result
on the `gh-pages` branch with
[github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark),
which keeps the history, charts it at
<https://xql.systems/xarray-sql/benchmarks/>, and comments on a commit that
looks like a regression. A new point appears on the chart when the docs are
next published.


## Before submitting a pull request...

Expand Down
7 changes: 7 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ geo = [
ml = [
"ddxdb>=0.4.6,<0.5; sys_platform != 'emscripten'",
]
# Benchmarks timed with pytest-benchmark and tracked over time in CI
# (see tests/test_benchmarks.py and CONTRIBUTING.md).
benchmark = [
"pytest-benchmark>=4.0",
]
test = [
"adbc-driver-flightsql>=1.12",
"adbc-driver-sqlite>=1.12",
Expand Down Expand Up @@ -133,6 +138,7 @@ ignore_missing_imports = true
dev = [
"xarray_sql[test]",
"xarray_sql[docs]",
"xarray_sql[benchmark]",
"py-spy>=0.4.0",
"ruff>=0.15.10",
"maturin>=1.9.1",
Expand All @@ -156,4 +162,5 @@ cache-keys = [{file = "pyproject.toml"}, {file = "rust/Cargo.toml"}, {file = "**
testpaths = ["tests"]
markers = [
"integration: needs network (anonymous GCS); excluded from the CI unit run",
"benchmark: timed by pytest-benchmark in its own CI job; the CI unit run runs each once, untimed",
]
59 changes: 59 additions & 0 deletions tests/test_benchmarks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
"""A benchmark tracked over time in CI (see CONTRIBUTING.md).

It uses the ``benchmark`` fixture of pytest-benchmark
(https://github.com/ionelmc/pytest-benchmark), a pytest plugin that runs
a function repeatedly and reports its timings. On every push to main, CI
stores the timings with github-action-benchmark
(https://github.com/benchmark-action/github-action-benchmark), which
keeps them as a history and flags a commit that looks like a regression.

Starts small: one benchmark, over an in-memory Dataset that needs no
network, so a regression in xarray-sql, DataFusion or the in-memory
table adapter shows up in that history.
"""

import numpy as np
import pytest
import xarray as xr

from xarray_sql import Context

pytestmark = pytest.mark.benchmark


@pytest.fixture(scope="module")
def grid() -> xr.Dataset:
"""A synthetic (time, lat, lon) grid: 100 x 50 x 50 cells, no I/O."""
rng = np.random.default_rng(0)
time, lat, lon = 100, 50, 50
return xr.Dataset(
{
"temperature": (
("time", "lat", "lon"),
rng.normal(size=(time, lat, lon)),
),
},
coords={
"time": np.arange(time),
"lat": np.arange(lat),
"lon": np.arange(lon),
},
)


def test_groupby_mean_over_time(benchmark, grid):
"""A climatology query: mean over time, grouped by lat and lon."""
ctx = Context()
ctx.from_dataset("grid", grid)

def run():
return ctx.sql(
"SELECT lat, lon, AVG(temperature) AS mean_temp "
'FROM "grid" GROUP BY lat, lon ORDER BY lat, lon'
).to_dataset()

result = benchmark(run)
assert result.sizes["lat"] == grid.sizes["lat"]
assert result.sizes["lon"] == grid.sizes["lon"]
expected = grid["temperature"].mean("time")
np.testing.assert_allclose(result["mean_temp"].values, expected.values)
31 changes: 29 additions & 2 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading