Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
31de2dd
test-companion: fixes from running every job as the GitHub App
naimish-browse Nov 12, 2026
566c282
test-companion: leave tool by-products out of a handed-back PR
naimish-browse Nov 12, 2026
46723f3
test-companion: keep left-out files off the branch and read file name…
naimish-browse Nov 12, 2026
39ec8a6
test-companion: say the CLI's real reason, and hide credentials in co…
naimish-browse Nov 12, 2026
e504bea
test-companion: always squash a handed-back branch, and never reopen …
naimish-browse Nov 12, 2026
0d6c8d9
test-companion: say why a hand-off push was refused, and never guess …
naimish-browse Nov 12, 2026
0f6271c
test-companion: one reply per request, newest plan wins, and say whic…
naimish-browse Nov 12, 2026
5eabd3c
test-companion: older plans stay spent, ask by run id, and fail close…
naimish-browse Nov 12, 2026
2882be5
test-companion: one approval settles a job's plans, and everyday word…
naimish-browse Nov 12, 2026
8d05931
test-companion: retry the App sign-in, and refuse a fix that rewrote …
naimish-browse Nov 12, 2026
72d2a20
test-companion: react 👀 to a request once it's picked up
naimish-browse Nov 12, 2026
4427a56
test-companion: Asked by shows who asked and the comment they asked in
naimish-browse Nov 12, 2026
acc1c57
test-companion: time-limit App calls, and never guess the reviewed co…
naimish-browse Nov 12, 2026
aa59121
test-companion: a plain go on a generate plan saves exactly the plann…
naimish-browse Nov 13, 2026
181fa6d
test-companion: take the save destination from the approved plan's ow…
naimish-browse Nov 13, 2026
ac2478a
test-companion: commit exactly the files the Action chose, and ignore…
naimish-browse Nov 16, 2026
425eafe
test-companion: approving a fix plan reads the failed run that plan l…
naimish-browse Nov 16, 2026
e76f3f9
test-companion: approving a generate plan saves exactly the cases the…
naimish-browse Nov 16, 2026
40ef7c1
test-companion: never read a failed run that was started from a fork
naimish-browse Nov 16, 2026
d3e9c79
test-companion: say when an older CLI is why a plan's cases can't be …
naimish-browse Nov 16, 2026
3fc6bce
test-companion: bring the README up to date
naimish-browse Nov 16, 2026
2dbf7a4
test-companion: safer exact saves (interrupted saves, lookups, reply …
naimish-browse Nov 16, 2026
045938c
test-companion: never answer our own comments when github-token is a …
naimish-browse Nov 16, 2026
3c1fea8
test-companion: read every run on a commit, not just the first 100
naimish-browse Nov 16, 2026
55b36cb
test-companion: cap the whole App sign-in at about 30 seconds
naimish-browse Nov 16, 2026
e718ed0
test-companion: don't hand back a pull request built on GitHub's merg…
naimish-browse Nov 16, 2026
fa37db7
test-companion: a generate plan shows every field approving saves
naimish-browse Nov 16, 2026
52da1a1
test-companion: a staged rename is committed as a move, not a copy
naimish-browse Nov 16, 2026
9d0f22b
test-companion: say a save saved nothing when that's certain, and onl…
naimish-browse Nov 16, 2026
095fccb
test-companion: check the pushed tree against the base as listed befo…
naimish-browse Nov 16, 2026
f2cb285
test-companion: use a neutral placeholder reference in a test
naimish-browse Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions test-companion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@
Ask BrowserStack Test Companion for test work from a pull request.

Comment `@browserstack test this PR` and it reads the change, proposes the
tests it would write, and writes them once someone replies `go`. The result
comes back as one comment and one check.
tests it would write, and writes them once someone replies `go`. Each request
(an `@browserstack` comment, or a `go`) gets its own reply with that run's
report, plus a "Test Companion" check. Automatic runs, which answer no
request, keep one status comment per pull request up to date.

## Quick start

Expand Down Expand Up @@ -34,6 +36,10 @@ are easy to miss:
- concurrency rules that keep one run per pull request and let closing a pull
request cancel its run.

**Already using an older copy?** Copy the starter file again. Older copies let
comments written by bots, Test Companion's own included, start a run, which
costs a runner each time and can push a waiting `go` out of the queue.

## Pin the CLI version

Without a pin the CLI upgrades itself on a release day, so two runs a day apart
Expand All @@ -45,7 +51,7 @@ from the release you tested; a version without a checksum is refused.
with:
username: ${{ secrets.BROWSERSTACK_USERNAME }}
access-key: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
cli-version: "1.2.0"
cli-version: "1.1.16"
cli-sha256: "<sha256 of that release for linux-amd64>"
```

Expand All @@ -62,11 +68,50 @@ from the release you tested; a version without a checksum is refused.
| `preview` | no | Report what would happen and change nothing |
| `allow-unisolated` | no | `"true"` runs the agent as the job's own user when it cannot be locked down, where it can reach this job's tokens and secrets. Off by default |
| `app-id` / `app-private-key` | no | Post as the GitHub App. Keep the workflow's `permissions:` block: the workflow token is the fallback |
| `github-token` | no | The token for the Action's own GitHub calls. Leave it as the default (the workflow token) or use the GitHub App's token, never a personal access token: Test Companion recognises its own comments by their bot account, and a person's token would make them look like that person's |

## Approving a plan

Before a job changes anything, Test Companion posts its plan and waits for
someone with write access to reply `go`. Read the plan before you reply.
Test Companion reacts 👀 to a request, or a `go`, once it picks it up.

For `generate`, a plain `go` saves exactly the test cases in the plan to Test
Management, in the project and folder the plan shows under "Will be saved to":
the plan's run keeps them (the `test-companion-cases` artifact, for 7 days) and
nothing is generated again. This needs **Test Companion CLI 1.1.16 or later**,
for both the plan and the `go`; with an older CLI nothing is saved and the reply
says to update it. A `@browserstack go` with changes generates the cases again
with your changes as guidance, and says so. If the plan's cases can't be found
or have changed, or the plan recorded no Test Management destination (set
`tm_project` and `tm_folder` for generate in the pipeline file), nothing is
saved and you are asked to generate a fresh plan.

For `fix` and `automate`, the work is handed back as a pull request:

- opened as a draft (a normal pull request where drafts aren't available);
- based on the pull request it was asked on, so it carries only the fix (if the
run started on the default branch instead of that pull request's commit, it's
based on the default branch, and the reply says so; if it started on GitHub's
merge of the pull request, nothing is opened, because it would carry the
whole pull request);
- one squashed commit by Test Companion's bot account, so the agent's own
commits never reach GitHub;
- tool by-products (logs, `.env`, `browserstack.err` and the like) left out;
- refused, with nothing pushed, if the agent's branch rewrote history, for
example by resetting past the commit that was reviewed.

### The GitHub App's permissions

Installing the App asks for these repository permissions, and each run's token
is limited to the one repository it runs in:

| Permission | Why |
|---|---|
| Contents: read and write | Commits approved tests to the pull request's branch, and pushes the branch for a hand-back pull request |
| Pull requests: read and write | Posts its replies, and opens the hand-back pull request |
| Checks: read and write | Posts the "Test Companion" check |
| Actions: read and write | Read: the failed run's logs for `explain`, `fix` and `find the bad PR`, and the plan's saved cases when a `generate` plan is approved. Write: cancelling its own runs when a pull request is closed without merging |

Without the GitHub App, Test Companion posts as `github-actions[bot]`, the
account every workflow in your repository shares, and recognises its own plans
Expand Down
14 changes: 14 additions & 0 deletions test-companion/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,8 @@ runs:
TC_ACTION_APP_ID: ${{ inputs.app-id }}
TC_ACTION_APP_PRIVATE_KEY: ${{ inputs.app-private-key }}
TC_ACTION_ALLOW_UNISOLATED: ${{ inputs.allow-unisolated }}
# The `tc --version` line, so a plan records which CLI made it.
TC_ACTION_CLI_VERSION: ${{ steps.install.outputs.cli-version }}
run: node "${{ github.action_path }}/src/main.mjs"

- name: Attach the report
Expand All @@ -234,6 +236,18 @@ runs:
path: ${{ steps.run.outputs.result-file }}
if-no-files-found: ignore

# A generate plan's cases as data. Approving the plan saves exactly these, so nothing
# is generated again. Only a plan sets cases-dir.
- name: Keep the planned test cases for approval
if: always() && steps.run.outputs.cases-dir != ''
# v4.6.2, pinned to a commit: a tag can be repointed.
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: test-companion-cases
path: ${{ steps.run.outputs.cases-dir }}
if-no-files-found: ignore
retention-days: 7

# Matters on a self-hosted runner, which outlives the job.
- name: Take gh's credentials back off the runner
if: always()
Expand Down
28 changes: 28 additions & 0 deletions test-companion/src/access.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,34 @@ export function mayAsk({ hasWriteAccess, askTeam, inTeam } = {}) {
return inTeam === true
}

/**
* The comment to mark with 👀 once a request is picked up, or undefined. Only a comment a
* person with access wrote to us: never a bot's, never a refused one, and a bare `go` only
* when there is a plan for it to approve.
*/
export function pickupReactionTarget({ eventName, comment, plan, allowed, approvalHasPlan } = {}) {
if (eventName !== "issue_comment" || !plan || plan.skip || plan.event !== "pr_comment") return undefined
if (allowed !== true) return undefined
const author = comment?.user
if (!author || author.type === "Bot" || /\[bot\]$/i.test(author.login ?? "")) return undefined
if (plan.approvalOnly && approvalHasPlan !== true) return undefined
const id = Number(comment?.id)
return Number.isSafeInteger(id) && id > 0 ? id : undefined
}

/**
* Add 👀 to a comment. Best-effort: `post(path, body)` is the API call; any failure,
* thrown or returned, is reported back and never stops the run. Returns true when added.
*/
export function addPickupReaction({ repo, commentId, post }) {
if (!repo || !commentId || typeof post !== "function") return false
try {
return post(`repos/${repo}/issues/comments/${commentId}/reactions`, { content: "eyes" }) === true
} catch {
return false
}
}

/** Whether this run can write to the configured tests repository. `canPush` undefined counts as no. */
export function describeTestRepoAccess({ testsRepo, thisRepo, canPush } = {}) {
const target = String(testsRepo ?? "").trim()
Expand Down
76 changes: 70 additions & 6 deletions test-companion/src/answer.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,41 @@ import { StringDecoder } from "node:string_decoder"
// Cap on an unterminated line; the agent's commands can print megabytes without a newline.
export const MAX_PARTIAL_LINE = 1 << 20

export function createAnswerCapture() {
/**
* `plainErrors`: also read `Error: …` lines. Only for the CLI's stderr; on stdout the
* agent's test output prints its own `Error:` lines (assertions, stack traces).
*/
export function createAnswerCapture({ plainErrors = true } = {}) {
// Chunks can split a multi-byte character.
const decoder = new StringDecoder("utf8")
let pending = ""
// Discarding the rest of a line that overran MAX_PARTIAL_LINE.
let skipping = false
let answer
let plainError
let structuredError

const read = (line) => {
const t = line.trim()
// The CLI prints why it stopped as `Error: …` on stderr.
const plain = plainErrors && /^Error:\s*(.+)$/.exec(t)
if (plain) {
plainError = plain[1]
return
}
if (!t.startsWith("{") || !t.endsWith("}")) return
let event
try {
event = JSON.parse(t)
} catch {
return
}
if (!event || typeof event !== "object" || event.event !== "answer") return
if (!event || typeof event !== "object") return
if (event.event === "error" && typeof event.message === "string" && event.message.trim()) {
structuredError = event.message.trim()
return
}
if (event.event !== "answer") return
const text = typeof event.text === "string" ? event.text.trim() : ""
if (text) answer = text
}
Expand Down Expand Up @@ -51,25 +68,72 @@ export function createAnswerCapture() {
pending = ""
},
answer: () => answer,
structuredError: () => structuredError,
plainError: () => plainError,
error: () => structuredError ?? plainError,
pending: () => pending.length,
}
}

/** Run a command, echoing stdout live while scanning it for the answer. Never rejects. */
export function runCapturingAnswer(command, args, childEnv) {
return new Promise((resolve) => {
const capture = createAnswerCapture()
const child = spawn(command, args, { stdio: ["inherit", "pipe", "inherit"], env: childEnv })
const capture = createAnswerCapture({ plainErrors: false })
const errors = createAnswerCapture()
const child = spawn(command, args, { stdio: ["inherit", "pipe", "pipe"], env: childEnv })
child.stdout.on("data", (chunk) => {
process.stdout.write(chunk)
capture.push(chunk)
})
child.stderr.on("data", (chunk) => {
process.stderr.write(chunk)
errors.push(chunk)
})
child.stdout.on("error", () => {})
child.on("error", () => resolve({ status: null, answer: undefined }))
child.stderr.on("error", () => {})
child.on("error", () => resolve({ status: null, answer: undefined, error: undefined }))
// `close`, not `exit`: stdout may still be draining at exit.
child.on("close", (status) => {
capture.end()
resolve({ status, answer: capture.answer() })
errors.end()
// The CLI's own structured error is the precise reason; a plain line is the fallback.
// The CLI doesn't pass its child processes' stderr through, so the last plain
// `Error:` line on stderr is cobra's, i.e. the CLI's own.
const error = capture.structuredError() ?? errors.structuredError() ?? errors.plainError()
resolve({ status, answer: capture.answer(), error })
})
})
}

/**
* Hide credentials before text reaches a pull request comment. GitHub masks secrets in
* job logs, not in comments posted through the API.
*/
export function redactSecrets(text, secrets = []) {
if (text === undefined || text === null) return text
let out = String(text)
// Longest first, so a secret containing another is replaced whole.
const known = [...new Set(secrets.map((s) => String(s ?? "").trim()).filter((s) => s.length >= 6))].sort(
(a, b) => b.length - a.length,
)
for (const s of known) out = out.split(s).join("***")
// user:password@ in any URL.
return out.replace(/(\/\/)[^/\s:@]+:[^/\s@]+@/g, "$1***@")
}

/** redactSecrets over every string in a JSON payload about to be sent to GitHub. */
export function redactDeep(value, secrets = []) {
if (typeof value === "string") return redactSecrets(value, secrets)
if (Array.isArray(value)) return value.map((v) => redactDeep(v, secrets))
if (value && typeof value === "object") {
return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, redactDeep(v, secrets)]))
}
return value
}

/** Values of environment variables whose names mark them as credentials. */
export function secretsFrom(env = {}) {
return Object.entries(env)
.filter(([k]) => /(TOKEN|ACCESS_KEY|PRIVATE_KEY|SECRET|PASSWORD)/i.test(k))
.map(([, v]) => v)
}
71 changes: 69 additions & 2 deletions test-companion/src/app-auth.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,73 @@ export function jwtClaims(appId, nowSeconds) {
* and GitHub accepts an App's JWT only as `Authorization: Bearer`.
* Returns { ok, status, data, reason }; `reason` is GitHub's message, never the JWT.
*/
export async function appApi(path, { jwt, method = "GET", body, apiUrl, fetchImpl = fetch } = {}) {
export async function appApi(
path,
{
jwt,
method = "GET",
body,
apiUrl,
fetchImpl = fetch,
attempts = 3,
sleep = defaultSleep,
timeoutMs = APP_CALL_TIMEOUT_MS,
// When set, no try starts, and no wait runs, past this time (ms since the epoch).
deadline,
now = Date.now,
} = {},
) {
const left = () => (deadline === undefined ? Infinity : deadline - now())
let result
for (let attempt = 1; attempt <= attempts; attempt++) {
if (left() <= 0) return { ...(result ?? OUT_OF_TIME), attempts: attempt - 1 }
result = await appApiOnce(path, { jwt, method, body, apiUrl, fetchImpl, timeoutMs: Math.min(timeoutMs, left()) })
if (!isRetryable(result) || attempt === attempts) return { ...result, attempts: attempt }
const wait = retryDelayMs(result, attempt)
// A wait that would end past the deadline can't lead to another try.
if (wait >= left()) return { ...result, attempts: attempt }
await sleep(wait)
}
return { ...result, attempts }
}

const OUT_OF_TIME = { ok: false, status: 0, data: undefined, reason: "ran out of time to reach GitHub" }

/** The most the whole App sign-in may take, every call, try and wait included, before falling back. */
export const APP_SIGN_IN_BUDGET_MS = 30_000

const defaultSleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))

/** One App call gives up after this long; a timeout counts as a network error, so it is retried. */
export const APP_CALL_TIMEOUT_MS = 10_000

/** A network failure, a 5xx or a 429 may pass on a second try; 401/403/404/422 never will. */
export function isRetryable(result) {
const status = Number(result?.status)
return status === 0 || status === 429 || (status >= 500 && status <= 599)
}

/** 1s, then 3s; a Retry-After from GitHub wins, capped at 10s. */
export function retryDelayMs(result, attempt) {
const after = Number(result?.retryAfter)
if (Number.isFinite(after) && after > 0) return Math.min(after, 10) * 1000
return attempt === 1 ? 1000 : 3000
}

async function appApiOnce(path, { jwt, method, body, apiUrl, fetchImpl, timeoutMs }) {
const base = String(apiUrl || "https://api.github.com").replace(/\/+$/, "")
// Without a limit a stalled connection waits ~5 minutes, and each retry again. Covers the
// body too; aborting rejects fetch, which reads below as a network error and is retried.
const controller = new AbortController()
const timer = setTimeout(() => controller.abort(new Error(`no reply within ${timeoutMs / 1000}s`)), timeoutMs)
try {
return await appApiCall(base, path, { jwt, method, body, fetchImpl, signal: controller.signal })
} finally {
clearTimeout(timer)
}
}

async function appApiCall(base, path, { jwt, method, body, fetchImpl, signal }) {
let response
try {
response = await fetchImpl(`${base}/${path}`, {
Expand All @@ -41,13 +106,15 @@ export async function appApi(path, { jwt, method = "GET", body, apiUrl, fetchImp
...(body === undefined ? {} : { "Content-Type": "application/json" }),
},
body: body === undefined ? undefined : JSON.stringify(body),
signal,
})
} catch (e) {
return { ok: false, status: 0, data: undefined, reason: `could not reach GitHub: ${e?.message ?? e}` }
}
const data = await response.json().catch(() => undefined)
const reason = response.ok ? "" : `${data?.message || response.statusText || "no detail from GitHub"} (HTTP ${response.status})`
return { ok: response.ok, status: response.status, data, reason }
const retryAfter = response.headers?.get?.("retry-after") ?? undefined
return { ok: response.ok, status: response.status, data, reason, ...(retryAfter ? { retryAfter } : {}) }
}

/** What to log when looking up the App's installation on this repository fails. */
Expand Down
Loading
Loading