Repository navigation
feat(ai): send tool results and errors on tool spans - #1283
DanielVisca wants to merge 4 commits into
Conversation
Claude Agent SDK tool spans now wait for the matching tool result and carry the output, error flag, error text and latency. LangChain tool spans set $ai_span_type to tool, mark tools that return an error ToolMessage, and converted ToolMessages keep tool_call_id, name and error status, so tool results can be matched to tool names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
PostHog Review · Standard · reviewed f26dc95: 1 Must fix, 3 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
SDK performance: posthog-pythonBenchmarks completed. No changes flagged for review. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate. View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
…data positional args - Redact structured tool error content before serializing $ai_error. - Close processor.query() when the public query() stream closes, so pending tool spans are sent. - Make SpanMetadata.span_type keyword-only so GenerationMetadata positional args keep their order. - Regenerate the public API snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run every tool error value through finalize_ai_content, not just structured content, so a data URL result is redacted like $ai_output_state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
Re-adding the stamphog label gives the same result unless the changed files change.
stamphog can't auto-review this pull request. The deny-list gate refused it because it touches references/public_api_snapshot.txt, which matches the public_api category. Because of that match, the tier gate placed the whole PR in the never-auto-approve tier (T2-never). The size gate passed, so the refusal isn't about size.
The author's note already asks the AI observability team to review and own it, so the next step is to get that human review. Any change to the public API snapshot needs a person to confirm it was intended.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✗ | matches: public_api in references/public_api_snapshot.txt |
| every_file_attested | ✓ | not a bot PR |
| size | ✓ | 395L, 5F substantive, 746L/11F incl. docs/generated/snapshots — within ceiling |
| tier | ✗ | T2-never because the deny-list matched public_api |
| stamphog 2.9.1 | .stamphog/policy.yml @ unknown · reviewed head 8f4da4b |
…tadata field Keeps the public SpanMetadata/GenerationMetadata API unchanged; the public API snapshot is back to main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No review started. stamphog started a review of this pull request less than 10 minutes ago, and a label can request at most one review per 10 minutes. The |
There was a problem hiding this comment.
Approved.
The telemetry and privacy-sensitive changes have current-head independent assurance in the trusted context. Source inspection supports the described privacy protections, and no concrete unresolved substantive concerns were supplied or identified.
- PostHog Review (Standard) reviewed the current head: no findings.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| every_file_attested | ✓ | not a bot PR |
| size | ✓ | 394L, 5F substantive, 739L/10F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (739L, 10F, two-areas, feat) |
| stamphog 2.9.1 | .stamphog/policy.yml @ 2846827 · reviewed head 2846827 |
💡 Motivation and Context
AI observability counts how often each tool is called, but can't tell how often a tool fails. Two integrations send tool spans that are missing what's needed for that:
$ai_spanwas sent as soon as the model asked for the tool, before it ran. It never had the tool's output, error flag or latency.$ai_span_type, so they couldn't be told apart from chain spans. A tool that reports its failure as a result (handle_tool_error,ToolMessage(status="error")) ended without an exception, so the span wasn't marked as an error. ConvertedToolMessages also droppedtool_call_id,nameandstatus, so a tool result in the next generation's$ai_inputcouldn't be matched to the tool that produced it.Changes:
processor.py,client.py): each tool span waits for itsToolResultBlock(matched bytool_use_id, including subagent results), then goes out with$ai_output_state(hidden in privacy mode, like$ai_input_state),$ai_is_error,$ai_errorand$ai_latency. Spans still waiting are sent without a result before the trace, when the query ends, if the caller stops iterating early, and ondisconnect(). The query loop is now insidetry/finallyfor the early-stop case, so most of that hunk is re-indentation (git diff -wshows the real change).callbacks.py): tool runs set$ai_span_type: "tool". A tool that returns aToolMessagewithstatus == "error"marks its span$ai_is_errorwith$ai_error. ConvertedToolMessages keeptool_call_id,namewhen set, andstatus: "error".utils.py:_format_tool_errorbuilds$ai_errorfrom tool result content, joining text blocks. In privacy mode it returns the fixed stringToolError, because tool output can contain user data. It sends that label instead of leaving$ai_errorout so the span still reads as an error in the errors view. This is a judgement call for reviewers.Before / After
The same two runs against a mocked client: a Claude Agent SDK
Readcall whose result isis_error=True, and a LangChain tool that raisesToolExceptionwithhandle_tool_error=True.$ai_spanproperties, trimmed:Before (
main)After (this branch)
This pairs with an ingestion change in PostHog that reads tool results out of
$ai_inputto compute per-tool failure rates. Withtool_call_idkept, LangChain results can be matched to tool names there too.💚 How did you test it?
uv run pytest posthog/test/ai: 1881 passed, 15 skipped.test_processor.py(result attached, error, privacy mode, subagent match, sent once and before the trace, sent when the caller stops early),test_client.py(out-of-order results against both the processor and the client,disconnect()flush, span assertions in the multi-turn test), andtest_callbacks.py(span type, errorToolMessage, message conversion).ruff format,ruff check.mainand this branch with a mocked client. I haven't run it against a live Claude Agent SDK or LangChain agent.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Written by Claude Opus 5.5 and Codex in Claude Code, directed by Daniel, as a handover to the AI observability team. Tool spans now wait for their result rather than going out with a placeholder and an update event, because ingestion and the trace view read one span per tool call.
🤖 Generated with Claude Code