Repository navigation
fix(ai): record tool outcome on Claude Agent SDK and LangChain tool spans - #1282
DanielVisca wants to merge 2 commits into
Conversation
…pans Claude Agent SDK: hold each tool $ai_span until its ToolResultBlock arrives and emit it with $ai_output_state, $ai_is_error, $ai_error and $ai_latency. Spans without a result are flushed on ResultMessage, when the query stream ends, and on client disconnect. Applies to query() and PostHogClaudeSDKClient. LangChain: set $ai_span_type "tool" on tool spans, mark tool errors handled by handle_tool_error (ToolMessage status "error") with $ai_is_error and $ai_error, and keep tool_call_id, name and error status on ToolMessages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 4d24e9d: 3 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
SDK performance: posthog-pythonBenchmarks completed. No changes flagged for review. Flags require at least a 5% change in the median and all six pairs agreeing on direction. This is a noise filter, not proof of a performance change or a CI gate. View all results
Values are medians of paired measurements. Not flagged does not mean unchanged. Unavailable means a measurement could not be provided, not zero. Estimated allocation measurements retain their instrumentation uncertainty. RSS measures the whole process, not just SDK allocations; observed peaks are sampled checkpoints. Paired ranges in the diagnostics describe variability, not confidence intervals. Each shard uses its own runner; compare Before PR against With PR within a row, not across shards. Compared 102 metric rows; 102 rows shown in all results. Omitted or shortened data remain in |
…ing tool spans Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
Tool spans from two AI integrations can't tell you whether the tool worked, so per-tool failure rates can't be computed:
posthog.ai.claude_agent_sdk): the tool$ai_spanwas captured as soon as theToolUseBlockwas seen, before the tool ran. It never had$ai_output_state,$ai_is_error,$ai_erroror$ai_latency. This affected bothquery()/PostHogClaudeAgentProcessorandPostHogClaudeSDKClient.CallbackHandler): tool runs were captured as untyped$ai_spans (no$ai_span_type: "tool"). A tool error handled byhandle_tool_errorends the run normally with aToolMessage(status="error"), so it was recorded as a success.ToolMessages in$ai_inputalso losttool_call_id,nameandstatus, so tool results couldn't be matched to tool names.Changes
Claude Agent SDK (
processor.py,client.py)tool_use_idand emitted when its matchingToolResultBlockarrives. It then carries:$ai_output_state: the result content, redacted and truncated like tool results in$ai_input$ai_is_error: fromblock.is_error$ai_error: the result text, only when the tool errored$ai_latency: time from tool use to tool result$ai_is_errorand$ai_latencyare still sent; the output and error text are not.UserMessages were skipped entirely, but their tool spans were still emitted.query(): onResultMessage, or in afinallyif the stream ends early, fails or is cancelledPostHogClaudeSDKClient: onResultMessageand ondisconnect()query()loop is wrapped intry/finally, which re-indents it. Review with whitespace changes hidden.LangChain (
callbacks.py)$ai_span_type: "tool". Chains, retrievers and agent steps are unchanged.on_tool_end, aToolMessageoutput withstatus == "error"sets$ai_is_error: Trueand$ai_errorto the message text.$ai_errorisNonein privacy mode._convert_message_to_dictkeepstool_call_idandnameon tool messages, plusstatuswhen it is"error".Also adds a sampo changeset (patch).
Before / After
Printed by a small script that runs each integration against a
MagicMockPostHog client and prints the$ai_spanproperties: (a) a Claude Agent SDKBashtool call whoseToolResultBlockhasis_error=True; (b) a LangChain@toolthat raisesToolExceptionwithhandle_tool_error=True, invoked with a tool call inside a chain. Trimmed to the relevant keys.Before
After
Tests
posthog/test/ai/claude_agent_sdk/test_processor.py:posthog/test/ai/claude_agent_sdk/test_client.py: tool results arriving out of order match their own spans, and a span without a result is still emitted. Both run againstquery()andPostHogClaudeSDKClient.posthog/test/ai/langchain/test_callbacks.py:$ai_span_typeis set on tool spans onlyToolMessageconversionuv run pytest posthog/test/ai/claude_agent_sdk posthog/test/ai/langchain: 251 passed, 8 skipped.ruff format --check,ruff checkandmypy | mypy-baseline filterare clean.Open questions for reviewers
$ai_output_stateon Claude tool spans uses the same 5000-character truncation as tool results in$ai_input.$ai_output_statefor aToolMessageis still itsstr()form (unchanged). I left it alone to keep this PR small.🤖 Generated with Claude Code