-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Description
π΄ Required Information
Describe the Bug:
When the ADK's MCP toolset makes outbound HTTP requests to downstream MCP servers, the httpx client used does not propagate W3C traceparent headers. This means agent β MCP service calls create disconnected traces, breaking distributed trace correlation between the agent and the services it calls tools on.
The root cause is that StreamableHTTPConnectionParams uses a default httpx_client_factory (create_mcp_http_client) that creates a plain httpx.AsyncClient without OpenTelemetry instrumentation. Even though the TracerProvider and TraceContextTextMapPropagator are correctly configured by the ADK, the MCP HTTP client is never instrumented, so the propagator is never invoked on outbound requests.
Steps to Reproduce:
- Deploy an ADK agent with an MCP toolset connecting to a downstream service over HTTP:
campaign_tools = McpToolset( connection_params=StreamableHTTPConnectionParams( url="http://some-mcp-server:8080/mcp", ), )
- Set
OTEL_EXPORTER_OTLP_ENDPOINTon both the agent and the downstream service - Send a message that triggers the agent to call an MCP tool
- Observe in Jaeger that the agent trace and the downstream service trace are separate trace roots β the MCP HTTP request does not carry a
traceparentheader
Expected Behavior:
Outbound MCP HTTP requests should carry the W3C traceparent header from the current trace context, so that the downstream service creates child spans under the agent's trace, producing a single end-to-end distributed trace (e.g. web β agent β MCP service).
Observed Behavior:
The MCP service receives requests without a traceparent header and creates a new trace root. The agent's trace shows execute_tool get_order_details but the downstream service's POST /mcp span is in a separate, unrelated trace.
Workaround: provide a custom httpx_client_factory that instruments the client:
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from google.adk.tools.mcp_tool.mcp_session_manager import create_mcp_http_client
def create_instrumented_mcp_client(headers=None, timeout=None, auth=None):
client = create_mcp_http_client(headers=headers, timeout=timeout, auth=auth)
HTTPXClientInstrumentor.instrument_client(client)
return client
campaign_tools = McpToolset(
connection_params=StreamableHTTPConnectionParams(
url="http://campaign:8080/mcp/mcp",
httpx_client_factory=create_instrumented_mcp_client,
),
)Environment Details:
- ADK Library Version:
google-adk 1.25.0 - Desktop OS: Linux 6.10.14-linuxkit (Docker)
- Python Version: 3.14.3
Model Information:
- Are you using LiteLLM: No
- Which model is being used: gemini-2.5-flash
π‘ Optional Information
Regression:
N/A β create_mcp_http_client has never applied OTel instrumentation. The httpx_client_factory parameter on StreamableHTTPConnectionParams was added as an extension point but the default factory does not instrument.
Logs:
# Default MCP client factory creates a plain httpx.AsyncClient:
def create_mcp_http_client(headers=None, timeout=None, auth=None):
kwargs = {"follow_redirects": True}
if timeout is None:
kwargs["timeout"] = httpx.Timeout(MCP_DEFAULT_TIMEOUT, read=MCP_DEFAULT_SSE_READ_TIMEOUT)
else:
kwargs["timeout"] = timeout
if headers is not None:
kwargs["headers"] = headers
if auth is not None:
kwargs["auth"] = auth
return httpx.AsyncClient(**kwargs)
# ^ No OTel instrumentation applied
# Meanwhile _setup_instrumentation_lib_if_installed() only instruments GoogleGenAiSdkInstrumentor,
# not httpx clients.
Suggested Fix:
In create_mcp_http_client, conditionally instrument the client when OpenTelemetry is available:
def create_mcp_http_client(headers=None, timeout=None, auth=None):
# ... existing logic ...
client = httpx.AsyncClient(**kwargs)
try:
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
HTTPXClientInstrumentor.instrument_client(client)
except ImportError:
pass
return clientThis is consistent with the existing try/except ImportError pattern used for GoogleGenAiSdkInstrumentor in _setup_instrumentation_lib_if_installed().
How often has this issue occurred?:
- Always (100%)