LangChain affected by SSRF via image_url token counting in ChatOpenAI.get_num_tokens_from_messages

Server-Side Request Forgery (SSRF) in ChatOpenAI Image Token Counting

Summary

The ChatOpenAI.get_num_tokens_from_messages() method fetches arbitrary image_url values without validation when computing token counts for vision-enabled models. This allows attackers to trigger Server-Side Request Forgery (SSRF) attacks by providing malicious image URLs in user input.

Severity

Low - The vulnerability allows SSRF attacks but has limited impact due to:

• Responses are not returned to the attacker (blind SSRF)
• Default 5-second timeout limits resource exhaustion
• Non-image responses fail at PIL image parsing

Impact

An attacker who can control image URLs passed to get_num_tokens_from_messages() can:

• Trigger HTTP requests from the application server to arbitrary internal or external URLs
• Cause the server to access internal network resources (private IPs, cloud metadata endpoints)
• Cause minor resource consumption through image downloads (bounded by timeout)

Note: This vulnerability occurs during token counting, which may happen outside of model invocation (e.g., in logging, metrics, or token budgeting flows).

Details

The vulnerable code path:

1. get_num_tokens_from_messages() processes messages containing image_url content blocks
2. For images without detail: "low", it calls _url_to_size() to fetch the image and compute token counts
3. _url_to_size() performs httpx.get(image_source) on any URL without validation
4. Prior to the patch, there was no SSRF protection, size limits, or explicit timeout