Every agent you've built so far operates in a text-only world — reading strings, calling functions, producing text. But most real-world interfaces are visual. This lesson teaches agents to see: understanding images, navigating GUIs, and automating visual workflows through the perception-decision-action loop.

SDK Focusimage content blockstool usestructured outputmulti-step loops

Why Agents Need to See

Text-based tools work when you control the interface — APIs, CLIs, structured data. But many real-world systems only expose a visual interface:

text

Text tools work:
  Database → SQL query → Structured result
  API → HTTP request → JSON response
  CLI → Command → Text output

Text tools fail:
  Legacy enterprise app → Only has a GUI, no API
  PDF with charts → Visual data that OCR misses
  Video content → Temporal visual information
  Web application → Dynamic UI that changes with interaction

Multimodal agents extend the tool-use pattern from P2 into the visual domain. Instead of function_call → text_result, the loop becomes screenshot → visual_understanding → action.

Multimodal Input

Image Understanding

Modern LLMs accept images as input alongside text. The agent can analyze screenshots, charts, documents, and photos:

typescript

import { query, type SDKUserMessage } from "@anthropic-ai/claude-agent-sdk";

// `query()` accepts `prompt: string | AsyncIterable<SDKUserMessage>`. To send
// image content blocks, wrap them in a streamed user message.
async function* userInput(screenshotBase64: string): AsyncIterable<SDKUserMessage> {
  yield {
    type: "user",
    session_id: "",
    parent_tool_use_id: null,
    message: {
      role: "user",
      content: [
        {
          type: "image",
          source: { type: "base64", media_type: "image/png", data: screenshotBase64 },
        },
        {
          type: "text",
          text:
            "Describe the UI elements visible in this screenshot. " +
            "List all buttons, input fields, and their current states.",
        },
      ],
    },
  };
}

const response = query({ prompt: userInput(screenshotBase64) });

// query() returns an AsyncIterable — stream messages and grab the final result.
for await (const message of response) {
  if (message.type === "result" && message.subtype === "success") {
    console.log(message.result);
  }
}

Multimodal & GUI Agents

Why Agents Need to See

Multimodal Input

Image Understanding