Giving the Agent Eyes: Why Web Search Matters and How I Set It Up

An AI agent without web search is just a very confident liar. Here's how I wired up Tavily + Chrome CDP and cut the dead weight.

2026-04-17 · 4 min read

The Problem with Blind Agents

An AI agent that can't search the web is a closed system. It knows what was in its training data and nothing else. Ask it about yesterday's kernel update, this week's HN trending posts, or whether a package has a known CVE — and it'll either apologise or, worse, confidently hallucinate something plausible.

I learned this the hard way. My agent (Dade, running locally via Hermes) was fast at terminal work, file ops, and code generation, but the moment a task touched the outside world it hit a wall. "What's the latest Ubuntu HWE kernel?" — silence. "Is there a fix for this diffusers bug?" — guesswork.

Web search isn't a nice-to-have. It's the difference between an agent that reasons about the world and one that reasons about its training snapshot.

The Stack I Ended Up With

After trying a few combinations, here's what stuck:

web_search  →  Tavily API (fast, structured results)
web_extract →  Tavily API (clean markdown extraction)
browser     →  Chrome CDP headless (JS-heavy pages, interactive content)

Tavily for Search and Extract

Tavily is an API built specifically for AI agents. It returns structured results — title, URL, description — without the HTML soup you'd get from scraping a search page. The free tier gives 1,000 searches per month, which is more than enough for personal use.

The extract endpoint is the real gem. Give it a URL and it returns the page content as clean markdown — headings, lists, code blocks, all preserved. I tested it on an OMG Ubuntu article and got the full structured content back, not a garbled text dump.

# config.yaml
web:
  backend: tavily

# .env
TAVILY_API_KEY=tvly-...

Chrome CDP for the Hard Cases

Not every page yields to a simple HTTP fetch. JavaScript-rendered SPAs, pages behind auth flows, anything that needs interaction — these need a real browser. I set up Chrome headless with remote debugging:

# Start Chrome headless with CDP on port 9222
~/apps/google-chrome/google-chrome \
  --headless=new \
  --no-sandbox \
  --remote-debugging-port=9222 \
  --disable-gpu \
  --disable-dev-shm-usage \
  --user-data-dir=/tmp/chrome-hermes-profile

Wrapped it in a systemd user service so it auto-starts on boot. The BROWSER_CDP_URL=http://localhost:9222 env var tells Hermes to route browser tools there.

Cost: literally zero. It's just a local Chrome instance.

The Fallback Chain

In practice, the three tiers form a natural fallback:

1. Tavily search — fastest, no browser needed. Good for "find me X" queries.
2. Tavily extract — pulls full page content as markdown. Works for ~90% of articles.
3. Chrome CDP — for anything Tavily can't handle. JS-heavy pages, dashboards, interactive content.

If all three fail, the task probably needs human intervention anyway.

Cutting the Dead Weight

I started with Firecrawl in the stack too. It's a dedicated scraping API — handles JS rendering, anti-bot, the works. But in practice, Tavily extract handled almost everything I threw at it, and Chrome CDP covered the rest. Firecrawl became the middle child: not the fastest option, not the most capable.

Same story with fal.ai for image generation. The API was exhausted, and I moved to local generation (more on that in another post). Having dead keys in .env is just attack surface and config clutter.

# Removed from .env
# FIRECRAWL_API_KEY=...  ← Tavily + Chrome cover this
# FAL_KEY=...            ← local generation covers this

Clean config, fewer dependencies, same capability. Always a win.

Why This Matters for Local-First AI

The whole point of running locally is control. But control without information is just isolation. Your agent needs to reach the web, and it needs to do it without routing everything through a cloud overlay that charges per request and goes down when their servers do.

Tavily is a lightweight API — one HTTP call, structured results, done. Chrome CDP is a local process you own completely. Neither requires a Docker container, a serverless function, or a managed browser fleet.

The setup took maybe 15 minutes. The difference in agent capability? Massive.

The Config at a Glance

Three tools, two API keys, one local browser. That's the minimum viable web presence for an AI agent.

Found this useful? 👉 Follow @Raf_VRS for more AI Guides.

👉 Support the work: ko-fi.com/rafvrs

Stop Scrolling. Start Building. #HardInterference #AIAgents #SelfHosting