---
name: tavilyai
description: Use when building agents that need web search, content extraction,
  site crawling, or research capabilities. Reach for Tavily when your agent must
  access current information, pull content from URLs, map site structures, or
  synthesize multi-source research reports.
metadata:
  mintlify-proj: tavilyai
  version: "1.0"
---

# Tavily Skill

## Product Summary

Tavily is an AI-optimized web layer for agents: real-time search, page extraction, site crawling and mapping, and cited research. It returns clean, LLM-ready content instead of raw HTML or shallow snippets, reducing token waste and improving agent reliability.

**Key files and commands:**
- CLI: `tvly` (install via `curl -fsSL https://cli.tavily.com/install.sh | bash`)
- SDKs: Python (`tavily-python`), JavaScript (`@tavily/core`)
- MCP server: `https://mcp.tavily.com/mcp/` (for Claude, Cursor, ChatGPT, Codex, etc.)
- API endpoints: `/search`, `/extract`, `/crawl`, `/map`, `/research`
- Free tier: 1,000 credits/month (no credit card required)
- Keyless access: `/search` and `/extract` work with zero setup via `X-Tavily-Access-Mode: keyless` header

**Primary docs:** https://docs.tavily.com

## When to Use

Reach for Tavily when:
- **Agent needs current information** — search the web for real-time data, news, or recent developments
- **Extract content from URLs** — pull clean markdown/text from 1–20 URLs without parsing HTML or handling JavaScript
- **Crawl a site** — traverse multiple pages and extract content from a domain (e.g., documentation, competitive research)
- **Map site structure** — discover URLs on a site without extracting content (faster than crawling)
- **Research a topic** — synthesize multi-source research into a cited report
- **Enrich data** — fill in company/person records from the web (via Agent Skills)
- **Monitor for changes** — track news, pricing pages, or job postings on a cadence

Do not use Tavily for:
- Accessing private/authenticated content (Tavily only crawls public web)
- Bypassing robots.txt or site terms of service
- Bulk scraping without respecting rate limits and site policies

## Quick Reference

### API Endpoints & Credits

| Endpoint | Cost | Use case |
|----------|------|----------|
| `/search` | 1 credit (basic/fast/ultra-fast), 2 credits (advanced) | Web search with ranked results |
| `/extract` | 1 credit per 5 URLs (basic), 2 credits per 5 URLs (advanced) | Pull content from URLs |
| `/crawl` | 1 credit per 10 pages (basic), 2 credits per 10 pages (advanced) | Traverse and extract from a site |
| `/map` | 1 credit per 10 URLs | Discover site structure |
| `/research` | Variable (multi-step) | Comprehensive research report |

### CLI Commands

```bash
tvly search "query"                    # Search the web
tvly extract <url> [<url> ...]        # Extract from 1–20 URLs
tvly crawl <url>                      # Crawl a site
tvly map <url>                        # Discover URLs
tvly research "topic"                 # Deep research
tvly login --api-key tvly-YOUR_KEY    # Authenticate
tvly --status                         # Check auth status
```

### Authentication

| Method | When to use |
|--------|------------|
| **Keyless** (`X-Tavily-Access-Mode: keyless` header) | Testing, light use, `/search` and `/extract` only |
| **API key** (`Authorization: Bearer tvly-YOUR_KEY`) | Production, all endpoints, higher limits |
| **OAuth** (MCP clients) | Claude, Cursor, ChatGPT — sign in via browser |
| **Environment variable** (`TAVILY_API_KEY`) | CLI and SDKs |

### Search Depth Trade-offs

| Depth | Latency | Relevance | Content type | Cost | Best for |
|-------|---------|-----------|--------------|------|----------|
| `ultra-fast` | Lowest | Lower | Summary | 1 credit | Real-time, speed-critical |
| `fast` | Low | Good | Chunks | 1 credit | Quick, targeted snippets |
| `basic` | Medium | High | Chunks | 1 credit | General-purpose (default) |
| `advanced` | Higher | Highest | Chunks | 2 credits | Niche topics, detailed queries |

### Extract Parameters

| Parameter | Use case |
|-----------|----------|
| `query` | Rerank chunks by relevance to a query (enables `chunks_per_source`) |
| `chunks_per_source` | Limit content per URL (1–5 chunks, ~800 chars each) |
| `extract_depth: advanced` | Handle JavaScript-rendered pages, tables, media |
| `format: markdown` or `text` | Output format |

### Crawl Parameters

| Parameter | Use case |
|-----------|----------|
| `max_depth` | How many levels deep to crawl (1–5) |
| `max_breadth` | Links to follow per page (default 20) |
| `limit` | Total page cap (default 50) |
| `instructions` | Natural language guidance (e.g., "only API docs") |
| `select_paths` / `exclude_paths` | Regex patterns to filter URLs |
| `chunks_per_source` | Relevant chunks per page (requires `instructions`) |

## Decision Guidance

### When to use Search vs Extract vs Crawl

| Scenario | Use |
|----------|-----|
| "Find recent news about X" | `/search` with `topic: news` |
| "Get content from this URL" | `/extract` (1–20 URLs) |
| "Get content from multiple URLs I already have" | `/extract` (batch up to 20) |
| "Explore a site and extract from many pages" | `/crawl` (auto-discover and extract) |
| "Just find URLs on a site, don't extract yet" | `/map` (faster, cheaper) |
| "Synthesize research from many sources" | `/research` (multi-step, cited report) |

### When to use Search Depth

| Query type | Depth | Reason |
|-----------|-------|--------|
| "Latest AI news" | `basic` | Fast, general-purpose |
| "Niche technical topic" | `advanced` | Reaches more sources, higher relevance |
| "Real-time stock price" | `ultra-fast` | Speed matters most |
| "Detailed comparison of X vs Y" | `advanced` | Complex query benefits from broader search |

### When to use Extract Depth

| Content | Depth |
|---------|-------|
| Simple blog posts, news articles | `basic` |
| JavaScript-heavy SPAs, tables, embedded media | `advanced` |

### Search with include_raw_content vs Extract API

| Approach | When to use |
|----------|------------|
| `search(..., include_raw_content=True)` | Quick prototyping, search results are likely relevant |
| `extract(urls=[...])` | You already have specific URLs; need control over which ones |

## Workflow

### Typical agent task: Research a topic

1. **Understand the request** — What does the user need? A quick fact, a comparison, a report, or ongoing monitoring?

2. **Choose the right endpoint:**
   - Quick fact → `/search` with `max_results: 3–5`
   - Detailed comparison → `/search` with `search_depth: advanced`, `max_results: 10`
   - Comprehensive report → `/research` (multi-step, cited)
   - Extract from known URLs → `/extract`

3. **Set parameters:**
   - For search: set `search_depth`, `max_results`, `topic` (news/general/finance), date filters if needed
   - For extract: set `query` (to rerank chunks), `chunks_per_source` (1–5), `extract_depth` if JavaScript-heavy
   - For crawl: set `max_depth`, `limit`, `instructions` (natural language guidance)

4. **Make the request:**
   ```python
   from tavily import TavilyClient
   client = TavilyClient(api_key="tvly-YOUR_KEY")
   response = client.search("your query", search_depth="basic", max_results=5)
   ```

5. **Handle the response:**
   - Check `response['results']` for search results (title, url, content, score)
   - Check `response['answer']` if `include_answer` was set
   - Check `response['failed_results']` for extract/crawl failures per URL

6. **Filter and deduplicate:**
   - Use `score` to filter low-relevance results (e.g., `score > 0.5`)
   - Deduplicate URLs across multiple searches (canonicalize by removing `?` params and trailing `/`)
   - Combine unique chunks with `[...]` separator to save tokens

7. **Verify and return:**
   - Confirm results are relevant and current
   - Pass `include_usage: True` to track credits spent
   - Return results to the user or feed into downstream processing

### Typical agent task: Extract content from URLs

1. **Gather URLs** — From search results, user input, or prior crawl

2. **Batch up to 20 URLs** — Extract accepts 1–20 per request

3. **Set extraction parameters:**
   - If you have a query: set `query` + `chunks_per_source` (1–5) for focused extraction
   - If you need full content: omit `query` (returns all content)
   - If page is JavaScript-heavy: set `extract_depth: advanced`

4. **Make the request:**
   ```python
   response = client.extract(
       urls=["https://example.com/page1", "https://example.com/page2"],
       query="authentication API",
       chunks_per_source=3,
       extract_depth="basic"
   )
   ```

5. **Handle results:**
   - `response['results']` contains successful extractions (url, raw_content)
   - `response['failed_results']` lists URLs that failed (url, error)
   - Retry failed URLs individually or skip them

6. **Process content:**
   - Parse markdown/text from `raw_content`
   - If `query` was set, chunks are already reranked by relevance
   - Combine chunks with `[...]` separator if needed

### Typical agent task: Crawl a site

1. **Start with `/map`** — Discover structure first, cheaper than crawling

2. **Set map parameters:**
   ```bash
   tvly map https://docs.example.com --max-depth 2 --limit 100
   ```

3. **Review discovered URLs** — Decide if you need to crawl or extract specific pages

4. **Crawl if needed:**
   ```bash
   tvly crawl https://docs.example.com \
     --max-depth 2 \
     --instructions "focus on API reference pages" \
     --chunks-per-source 3 \
     --output-dir ./docs
   ```

5. **Monitor results:**
   - Check `response['results']` for extracted pages
   - Check `response['failed_results']` for pages that failed
   - Adjust `max_depth`, `limit`, or `instructions` if results are incomplete

## Common Gotchas

- **Keyless access is rate-limited** — Test with keyless, but upgrade to an API key for production. Keyless responses are identical to keyed responses; no code changes needed.

- **Search returns chunks, not full pages** — By default, `/search` returns 3 snippets per source (~500 chars each). Use `/extract` if you need full page content.

- **Extract accepts up to 20 URLs per request** — Batch them, but don't exceed 20 in one call. Retry failed URLs individually.

- **Crawl can be expensive** — Start with `max_depth: 1`, `limit: 50`. Increase gradually. Use `instructions` to focus crawling on relevant pages.

- **Query parameter is required for chunks_per_source** — You can't use `chunks_per_source` without `query` in extract/crawl. If you need to limit content without a query, use `include_raw_content: false` in search.

- **robots.txt and rate limits matter** — Respect site policies. Implement delays between requests. Monitor API usage with `/usage` endpoint.

- **Session tracking is optional but recommended** — Pass `session_id` and `human_id` across multi-step workflows so Tavily can link related calls and improve response quality.

- **Search depth affects cost** — `advanced` costs 2 credits vs 1 for `basic/fast/ultra-fast`. Set it explicitly to control costs; don't rely on `auto_parameters` unless you want the extra cost.

- **Extract depth affects latency** — `advanced` extraction is slower but handles JavaScript-rendered pages and tables better. Use `basic` for simple content.

- **Date filters don't guarantee freshness** — `time_range` filters by publish date, but cached pages may be old. Use `max_age_hours: 0` to always fetch live (adds latency).

- **Exact match narrows results** — `exact_match: true` requires the phrase to appear verbatim. May return fewer results or empty fields.

## Verification Checklist

Before submitting work:

- [ ] **API key is set** — Check `TAVILY_API_KEY` env var or passed to client
- [ ] **Endpoint is correct** — Use `/search` for discovery, `/extract` for content, `/crawl` for multi-page, `/research` for reports
- [ ] **Parameters are reasonable** — `max_results` ≤ 20, `chunks_per_source` 1–5, `max_depth` 1–5, crawl `limit` ≤ 100
- [ ] **Search depth is intentional** — `basic` by default; use `advanced` only when needed (costs 2 credits)
- [ ] **Batch size is correct** — Extract/crawl: up to 20 URLs per request
- [ ] **Error handling is in place** — Check `failed_results`, retry with backoff, don't fail on single URL errors
- [ ] **Rate limits are respected** — Cap concurrent requests (e.g., 5–20 in-flight), implement exponential backoff on 429
- [ ] **Session tracking is used** — Pass `session_id` and `human_id` for multi-step workflows
- [ ] **Results are deduplicated** — Canonicalize URLs, combine unique chunks
- [ ] **Credits are tracked** — Pass `include_usage: True`, sum usage across batch
- [ ] **Content is validated** — Confirm results are relevant, current, and not duplicates

## Resources

**Comprehensive navigation:** https://docs.tavily.com/llms.txt (page-by-page index for agents)

**Critical docs:**
- [API Reference](https://docs.tavily.com/documentation/api-reference/endpoint/search) — All endpoints, parameters, response schemas
- [Best Practices for Search](https://docs.tavily.com/documentation/best-practices/best-practices-search) — Query optimization, depth trade-offs, batch workflows, deduplication
- [Best Practices for Extract](https://docs.tavily.com/documentation/best-practices/best-practices-extract) — Extraction strategies, filtering, multi-step pipelines

---

> For additional documentation and navigation, see: https://docs.tavily.com/llms.txt