Skip to main content

​
What You’ll Build

A ReAct agent that conducts comprehensive research on any company by combining website crawling with targeted web search. Give it a company name, website URL, and optional research focus — it autonomously crawls the site, extracts key pages, searches for external coverage, and produces a cited research report.

View Source on GitHub

​
Architecture

The agent autonomously decides which tools to use and in what order. A typical research flow:
  1. Crawl the company website to discover and summarize pages
  2. Extract detailed content from specific URLs found during crawling
  3. Search the web for external information (news, funding, reviews, competitors)
  4. Synthesize everything into a structured report with citations

​
Tools Used

ToolPurposeTavily Toolkit Function
crawl_company_websiteCrawl and summarize company website pagescrawl_and_summarize
extract_from_urlsExtract detailed content from specific URLsextract_and_summarize
tavily_searchSearch the web for external informationsearch_dedup

​
Quick Start

Source File

pip install tavily-agent-toolkit anthropic claude-agent-sdk python-dotenv

export TAVILY_API_KEY="your-tavily-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"

python company_intelligence_deep_agent.py

​
How It Works

Each tool wraps a Tavily Agent Toolkit function with agent-friendly parameters:
  • crawl_company_website calls crawl_and_summarize with the company URL, optional extraction instructions, and depth/breadth controls. Returns a summarized overview of the crawled pages.
  • extract_from_urls calls extract_and_summarize with specific URLs and an optional query focus. Uses extract_depth="advanced" for full content extraction.
  • tavily_search calls search_dedup with multiple queries in parallel, returning deduplicated and formatted results with search_depth="advanced".
The agent is prompted as a business intelligence analyst:
You are a business intelligence analyst researching companies.

You have three tools available:
- crawl_company_website - Crawl a company's website
- extract_from_urls - Extract content from specific URLs
- tavily_search - Search the web for news, funding, reviews

Combine website insights with external sources for a
complete picture. Include citations [1], [2], etc.
Both implementations stream tool calls as they happen, so you can see the agent’s progress in real time:
[1] Crawling website -> https://anthropic.com
[2] Searching the web -> 3 query/queries
[3] Extracting URLs -> 2 URL(s)
The crawl_and_summarize and extract_and_summarize tools use a dedicated summarizer model (configured via ModelConfig). In the examples, a smaller model is used for summarization to keep costs low while the main agent model handles reasoning.

​
Example Interaction

============================================================
Company Intelligence Research Agent
============================================================

Company name: Anthropic
Website URL:  https://anthropic.com
Research focus: leadership team and recent funding

------------------------------------------------------------
Researching Anthropic (https://anthropic.com)
Focus: leadership team and recent funding
------------------------------------------------------------

[1] Crawling website -> https://anthropic.com
[2] Searching the web -> 3 query/queries
[3] Extracting URLs -> 2 URL(s)

Completed in 23.4s | 4 turns

============================================================
RESEARCH REPORT
============================================================

Anthropic is an AI safety company founded in 2021...
[Comprehensive report with citations]

​
Example Research Topics

  • Company overview and products
  • Leadership team and organizational structure
  • Recent funding rounds and investors
  • Competitive landscape
  • Customer reviews and reputation
  • Technology stack and engineering culture

​
Key Parameters to Tune

ParameterWhereEffect
max_depthcrawl_company_websiteHow deep to crawl from the homepage (default: 2)
max_breadthcrawl_company_websitePages per crawl level (default: 10)
limitcrawl_company_websiteTotal page cap (default: 20)
extract_depthextract_from_urls"basic" or "advanced" for full content
max_resultstavily_searchResults per search query (default: 5)
topictavily_search"general", "news", or "finance"

​
Next Steps

Social Media Research

Add social media intelligence to your agent with platform-specific search.

Hybrid Research

Combine internal company data with web research for deeper analysis.