Tavily crawl integration

Integrate with the Tavily crawl tool using LangChain Python.

Tavily is a search engine built specifically for AI agents (LLMs), delivering real-time, accurate, and factual results at speed. Tavily offers a Crawl endpoint that performs a structured web traversal from a starting URL, with built-in content extraction and intelligent discovery.

Overview

Integration details

Class	Package	Serializable	JS support	Version
TavilyCrawl	langchain-tavily	✅	✅

Tool features

Returns artifact	Native async	Return data	Pricing
❌	✅	base URL, page URL, raw content, images, favicon	1,000 free credits / month

Setup

The integration lives in the langchain-tavily package.

pip install -qU langchain-tavily

Credentials

We also need to set our Tavily API key. You can get an API key by visiting this site and creating an account.

import getpass
import os

if not os.environ.get("TAVILY_API_KEY"):
    os.environ["TAVILY_API_KEY"] = getpass.getpass("Tavily API key:\n")

Instantiation

The tool accepts the following parameters during instantiation:

• max_depth (optional, int): Maximum number of hops from the starting URL. Default is 1.
• max_breadth (optional, int): Maximum number of pages to crawl per level. Default is 20.
• limit (optional, int): Maximum total number of pages to crawl. Default is 50.
• instructions (optional, str): Natural-language instructions that guide the crawler.
• select_paths (optional, list[str]): Only crawl URLs containing these path regexes.
• select_domains (optional, list[str]): Only crawl these domain regexes.
• exclude_paths (optional, list[str]): Skip URLs containing these path regexes.
• exclude_domains (optional, list[str]): Skip URLs from these domain regexes.
• allow_external (optional, bool): Allow the crawler to follow external links.
• extract_depth (optional, str): "basic" or "advanced" extraction depth for each crawled page.
• format (optional, str): "markdown" or "text" content format.
• include_images (optional, bool): Include images discovered on each page.

For a comprehensive overview of the available parameters, refer to the Tavily Crawl API documentation.

from langchain_tavily import TavilyCrawl

tool = TavilyCrawl(
    max_depth=1,
    max_breadth=20,
    limit=50,
    # extract_depth="basic",
    # format="markdown",
    # include_images=False,
    # allow_external=False,
)

Invocation

Invoke directly with args

The Tavily crawl tool accepts the following arguments during invocation:

• url (required): The base URL to start crawling from.
• The following arguments can also be set during invocation: instructions, select_paths, select_domains, exclude_paths, exclude_domains, allow_external.

NOTE: The optional arguments are available for agents to dynamically set. If you set an argument during instantiation and then invoke the tool with a different value, the tool will use the value you passed during invocation.

tool.invoke({"url": "https://docs.tavily.com"})

{
  "base_url": "https://docs.tavily.com",
  "results": [
    {
      "url": "https://docs.tavily.com/",
      "raw_content": "![light logo](https://mintcdn.com/tavilyai/HY1Rnt85q4usR4-R/logo/light.svg?fit=max&auto=format&n=HY1Rnt85q4usR4-R&q=85&s=c5c878011f13d458af0997f3a540eb4f)\n![dark logo](https://mintcdn.com/tavilyai/HY1Rnt85q4usR4-R/logo/dark.svg?fit=max&auto=format&n=HY1Rnt85q4usR4-R&q=85&s=1521677768a1f26b34a9ad86d04c62cc)\n\n##### Welcome\n\n## Documentation Index\n\nFetch the complete documentation index at: <https://docs.tavily.com/llms.txt>\n\nUse this file to discover all available pages before exploring further.\n\n# Build with Tavily\n\nYour journey to state-of-the-art web search starts right here.\n\n## Python SDK\n\n`pip install tavily-python`\n\n## JavaScript SDK\n\n`npm i @tavily/core`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.search(\"Who is Leo Messi?\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.extract(\"https://en.wikipedia.org/wiki/Artificial_intelligence\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.crawl(\"https://docs.tavily.com\", instructions=\"Find all pages on the Python SDK\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.map(\"https://docs.tavily.com\")\n\nprint(response)`\n`from tavily import TavilyClient\n\ntavily_client = TavilyClient(api_key=\"tvly-YOUR_API_KEY\")\nresponse = tavily_client.research(\"What are the latest developments in AI?\")\n\nprint(response)`\n\n### Developer Resources\n\n## API Credits Overview\n\n## Rate Limits\n\n## Playground",
      "favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
    },
    {
      "url": "https://docs.tavily.com/documentation/about",
      "raw_content": "Performance\n\nCommitted to speed and efficiency, our API guarantees real-time and trusted information. Our team works hard to improve Tavily’s performance over time.\n\nIntegration-friendly\n\nWe appreciate the essence of adaptability. That’s why integrating our API with your existing setup is a breeze. You can choose our Python library, JavaScript package or a simple API call. You can also use Tavily through any of our supported partners such as LangChain and LlamaIndex.\n\nTransparent & Informative\n\nOur detailed documentation ensures you’re never left in the dark. From setup basics to nuanced features, we’ve got you covered.\n\n## ​\n\nHow does the Search API work? [...] With LLM hallucinations, it’s crucial to optimize for RAG with the right context and information. This is where Tavily comes in, delivering accurate and precise information for your RAG applications.\n\n## ​\n\nGetting started\n\nSign up for Tavily to get your API key. You get 1,000 free API Credits every month. No credit card required.\n\n## Get your free API key You get 1,000 free API Credits every month. No credit card required.\n\nHead to our API Playground to familiarize yourself with our API.To get started with Tavily’s APIs and SDKs using code, head to our Quickstart Guide and follow the steps.\n\nGot questions? Stumbled upon an issue? Simply intrigued? Don’t hesitate! Our support team is always on standby, eager to assist. Join us, dive deep, and redefine your search experience! Contact us! [...] # About - Tavily Docs\n\nSkip to main content\n\nTavily Docs home pageImage 1: light logoImage 2: dark logo\n\nSearch...\n\nCtrl K Ask AI\n\n   Support\n   Get an API key\n   Get an API key\n\nSearch...\n\nNavigation\n\nGetting Started\n\nAbout\n\nHomeIntroductionAPI & SDKsEcosystemExamplesChangelogHelp\n\n   API Playground\n   Community\n   Blog\n\n##### Getting Started\n\n   About\n   Quickstart\n   Credits & Pricing\n   Rate Limits\n\n##### FAQ\n\n   Frequently Asked Questions\n\nOn this page\n\n   Who are we?\n   What is the Tavily Search Engine?\n   Why choose Tavily?\n   How does the Search API work?\n   Getting started\n\nGetting Started\n\n# About\n\nCopy page\n\nWelcome to Tavily!\n\nCopy page\n\nLooking for a step-by-step tutorial to get started in under 5 minutes? Head to our Quickstart guide and start coding!\n\n## ​\n\nWho are we?",
      "favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
    },
    {
      "url": "https://docs.tavily.com/documentation/api-reference/introduction",
      "raw_content": "# Introduction - Tavily Docs\n\nSkip to main content\n\nTavily Docs home pageImage 1: light logoImage 2: dark logo\n\nSearch...\n\n⌘K Ask AI\n\n   Support\n   Get an API key\n   Get an API key\n\nSearch...\n\nNavigation\n\nAPI Reference\n\nIntroduction\n\nHomeIntroductionAPI & SDKsEcosystemExamplesChangelogHelp\n\n   API Playground\n   Community\n   Blog\n\n##### API Reference\n\n   Introduction\n   POST Search\n   POST Extract\n   POST Crawl\n   POST Map\n   Research  \n   GET Usage\n\n##### Enterprise API Reference\n\n   API Key Generator  \n\n##### Python SDK\n\n   Quickstart\n   SDK Reference\n\n##### JavaScript SDK\n\n   Quickstart\n   SDK Reference\n\n##### Best Practices\n\n   Search\n   Extract\n   Crawl\n   Research\n   API Key Management\n\nOn this page\n\n   Base URL\n   Authentication\n   Endpoints\n   Project Tracking\n\nAPI Reference [...] When using the SDKs, you can specify a project using the `project_id` parameter when instantiating the client, or by setting the `TAVILY_PROJECT` environment variable.\n\nTavily Search Next\n\n⌘I\n\nTavily Docs home pageImage 3: light logoImage 4: dark logo\n\nlinkedinxgithubyoutube\n\nResources\n\nAboutTavily Search CrawlerTavily Status\n\nLegal\n\nSecurity & CompliancePrivacy PolicyWebsite Terms of UsePlatform Terms of UseCookie Notice\n\nlinkedinxgithubyoutube\n\nPowered by This documentation is built and hosted on Mintlify, a developer documentation platform\n\nAssistant\n\nResponses are generated using AI and may contain mistakes. [...] API Reference\n\n# Introduction\n\nCopy page\n\nEasily integrate our APIs with your services.\n\nCopy page\n\n> ## Documentation Index\n> \n> \n> Fetch the complete documentation index at: \n> \n> \n> Use this file to discover all available pages before exploring further.\n\n## ​\n\nBase URL\n\nThe base URL for all requests to the Tavily API is:\n\n```\n\n```\n\n## ​\n\nAuthentication\n\nAll Tavily endpoints are authenticated using API keys. Get your free API key.\n\n```\ncurl -X POST  \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer tvly-YOUR_API_KEY\" \\\n  -d '{\"query\": \"Who is Leo Messi?\"}'\n```\n\n## ​\n\nEndpoints\n\n`/search`Tavily’s powerful web search API.\n\n`/extract`Tavily’s powerful content extraction API.\n\n`/crawl` , `/map`Tavily’s intelligent sitegraph navigation and extraction tools.",
      "favicon": "https://docs.tavily.com/mintlify-assets/_mintlify/favicons/tavilyai/SXaxSfweEU3ftIlh/_generated/favicon/apple-touch-icon.png"
    },
    {
      "url": "https://github.com/tavily-ai",
      "raw_content": "JavaScript1.8k239  \n2.   tavily-pythontavily-python Public \nThe Tavily Python SDK allows for easy interaction with the Tavily API, offering the full range of our search, extract, crawl, map, and research functionalities directly from your Python programs. E…\n\nPython1.2k148  \n3.   skillsskills Public \nPython23321  \n4.   meeting-prep-agentmeeting-prep-agent Public \nPrepare for meetings with company and attendee research notes, integrated with Google Calendar through MCP\n\nPython19334  \n5.   tavily-jstavily-js Public \nOfficial TypeScript/JavaScript library for Tavily.\n\nTypeScript839  \n6.   tavily-chattavily-chat Public \nConversational agent that fuses chat data with live web results through Tavily search, extract, and crawl.\n\nTypeScript7216  \n\n### Repositories\n\nLoading\n\nType\n\nSelect type [...] ### Repositories\n\nLoading\n\nType\n\nSelect type\n\nAll Public Sources Forks Archived Mirrors Templates \n\nLanguage\n\nSelect language\n\nAll JavaScript Jupyter Notebook Python TypeScript \n\nSort\n\nSelect order\n\nLast updated Name Stars \n\nShowing 10 of 33 repositories [...] langchain-tavily Public  ### Uh oh! There was an error while loading. [Please reload this page.]( past year of commit activity Python22MIT 1511 Updated Apr 16, 2026  \n   langchainjs Public  Forked from langchain-ai/langchainjs\n🦜🔗 Build context-aware reasoning applications ### Uh oh! There was an error while loading. [Please reload this page.]( past year of commit activity TypeScript 0 MIT 3,20800 Updated Apr 16, 2026",
      "favicon": "https://github.githubassets.com/favicons/favicon.svg"
    }
  ],
  "response_time": 13.45
}

Invoke with ToolCall

We can also invoke the tool with a model-generated ToolCall, in which case a ToolMessage will be returned:

model_generated_tool_call = {
    "args": {"url": "https://docs.tavily.com", "instructions": "Find API reference pages"},
    "id": "1",
    "name": "tavily_crawl",
    "type": "tool_call",
}
tool_msg = tool.invoke(model_generated_tool_call)

print(tool_msg.content[:400])

Use within an agent

We can use the crawl tool directly with an agent by binding it to the model. The agent can then dynamically set the available arguments to control where and how deep to crawl.

if not os.environ.get("OPENAI_API_KEY"):
    os.environ["OPENAI_API_KEY"] = getpass.getpass("OPENAI_API_KEY:\n")

from langchain.chat_models import init_chat_model

model = init_chat_model(model="gpt-5.5", model_provider="openai", temperature=0)

from langchain_tavily import TavilyCrawl
from langchain.agents import create_agent


tavily_crawl_tool = TavilyCrawl(max_depth=1, max_breadth=10, limit=10)

agent = create_agent(model, [tavily_crawl_tool])

user_input = "Crawl https://docs.tavily.com and find the API reference for the search endpoint."

stream = agent.stream_events({"messages": user_input}, version="v3")
for snapshot in stream.values:
    snapshot["messages"][-1].pretty_print()

---

API reference

For detailed documentation of all Tavily Crawl API features and configurations head to the API reference: docs.tavily.com/documentation/api-reference/endpoint/crawl

---

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.

Edit this page on GitHub or file an issue.