AnyCrawl

Search

Search Engine Results Page (SERP) API for querying search engines and retrieving structured results.

Introduction

AnyCrawl Search API enables you to programmatically access search engine results from multiple search engines. Built specifically for Large Language Models (LLMs), it returns clean, structured data that's optimized for AI processing and analysis.

Key Features: The API returns data immediately and synchronously - no polling or webhooks required. It supports multiple search engines and multi-page result retrieval for comprehensive data collection.

Core Features

  • Multi-Engine Support: Currently supports Google search with plans for additional engines
  • LLM Optimized: Returns structured JSON data perfect for AI processing
  • Multi-Language Support: Search in different languages and regions
  • Multi-Page Results: Retrieve multiple pages of search results in a single request
  • High Performance: Native high concurrency support with asynchronous processing
  • Immediate Response: Synchronous API - get results instantly without polling
  • Comprehensive Results: Returns both search results and search suggestions

API Endpoint

POST https://api.anycrawl.dev/v1/search

Request Parameters

ParameterTypeRequiredDefaultDescription
querystringYes-Search query string to be executed
engineenumNogoogleSearch engine to use. Currently supported: google
limitnumberNo10Maximum number of results per page
offsetnumberNo0Number of results to skip (for pagination)
pagesnumberNo1Number of search result pages to retrieve (min: 1, max: 20)
langstringNoenLanguage locale for search results (e.g., 'en', 'zh', 'es', 'fr')
safeSearchnumber | nullNonullSafe search filter level for Google: 0 (off), 1 (medium), 2 (high), or null (default). Only applicable to Google search engine.

Supported Search Engines

google (Default)

  • Use Case: General web search with comprehensive results
  • Advantages: Most comprehensive search results, supports multiple languages and regions
  • Features: Returns both search results and search suggestions. Supports safe search filtering (off/medium/high)
  • Recommended For: General web searches, research, content discovery
  • Safe Search: Supports safeSearch parameter with values 0 (off), 1 (medium), 2 (high), or null (default)

Response Format

Success Response (HTTP 200)

{
    "success": true,
    "data": [
        {
            "title": "AlsoAsked: People Also Ask keyword research tool",
            "url": "https://alsoasked.com/",
            "description": "Find the questions people also ask. Enter a question, brand or search query. e.g. 'keyword research'.",
            "source": "Google Search Result"
        },
        {
            "title": "OpenAI ChatGPT Features and Benefits",
            "url": "https://openai.com/chatgpt",
            "description": "ChatGPT is an AI assistant that can help with writing, research, and problem-solving.",
            "source": "Google Search Result"
        },
        {
            "title": "Keyword tool",
            "source": "Google Suggestions"
        }
    ]
}

Search Result Types

The API returns two types of results:

  1. Search Results: Complete results with title, URL, description, and source
  2. Search Suggestions: Related query suggestions from the search engine

Error Responses

400 - Validation Error

{
    "success": false,
    "error": "Validation error",
    "details": {
        "issues": [
            {
                "field": "engine",
                "message": "Invalid enum value. Expected 'google', received 'invalid'",
                "code": "invalid_enum_value"
            }
        ],
        "messages": ["Invalid enum value. Expected 'google', received 'invalid'"]
    }
}

401 - Authentication Error

{
    "success": false,
    "error": "Invalid API key"
}

402 - Payment Required

{
    "success": false,
    "error": "Insufficient credits. Please upgrade your plan or purchase more credits."
}

Usage Examples

cURL

Basic Search (Default Google Engine)

curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "OpenAI ChatGPT"
  }'
curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "machine learning tutorials",
    "pages": 3,
    "limit": 10
  }'

Search with Language/Region Specification

curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "人工智能",
    "lang": "zh",
    "pages": 2
  }'

Search with Pagination

curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "web scraping tools",
    "offset": 20,
    "limit": 10
  }'
curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "educational content",
    "safeSearch": 0
  }'

JavaScript/TypeScript

const response = await fetch("https://api.anycrawl.dev/v1/search", {
    method: "POST",
    headers: {
        Authorization: "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    body: JSON.stringify({
        query: "artificial intelligence trends",
        engine: "google",
    }),
});

const result = await response.json();
console.log(result.data); // Array of search results

Advanced Search with Multiple Pages

interface SearchOptions {
    query: string;
    engine?: "google";
    pages?: number;
    limit?: number;
    lang?: string;
    safeSearch?: number | null; // 0: off, 1: medium, 2: high, null: default (Google only)
}

async function searchWithPagination(options: SearchOptions) {
    const response = await fetch("https://api.anycrawl.dev/v1/search", {
        method: "POST",
        headers: {
            Authorization: "Bearer YOUR_API_KEY",
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            query: options.query,
            engine: options.engine || "google",
            pages: options.pages || 1,
            limit: options.limit || 10,
            lang: options.lang || "en",
            safeSearch: options.safeSearch,
        }),
    });

    const result = await response.json();

    if (result.success) {
        return result.data.filter((item) => item.url); // Filter only results with URLs
    } else {
        throw new Error(result.error);
    }
}

// Usage
const searchResults = await searchWithPagination({
    query: "best web scraping APIs",
    pages: 3,
    limit: 10,
    lang: "en",
});

Python

Basic Search

import requests

response = requests.post(
    'https://api.anycrawl.dev/v1/search',
    headers={
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json'
    },
    json={
        'query': 'data science tools',
        'engine': 'google'
    }
)

result = response.json()
if result['success']:
    for item in result['data']:
        if 'url' in item:  # Search result
            print(f"Title: {item['title']}")
            print(f"URL: {item['url']}")
            print(f"Description: {item['description']}")
            print("---")
        else:  # Search suggestion
            print(f"Suggestion: {item['title']}")
else:
    print(f"Search failed: {result.get('error', 'Unknown error')}")

Advanced Search with Error Handling

import requests
from typing import List, Dict, Any

def search_web(query: str, pages: int = 1, lang: str = "en") -> List[Dict[str, Any]]:
    """
    Search the web using AnyCrawl Search API

    Args:
        query: Search query string
        pages: Number of pages to retrieve
        lang: Language locale

    Returns:
        List of search results

    Raises:
        Exception: If search fails
    """
    try:
        response = requests.post(
            'https://api.anycrawl.dev/v1/search',
            headers={
                'Authorization': 'Bearer YOUR_API_KEY',
                'Content-Type': 'application/json'
            },
            json={
                'query': query,
                'pages': pages,
                'lang': lang,
                'limit': 10
            },
            timeout=30
        )

        response.raise_for_status()
        result = response.json()

        if result['success']:
            return result['data']
        else:
            raise Exception(f"Search failed: {result.get('error', 'Unknown error')}")

    except requests.RequestException as e:
        raise Exception(f"Request failed: {str(e)}")

# Usage
try:
    results = search_web("machine learning frameworks", pages=2, lang="en")
    print(f"Found {len(results)} results")

    # Separate search results from suggestions
    search_results = [r for r in results if 'url' in r]
    suggestions = [r for r in results if 'url' not in r]

    print(f"Search Results: {len(search_results)}")
    print(f"Suggestions: {len(suggestions)}")

except Exception as e:
    print(f"Error: {e}")

Best Practices

Search Engine Selection Guidelines

  1. General Web Search → Use google (currently the only supported engine)
  2. Multi-language Searches → Specify appropriate lang parameter
  3. Comprehensive Research → Use multiple pages to get more results

Performance Optimization

  • Batch Requests: Use the pages parameter to get multiple pages in one request instead of making separate calls
  • Language Targeting: Specify language for region-specific and localized results
  • Query Optimization: Use specific keywords and phrases for better result quality

Error Handling

try {
    const response = await fetch("/v1/search", {
        method: "POST",
        headers: {
            Authorization: "Bearer YOUR_API_KEY",
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            query: "artificial intelligence",
            pages: 2,
        }),
    });

    const result = await response.json();

    if (result.success) {
        // Handle successful result
        console.log("Found results:", result.data.length);

        // Separate search results from suggestions
        const searchResults = result.data.filter((item) => item.url);
        const suggestions = result.data.filter((item) => !item.url);

        console.log("Search Results:", searchResults.length);
        console.log("Suggestions:", suggestions.length);
    } else {
        // Handle search failure
        console.error("Search failed:", result.error);
    }
} catch (error) {
    // Handle network error
    console.error("Request failed:", error);
}

High Concurrency Usage

The API natively supports high concurrency. You can make multiple simultaneous search requests without rate limiting concerns:

// Concurrent searching example
const queries = ["machine learning", "artificial intelligence", "data science"];

const searchPromises = queries.map((query) =>
    fetch("/v1/search", {
        method: "POST",
        headers: {
            Authorization: "Bearer YOUR_API_KEY",
            "Content-Type": "application/json",
        },
        body: JSON.stringify({ query, engine: "google", pages: 2 }),
    }).then((res) => res.json())
);

// All requests execute concurrently and return immediately
const results = await Promise.all(searchPromises);
console.log(`Completed ${results.length} searches simultaneously`);

Rate Limits and Credits

  • Each search request consumes credits based on the number of pages requested
  • Credit consumption = number of pages (e.g., pages: 3 = 3 credits)
  • No rate limiting for concurrent requests - the API natively supports high concurrency
  • Monitor your credit usage through the API response headers

Frequently Asked Questions

Q: When should I use different language settings?

A:

  • English (en): For global search results and international content
  • Chinese (zh): For Chinese-language content and China-focused results
  • Regional variants (e.g., en-GB, fr-CA): For country-specific results and localized content

Q: Why do some searches return fewer results than expected?

A: Possible reasons include:

  • Search query is too specific or uses uncommon terms
  • Language mismatch between query and lang parameter
  • Search engine limitations for certain topics or regions
  • Network connectivity issues

Q: How to handle searches requiring specific regions?

A: Currently the API automatically selects appropriate regions based on the lang parameter. Recommendations:

  • Use language codes that match your target region (e.g., en-GB for UK results)
  • Craft search queries with region-specific terms when needed

Q: What are the differences between search results and suggestions?

A:

  • Search Results: Complete entries with title, URL, description, and source
  • Search Suggestions: Related query suggestions without URLs, useful for query expansion

Q: Is there rate limiting for concurrent search requests?

A: No, the API natively supports high concurrency. You can make multiple simultaneous search requests without worrying about rate limits, and all requests return data immediately.

Language and Region Support

The Search API supports multiple languages and regions through the lang parameter. Below are all supported locales:

Major Languages

  • all - All languages

English 🌐

  • en - Global English
  • en-AU - English (Australia) 🇦🇺
  • en-CA - English (Canada) 🇨🇦
  • en-GB - English (United Kingdom) 🇬🇧
  • en-IE - English (Ireland) 🇮🇪
  • en-IN - English (India) 🇮🇳
  • en-NZ - English (New Zealand) 🇳🇿
  • en-PH - English (Philippines) 🇵🇭
  • en-PK - English (Pakistan) 🇵🇰
  • en-SG - English (Singapore) 🇸🇬
  • en-US - English (United States) 🇺🇸
  • en-ZA - English (South Africa) 🇿🇦

Chinese 🌐

  • zh - Global Chinese
  • zh-CN - Chinese (China) 🇨🇳
  • zh-HK - Chinese (Hong Kong) 🇭🇰
  • zh-TW - Chinese (Taiwan) 🇹🇼

Spanish 🌐

  • es - Global Spanish
  • es-AR - Spanish (Argentina) 🇦🇷
  • es-CL - Spanish (Chile) 🇨🇱
  • es-CO - Spanish (Colombia) 🇨🇴
  • es-ES - Spanish (Spain) 🇪🇸
  • es-MX - Spanish (Mexico) 🇲🇽
  • es-PE - Spanish (Peru) 🇵🇪

French 🌐

  • fr - Global French
  • fr-BE - French (Belgium) 🇧🇪
  • fr-CA - French (Canada) 🇨🇦
  • fr-CH - French (Switzerland) 🇨🇭
  • fr-FR - French (France) 🇫🇷

German 🌐

  • de - Global German
  • de-AT - German (Austria) 🇦🇹
  • de-BE - German (Belgium) 🇧🇪
  • de-CH - German (Switzerland) 🇨🇭
  • de-DE - German (Germany) 🇩🇪

Portuguese 🌐

  • pt - Global Portuguese
  • pt-BR - Portuguese (Brazil) 🇧🇷
  • pt-PT - Portuguese (Portugal) 🇵🇹

Italian 🌐

  • it - Global Italian
  • it-CH - Italian (Switzerland) 🇨🇭
  • it-IT - Italian (Italy) 🇮🇹

Japanese 🌐

  • ja - Global Japanese
  • ja-JP - Japanese (Japan) 🇯🇵

Korean 🌐

  • ko - Global Korean
  • ko-KR - Korean (South Korea) 🇰🇷

Russian 🌐

  • ru - Global Russian
  • ru-RU - Russian (Russia) 🇷🇺

European Languages

Dutch 🌐

  • nl - Global Dutch
  • nl-BE - Dutch (Belgium) 🇧🇪
  • nl-NL - Dutch (Netherlands) 🇳🇱

Polish 🌐

  • pl - Global Polish
  • pl-PL - Polish (Poland) 🇵🇱

Czech 🌐

  • cs - Global Czech
  • cs-CZ - Czech (Czech Republic) 🇨🇿

Hungarian 🌐

  • hu - Global Hungarian
  • hu-HU - Hungarian (Hungary) 🇭🇺

Romanian 🌐

  • ro - Global Romanian
  • ro-RO - Romanian (Romania) 🇷🇴

Greek 🌐

  • el - Global Greek
  • el-GR - Greek (Greece) 🇬🇷

Bulgarian 🌐

  • bg - Global Bulgarian
  • bg-BG - Bulgarian (Bulgaria) 🇧🇬

Croatian 🌐

  • hr - Global Croatian

Slovak 🌐

  • sk - Global Slovak

Slovenian 🌐

  • sl - Global Slovenian

Estonian 🌐

  • et - Global Estonian
  • et-EE - Estonian (Estonia) 🇪🇪

Latvian 🌐

  • lv - Global Latvian

Lithuanian 🌐

  • lt - Global Lithuanian

Finnish 🌐

  • fi - Global Finnish
  • fi-FI - Finnish (Finland) 🇫🇮

Swedish 🌐

  • sv - Global Swedish
  • sv-SE - Swedish (Sweden) 🇸🇪

Norwegian 🌐

  • nb - Norwegian Bokmål
  • nb-NO - Norwegian Bokmål (Norway) 🇳🇴

Danish 🌐

  • da - Global Danish
  • da-DK - Danish (Denmark) 🇩🇰

Icelandic 🌐

  • is - Global Icelandic

Other Languages

Arabic 🌐

  • ar - Global Arabic
  • ar-SA - Arabic (Saudi Arabia) 🇸🇦

Hebrew 🇮🇱

  • he - Hebrew

Turkish 🌐

  • tr - Global Turkish
  • tr-TR - Turkish (Turkey) 🇹🇷

Persian 🌐

  • fa - Persian

Hindi 🌐

  • hi - Hindi

Urdu 🌐

  • ur - Urdu

Thai 🌐

  • th - Global Thai
  • th-TH - Thai (Thailand) 🇹🇭

Vietnamese 🌐

  • vi - Global Vietnamese
  • vi-VN - Vietnamese (Vietnam) 🇻🇳

Indonesian 🌐

  • id - Global Indonesian
  • id-ID - Indonesian (Indonesia) 🇮🇩

Indian Languages

  • kn - Kannada 🌐
  • ml - Malayalam 🌐
  • mr - Marathi 🌐
  • ta - Tamil 🌐
  • te - Telugu 🌐

Other European Languages

  • af - Afrikaans 🌐
  • be - Belarusian 🌐
  • ca - Catalan 🌐
  • cy - Welsh 🌐
  • eu - Basque 🌐
  • ga - Irish 🌐
  • gd - Scottish Gaelic 🌐
  • gl - Galician 🌐
  • sq - Albanian 🌐
  • uk - Ukrainian 🌐

Usage Notes

  • Global codes (e.g., en, zh, es) provide broader search results without country restrictions
  • Country-specific codes (e.g., en-US, zh-CN, es-MX) return localized results for specific regions
  • The API automatically selects the appropriate Google domain and language settings based on your locale specification
  • For best results, match the lang parameter to your search query's language

On this page

IntroductionCore FeaturesAPI EndpointRequest ParametersSupported Search Enginesgoogle (Default)Response FormatSuccess Response (HTTP 200)Successful SearchSearch Result TypesError Responses400 - Validation Error401 - Authentication Error402 - Payment RequiredUsage ExamplescURLBasic Search (Default Google Engine)Multi-Page SearchSearch with Language/Region SpecificationSearch with PaginationSearch with Safe SearchJavaScript/TypeScriptBasic SearchAdvanced Search with Multiple PagesPythonBasic SearchAdvanced Search with Error HandlingBest PracticesSearch Engine Selection GuidelinesPerformance OptimizationError HandlingHigh Concurrency UsageRate Limits and CreditsFrequently Asked QuestionsQ: When should I use different language settings?Q: Why do some searches return fewer results than expected?Q: How to handle searches requiring specific regions?Q: What are the differences between search results and suggestions?Q: Is there rate limiting for concurrent search requests?Language and Region SupportMajor LanguagesEnglish 🌐Chinese 🌐Spanish 🌐French 🌐German 🌐Portuguese 🌐Italian 🌐Japanese 🌐Korean 🌐Russian 🌐European LanguagesDutch 🌐Polish 🌐Czech 🌐Hungarian 🌐Romanian 🌐Greek 🌐Bulgarian 🌐Croatian 🌐Slovak 🌐Slovenian 🌐Estonian 🌐Latvian 🌐Lithuanian 🌐Finnish 🌐Swedish 🌐Norwegian 🌐Danish 🌐Icelandic 🌐Other LanguagesArabic 🌐Hebrew 🇮🇱Turkish 🌐Persian 🌐Hindi 🌐Urdu 🌐Thai 🌐Vietnamese 🌐Indonesian 🌐Indian LanguagesOther European LanguagesUsage Notes