Templates

Introduction

Templates are reusable configurations for scraping, crawling, or searching. Instead of repeating the same options in every API call, you define them once in a template (or obtain a template from AnyCrawl Template Store) and reference it by template_id.

Benefits:

• Simplicity: Call APIs with just template_id + minimal inputs
• Consistency: Standardize behavior across your team or projects
• Safety: Templates can restrict allowed domains and expose only necessary variables
• Power: Optional custom handlers for advanced transformations

Supported types:

• scrape: single-page extraction via /v1/scrape
• crawl: multi-page crawling via /v1/crawl
• search: search engine results via /v1/search

Template Marketplace

Browse ready-to-use templates at anycrawl.dev/template.

How to use:

1. Browse the marketplace and find a template that fits your needs
2. Copy the template_id from the template detail page
3. Call the API with that template_id and required inputs

Example:

curl -X POST "https://api.anycrawl.dev/v1/scrape" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "content-extractor",
    "url": "https://example.com"
  }'

Dedicated Template Endpoints

Every template also gets its own URL, so you can call it directly instead of passing template_id inside a /v1/scrape, /v1/search, or /v1/crawl body. This is handy for sharing a template as a branded short link, wiring it into no-code tools, or letting a client discover a template's inputs before calling it.

POST  https://api.anycrawl.dev/v1/template/{ref}/execute   # run the template
GET   https://api.anycrawl.dev/v1/template/{ref}            # call-spec (discovery)

Addressing a template ({ref})

{ref} is either:

• the template's vanity slug (preferred — e.g. content-extractor), or
• the templateId (e.g. my-user~my-template).

Both resolve to the same template. The slug produces the nicer, brandable link.

Execute a template

POST /v1/template/{ref}/execute runs the template. The body accepts only url, query, and variables — every other option comes from the template. Providing any other field returns a 400 validation error.

The response shape depends on the template's type:

Scrape / Search templates (synchronous)

# Scrape template — pass `url`
curl -X POST "https://api.anycrawl.dev/v1/template/content-extractor/execute" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "variables": { "category": "tech" }
  }'

# Search template — pass `query`
curl -X POST "https://api.anycrawl.dev/v1/template/news-search/execute" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "machine learning",
    "variables": { "lang": "en" }
  }'

The synchronous response is the same envelope as the underlying endpoint:

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "status": "completed",
    "markdown": "# Page Title\n\nContent...",
    "metadata": { }
  }
}

Crawl templates (asynchronous)

A crawl template returns a job_id immediately. Poll /v1/crawl/{job_id}/status for progress and fetch pages from /v1/crawl/{job_id} — exactly as with a normal crawl.

# 1. Start the crawl — returns a job_id
curl -X POST "https://api.anycrawl.dev/v1/template/docs-crawler/execute" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "variables": { "maxPages": 50 }
  }'

# 2. Poll status
curl -H "Authorization: Bearer <YOUR_API_KEY>" \
  "https://api.anycrawl.dev/v1/crawl/<JOB_ID>/status"

# 3. Fetch results (paginated)
curl -H "Authorization: Bearer <YOUR_API_KEY>" \
  "https://api.anycrawl.dev/v1/crawl/<JOB_ID>?skip=0"

The start response:

{
  "success": true,
  "data": {
    "job_id": "7a2e165d-8f81-4be6-9ef7-23222330a396",
    "status": "created",
    "message": "Crawl job created successfully"
  }
}

Discover a template (call-spec)

GET /v1/template/{ref} returns a redacted call-spec describing the template's inputs, pricing, and endpoint — without ever exposing internal handlers or request options. Use it to self-describe a template in a UI or to validate inputs before calling.

curl -H "Authorization: Bearer <YOUR_API_KEY>" \
  "https://api.anycrawl.dev/v1/template/content-extractor"

Example response:

{
  "success": true,
  "data": {
    "template_id": "my-user~content-extractor",
    "slug": "content-extractor",
    "name": "Content Extractor",
    "description": "Extract clean article content from any page.",
    "template_type": "scrape",
    "version": 3,
    "endpoint": {
      "method": "POST",
      "path": "/v1/template/content-extractor/execute"
    },
    "variables": {
      "category": {
        "type": "string",
        "required": false,
        "defaultValue": "general",
        "description": "Content category tag"
      }
    },
    "pricing": {
      "perCall": 10,
      "currency": "credits"
    },
    "allowed_domains": {
      "type": "glob",
      "patterns": ["*.example.com"]
    }
  }
}

Pricing

• execute is billed the same as any template call — the template's pricing.perCall credits (plus any per-URL usage for crawl). Billing is identical to a body template_id call.
• GET /v1/template/{ref} (call-spec) is free — discovery is never billed.

Errors

Example error response:

{
  "success": false,
  "error": "Not found",
  "message": "Template not found: content-extractor",
  "data": { "type": "TEMPLATE_NOT_FOUND" }
}

Using Templates in API Calls

Request Parameters

When using template_id, only minimal fields are allowed:

Important notes:

• url or query may be optional if the template predefines them. Check the template description.
• variables passes dynamic inputs the template expects (see Variables section below).
• Other fields (like engine, formats, timeout, etc.) come from the template and cannot be overridden.
• Providing disallowed fields returns a 400 validation error.

Scrape with a Template

curl -X POST "https://api.anycrawl.dev/v1/scrape" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "my-scrape-template",
    "url": "https://example.com",
    "variables": { "category": "tech" }
  }'

Crawl with a Template

curl -X POST "https://api.anycrawl.dev/v1/crawl" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "my-crawl-template",
    "url": "https://docs.example.com",
    "variables": { "maxPages": 50 }
  }'

Search with a Template

curl -X POST "https://api.anycrawl.dev/v1/search" \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "my-search-template",
    "query": "machine learning tutorials",
    "variables": { "lang": "en" }
  }'

Variables

Templates can declare variables to accept dynamic inputs at call time.

• Each variable has a type: string, number, boolean, or url
• Variables can be required or optional with defaultValue
• Check the template description to see what variables it expects

Example request with variables:

{
    "template_id": "blog-scraper",
    "url": "https://example.com/blog/post-123",
    "variables": {
        "author": "john-doe",
        "includeComments": true,
        "maxComments": 50
    }
}

If you omit a required variable or provide the wrong type, you'll get a 400 validation error.

Response Format

Template responses follow the same format as standard API calls:

{
    "success": true,
    "data": {
        "url": "https://example.com",
        "markdown": "# Page Title\n\nContent...",
        "metadata": { ... },
        // Additional fields from custom handlers (if any)
        "extractedData": { ... }
    }
}

Templates with custom handlers may add extra fields to the response.

Error Handling

Common errors when using templates:

Example error response:

{
    "success": false,
    "error": "Validation error",
    "message": "When using template_id, only template_id, url, variables are allowed. Invalid fields: engine, formats",
    "data": {
        "type": "validation_error",
        "issues": [
            {
                "field": "engine",
                "message": "Field 'engine' is not allowed when using template_id",
                "code": "invalid_field"
            }
        ],
        "status": "failed"
    }
}

Best Practices

For API Callers

• Always check the template description for required variables and allowed domains/keywords
• Use marketplace templates when available to save time
• Handle 404 errors (template may have been deleted or archived)
• Don't try to override template settings (engine, formats, etc.) - it will fail

For Template Authors

• Keep templates focused on a single use case
• Document all variables clearly with descriptions
• Use domain restrictions to prevent misuse
• Set appropriate pricing based on complexity
• Test templates thoroughly before publishing

Creating Templates (Advanced)

If you're creating your own templates, you can configure:

Domain Restrictions

Limit where your template can be used:

{
    "allowedDomains": {
        "type": "glob",
        "patterns": ["*.example.com", "docs.mysite.com"]
    }
}

• type: "exact" (exact match) or "glob" (pattern matching)
• patterns: array of allowed domains or patterns

Keyword Restrictions (Search Templates)

Limit which queries can be used with a search template:

{
    "allowedKeywords": {
        "type": "glob",
        "patterns": ["*tutorial*", "*documentation*"]
    }
}

• allowedKeywords is validated before query execution.
• type: "exact" or "glob"
• patterns: allowed keyword values/patterns
• Query validation happens before queryTransform is applied.

Pricing

Set credit cost per call:

{
    "pricing": {
        "perCall": 10,
        "currency": "credits"
    }
}

Custom Handlers

Write JavaScript/TypeScript code to:

• requestHandler: Post-process scrape results and add custom fields
• failedRequestHandler: Handle failures with custom retry logic
• queryTransform (search only): Transform queries before searching
• urlTransform (scrape/crawl only): Transform URLs before processing

Both transforms support:

• Template mode with placeholders (query: {{query}}, url: {{url}})
• Append mode with prefix and suffix
• Optional regexExtract to pre-extract a substring before applying the mode

Example regex extraction for TikTok profiles:

{
    "customHandlers": {
        "urlTransform": {
            "enabled": true,
            "mode": "append",
            "prefix": "",
            "suffix": "",
            "regexExtract": {
                "pattern": "^(https?:\\/\\/www\\.tiktok\\.com\\/@[^\\/?#]+)",
                "flags": "i",
                "group": 1
            }
        }
    }
}

This extracts https://www.tiktok.com/@piperrockelle from inputs like:

• https://www.tiktok.com/@piperrockelle?abb=ccc
• https://www.tiktok.com/@piperrockelle

Example requestHandler:

// Extract structured data from page context
const title = context.data.title;
const content = context.data.markdown;

return {
    extractedTitle: title,
    wordCount: content.split(/\s+/).length,
    customMetric: calculateMetric(content),
};

Security Model

• Non-trusted templates: Run in a hardened VM sandbox with strict limitations
• Trusted templates: Can use async functions with controlled browser page access

Only templates reviewed and approved by AnyCrawl can be marked as trusted.

FAQ

Can I override template settings like engine or formats?

No. Templates are designed to be immutable configurations. You can only provide url/query and variables.

What happens if I use a template from the marketplace?

Marketplace templates are publicly available. You pay the credits defined by the template author.

Can templates see my API key?

No. Templates run in isolated sandboxes and have no access to your credentials.

How do I create my own templates?

Visit the AnyCrawl playground to create and test templates. Once published, they can be used via API.

See Also

• Scrape API Documentation
• Crawl API Documentation
• Search API Documentation
• Template Marketplace