Monitors

Introduction

Monitors let you watch URLs for meaningful changes over time. Each monitor runs on a cron schedule, scrapes the target page, compares the result against the previous snapshot, and notifies you when something important changed.

Key Features: Webpage text diffing, structured price extraction, optional AI judgment, webhook and email notifications, snapshot history, and on-demand checks.

Monitor Types

Monitors are built on top of Scheduled Tasks. Creating a monitor also creates a backing scheduled scrape task (1:1). You manage scheduling through the monitor API; the underlying task is managed automatically.

API Endpoints

POST   /v1/monitors                         # Create a monitor
GET    /v1/monitors                         # List monitors
GET    /v1/monitors/changes                 # List changes across all monitors (feed)
GET    /v1/monitors/:id                     # Get monitor details
PATCH  /v1/monitors/:id                     # Update monitor
DELETE /v1/monitors/:id                     # Delete monitor
POST   /v1/monitors/:id/pause                 # Pause monitoring
POST   /v1/monitors/:id/resume                # Resume monitoring
POST   /v1/monitors/:id/check                 # Trigger on-demand check
GET    /v1/monitors/:id/snapshots             # List snapshots
GET    /v1/monitors/:id/changes               # List detected changes
GET    /v1/monitors/:id/changes/:changeId     # Get change detail

Quick Start

Webpage Change Monitor

Watch a documentation page every hour and get webhook alerts when content changes:

curl -X POST "https://api.anycrawl.dev/v1/monitors" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Docs Changelog Watch",
    "monitor_type": "webpage",
    "cron_expression": "0 * * * *",
    "timezone": "UTC",
    "targets": [
      {
        "url": "https://example.com/changelog",
        "engine": "auto"
      }
    ],
    "diff_options": {
      "only_main_content": true,
      "min_change_ratio": 0.01
    },
    "notify_options": {
      "channels": ["webhook"],
      "only_meaningful": true
    }
  }'

Response

{
  "success": true,
  "data": {
    "monitor_id": "550e8400-e29b-41d4-a716-446655440000",
    "scheduled_task_id": "660e8400-e29b-41d4-a716-446655440001",
    "track_mode": "text",
    "next_execution_at": "2026-07-17T13:00:00.000Z"
  }
}

Price Monitor

Track a product price every 15 minutes with structured extraction:

{
  "name": "Product Price Tracker",
  "monitor_type": "price",
  "cron_expression": "*/15 * * * *",
  "timezone": "America/New_York",
  "targets": [
    {
      "url": "https://shop.example.com/product/12345",
      "engine": "auto"
    }
  ],
  "extract_schema": {
    "type": "object",
    "properties": {
      "price": { "type": "number" },
      "currency": { "type": "string" },
      "in_stock": { "type": "boolean" }
    },
    "required": ["price"]
  },
  "notify_options": {
    "channels": ["webhook"],
    "only_meaningful": true,
    "thresholds": {
      "price_change_pct": 5
    }
  }
}

Request Parameters

Core Configuration

Target Object

Diff Options

Notification Options

How Change Detection Works

Each scheduled run follows this pipeline:

1. Scrape — The backing scheduled task scrapes the target URL.
2. Normalize — Content is normalized (main content extraction, literal text-line exclusion).
3. Compare — Text and JSON are compared independently; mixed mode considers both. Invalid extraction, including on the first check, produces an error.
4. Judge — When goal is set, the judge receives both kinds of diff. Unavailable or incomplete judgment is recorded as unknown and preserves the change.
5. Commit — Snapshot, change, durable notification intents and check completion commit together.
6. Deliver — A recovery worker attempts each recipient/channel with bounded retries and records actual SMTP/HTTP delivery.

Snapshot Status Values

Managing Monitors

List Monitors

curl -X GET "https://api.anycrawl.dev/v1/monitors" \
  -H "Authorization: Bearer <your-api-key>"

Pause and Resume

# Pause
curl -X POST "https://api.anycrawl.dev/v1/monitors/:id/pause" \
  -H "Authorization: Bearer <your-api-key>"

# Resume
curl -X POST "https://api.anycrawl.dev/v1/monitors/:id/resume" \
  -H "Authorization: Bearer <your-api-key>"

Trigger On-Demand Check

curl -X POST "https://api.anycrawl.dev/v1/monitors/:id/check" \
  -H "Authorization: Bearer <your-api-key>"

Returns 202 Accepted when the check is queued. Returns 409 if a check is already in progress.

await client.runMonitor(monitorId);

Update a Monitor

curl -X PATCH "https://api.anycrawl.dev/v1/monitors/:id" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "cron_expression": "0 */2 * * *",
    "goal": "Alert only when pricing or availability changes"
  }'

Delete a Monitor

Deleting a monitor also removes its backing scheduled task, snapshots, and change history.

curl -X DELETE "https://api.anycrawl.dev/v1/monitors/:id" \
  -H "Authorization: Bearer <your-api-key>"

Snapshots and Changes

List Snapshots

curl -X GET "https://api.anycrawl.dev/v1/monitors/:id/snapshots?limit=20&offset=0" \
  -H "Authorization: Bearer <your-api-key>"

const snapshots = await client.getMonitorSnapshots(monitorId, { limit: 20 });

List Changes

curl -X GET "https://api.anycrawl.dev/v1/monitors/:id/changes?limit=20" \
  -H "Authorization: Bearer <your-api-key>"

const changes = await client.getMonitorChanges(monitorId, { limit: 20 });
const detail = await client.getMonitorChange(monitorId, changeId);

Cross-Monitor Change Feed

List detected changes across all of your monitors, newest first — the source for a unified "changes inbox". Rows are lightweight (monitor name/type, url, change type, AI judgment, timestamp) and omit the heavy diff payload; fetch the full diff per change via GET /v1/monitors/:id/changes/:changeId.

# Optional ?change_type= filters (e.g. content, price_up, price_down)
curl -X GET "https://api.anycrawl.dev/v1/monitors/changes?limit=50" \
  -H "Authorization: Bearer <your-api-key>"

Example Change Record

{
  "uuid": "change-uuid",
  "monitor_uuid": "monitor-uuid",
  "url": "https://example.com/changelog",
  "change_type": "text",
  "diff_text": "--- previous\n+++ current\n...",
  "judgment": {
    "meaningful": true,
    "confidence": "high",
    "reason": "New release section added"
  },
  "captured_at": "2026-07-17T12:00:00.000Z"
}

Webhook Events

Subscribe to monitor events via Webhooks:

Monitor webhook payloads include inline diff data so you can act without an extra API call:

{
  "event": "monitor.changed",
  "data": {
    "monitor_id": "550e8400-e29b-41d4-a716-446655440000",
    "monitor_name": "Docs Changelog Watch",
    "monitor_type": "webpage",
    "url": "https://example.com/changelog",
    "change_type": "text",
    "summary": { "total": 1, "same": 0, "changed": 1, "new": 0, "removed": 0, "error": 0 },
    "diff_text": "...",
    "judgment": { "meaningful": true, "confidence": "high", "reason": "..." },
    "captured_at": "2026-07-17T12:00:00.000Z"
  }
}

JavaScript SDK

Install @anycrawl/js-sdk 0.0.6+ for monitor support:

pnpm add @anycrawl/js-sdk

import { AnyCrawlClient } from "@anycrawl/js-sdk";

const client = new AnyCrawlClient(process.env.ANYCRAWL_API_KEY!);

// Create, list, get, update, delete
const created = await client.createMonitor({ /* ... */ });
const monitors = await client.listMonitors();
const monitor = await client.getMonitor(created.monitor_id);
await client.updateMonitor(created.monitor_id, { name: "Updated name" });
await client.deleteMonitor(created.monitor_id);

// Lifecycle
await client.pauseMonitor(created.monitor_id);
await client.resumeMonitor(created.monitor_id);
await client.runMonitor(created.monitor_id);

// History
await client.getMonitorSnapshots(created.monitor_id, { limit: 10 });
await client.getMonitorChanges(created.monitor_id, { limit: 10 });
await client.getMonitorChange(created.monitor_id, changeId);

Best Practices

1. Choose the Right Monitor Type

• Use webpage monitors for editorial content, docs, and legal pages.
• Use price monitors when you need structured field comparison (price, stock, SKU).

2. Reduce Noise

• Set only_main_content: true for webpage monitors.
• Add ignore_selectors for ads, timestamps, or dynamic widgets.
• Use min_change_ratio to ignore tiny edits.
• Set a goal and enable only_meaningful for AI filtering.

3. Set Sensible Schedules

• Match cron frequency to how often the page actually updates.
• Use max_executions_per_day to cap credit usage on expensive pages.

4. Use Webhooks for Automation

• Subscribe to monitor.changed or monitor.price.changed.
• Payloads include diffs inline — no need to poll the changes API.

Limitations

Troubleshooting

Monitor Not Running

• Confirm the monitor is active (is_active: true).
• Confirm it is not paused.
• Verify the cron expression and timezone.
• Ensure the scheduler worker is running (self-hosted).

Too Many Alerts

• Increase min_change_ratio.
• Add ignore_selectors for dynamic sections.
• Set a goal with only_meaningful: true.
• For price monitors, raise thresholds.price_change_pct.

On-Demand Check Returns 503

The scheduler worker must be running to process /check requests. On self-hosted deployments, ensure the scrape worker with scheduler enabled is up.

Related Documentation

• Scheduled Tasks — Underlying cron scheduling
• Webhooks — Event notifications including monitor events
• Scrape API — Scrape options passed through targets
• JSON Mode — Structured extraction used by price monitors

Reliable Checks, History and Delivery

A check has durable pending, ready, processing, completed or failed state. Concurrent processing uses leases; an expired processor cannot publish the result twice. Only one check per monitor is active across scraping and post-processing. skip drops an overlapping cron run; queue delays it. Manual requests also respect the guard. Monitor-backed tasks do not count toward the ordinary scheduled-task quota.

GET /v1/monitors/:id includes in_progress, last_check_state, last_check_at, last_check_error, pause_reason and revision. Use is_active && !is_paused for the effective state. HTTP 409 codes distinguish MONITOR_PAUSED and MONITOR_CHECK_IN_PROGRESS. Resume an auto-paused monitor directly.

PATCH merges option siblings and validates the resulting configuration inside the transaction. goal: null clears the goal and extraction prompt; schema removal is only allowed when the final mode does not require it. monitor_type cannot be changed. Unknown top-level fields return 400. Target options cannot use templates. Changes to effective scraping, tracking mode, goal or ignored text lines create a new revision and baseline. Notification/threshold-only edits retain the baseline.

Snapshot lists are slim. GET /v1/monitors/:id/snapshots/:snapshotId returns a bounded content preview and extracted data, with content_truncated, content_length, content_complete and monitor_revision. Complete comparison text is retained in the database. The default preview limit is 262,144 characters; the comparison limit is 2,000,000 characters. Oversized content is an error. Legacy incomplete snapshots remain readable but are not used as a new baseline.

Snapshot, monitor-change and owner-feed lists return pagination: {has_more, next_cursor} alongside data. Send the returned cursor for the next page; offset remains available. Changes support include_diff_text=false, with full diffs fetched from change detail. IDs break timestamp ties, so new arrivals do not shift cursor pages.

New notified values mean at least one actual SMTP acceptance or Webhook HTTP 2xx. notification_status distinguishes none, pending, queued, delivered, failed and skipped; legacy means the older boolean cannot prove delivery. Check detail uses GET /v1/monitors/:id/checks; per-recipient delivery history uses GET /v1/monitors/:id/notifications. Change detail also returns its notifications. Error checks send email when email is enabled. Missing SMTP configuration and temporary transport failures remain visible and retryable. Delivery is at least once; stable Webhook delivery IDs and email Message-IDs support deduplication.

The recovery worker runs with the scheduler, or separately with --queues=monitor. Defaults: ANYCRAWL_MONITOR_MAX_ATTEMPTS=5, ANYCRAWL_MONITOR_RETRY_DELAY_MS=5000, ANYCRAWL_MONITOR_LEASE_MS=120000, ANYCRAWL_MONITOR_POLL_MS=5000. Set ANYCRAWL_MONITOR_RETENTION_DAYS to enable bounded history cleanup; default 0 retains history. Cleanup protects healthy baselines, retained references, pending deliveries and legacy records. Upgrading never replays old alerts automatically.

capabilities.location=false means country locking is unavailable. ignore_selectors matches literal substrings of normalized text lines, not CSS. Price history should be grouped by URL, field path and known currency; loaded change points are not the complete price history.

const feed = await client.listMonitorChanges({ limit: 20 });
const page = await client.getMonitorSnapshotsPage(monitorId, { limit: 20 });
if (page.data[0]) {
  const detail = await client.getMonitorSnapshot(monitorId, page.data[0].uuid);
}
const checks = await client.getMonitorChecks(monitorId);
const deliveries = await client.getMonitorNotifications(monitorId);