// AI Platform Drift

Browser Run /crawl adds outcome field and robots array

Cloudflare developer changelog · published 2026-11-11 05:19 UTC
WORTH KNOWING VERIFY_ONLY API
WHAT CHANGEDCloudflare Browser Run /crawl endpoint response now includes an outcome field for each URL record (values: success, robots_disallowed, timeout, page_error) and a job-level robots array reporting robots.txt state for up to 50 origins with origin, state, and HTTP status fields.
WHY IT MATTERSExisting parsers that strictly validate response schemas will reject the new fields. Applications that rely on status alone to determine crawl success cannot distinguish between robots.txt blocks, timeouts, and page errors without checking outcome.
WHAT TO DOUpdate response parsing to handle the outcome field and robots array. Test against the new schema with a sample crawl to confirm your client tolerates the additional fields.
Model-reported confidence 85% (not measured accuracy) Primary source ↗ written by anthropic/claude-sonnet-4.5
Show the reasoning — raw diff, triage verdict, confidence
RAW DIFF
+## Browser Run - Crawl results include URL outcomes and robots.txt diagnostics
+https://developers.cloudflare.com/changelog/post/2026-11-03-crawl-result-diagnostics/
+Tue, 03 Nov 2026 00:00:00 GMT
+
+Browser Run /crawl results now include more detailed diagnostics. Each URL record can include an outcome that identifies the specific processing result, such as success, robots_disallowed, timeout, or page_error.
+The job-level robots array reports the robots.txt state for up to 50 origins encountered by the crawl. Each entry includes the origin, whether usable rules were available, and the HTTP status when the crawler received a response.
+{
+"robots": [
+{
+"origin": "https://example.com",
+"state": "available",
+"status": 200
+}
+],
+"records": [
+{
+"url": "https://example.com/docs/",
+"status": "completed",
+"outcome": "success"
+}
+]
+}
+For field definitions and all possible outcome values, refer to URL outcomes and robots.txt diagnostics.
HOW IT WAS JUDGED
Triage signals below are model-reported assertions, preserved verbatim; they are not independent arithmetic verification.
triage verdictMATERIAL at 0.92, needed 0.5
triage reasoningBrowser Run /crawl endpoint now returns additional diagnostic fields and outcome enums. Clients must update parsing logic.
signalsnew API response fields added (outcome, robots array) · new enum values documented (success, robots_disallowed, timeout, page_error) · breaking change to /crawl endpoint schema · future date (2026-11-03) indicates feature release · integration impact: clients must handle new fields and outcomes
suppression—
triage modelanthropic/claude-haiku-4.5 ($0.00000)
writer modelanthropic/claude-sonnet-4.5 ($0.00000)