Applebot-Extended robots.txt – The 2025 Playbook

TL;DR

Add explicit robots.txt rules for Applebot (search & Siri) and Applebot-Extended (AI training). Allowing Applebot-Extended opens your catalog to Apple Intelligence shopping cards and Spotlight answers; disallowing it keeps rankings intact but forfeits AI exposure. Use the snippet below, test with cURL, and enrich pages with Product/FAQ schema.

Why Applebot-Extended Matters to Revenue

Apple shipped Apple Intelligence system-wide in iOS 18, iPadOS 18, macOS 15, and visionOS 2.4. Three months later it reported 180 million monthly active users across Siri, Spotlight, and Messages. Those answers quote—and link back to—sites fetched by Applebot. Applebot-Extended is the new flag controlling whether your content trains Apple's models and appears in generative snippets.

2025 Adoption & Traffic Stats

Meet Apple's Crawlers

Applebot vs Applebot-Extended

Key: Applebot-Extended obeys only Allow/Disallow. To throttle bursts use HTTP 429, not crawl-delay.

How to Spot Them in Logs

grep -E "Applebot|Applebot-Extended" access.log | awk '{print $1,$12}' | head

Robots.txt Configuration

Quick-Start Allow / Disallow Blocks

# — Apple crawlers —
User-agent: Applebot
Allow: /

# Allow training + AI previews
User-agent: Applebot-Extended
Allow: /

# To keep search but block AI training:
# User-agent: Applebot-Extended
# Disallow: /

Place above any wildcard sections so other groups don't override them.

Burst Protection & Rate Limiting

Applebot-Extended ignores crawl-delay. Throttle with HTTP 429 + Retry-After headers when total requests exceed 12 req/s. Check Applebot IP ranges via the JSON feed in Apple Search Console.

Troubleshooting Flowchart

1. Add rules to robots.txt
2. Test: curl -A "Applebot-Extended" https://yoursite.com/robots.txt — expect 200
3. Observe logs for hits within 48 hours
4. Bandwidth spike? Enable 429 gating

Schema & Content Optimisation

Product / Article JSON-LD Essentials

Apple Intelligence lifts price, rating, and eco attributes directly from Product schema into shopping answers:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ocean-Cycle Trail Shoes",
  "sku": "OC-TS-001",
  "offers": {
    "@type": "Offer",
    "price": "98.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.8",
    "reviewCount": "267"
  }
}

Add Article plus FAQPage schemas. Apple shows inline FAQ answers in Spotlight previews.

Cross-Industry Mini Cases

Risk, Compliance & Core-Update Alignment

Bandwidth — Use 429 gating; crawl-delay is unreliable.

IP Ranges — Apple lists CIDR blocks in Search Console JSON feed.

Licensing — Applebot-Extended honours opt-out, but some publishers negotiate licensing deals. State your policy in /terms.

EEAT — Include author bios, peer citations, and original media; Apple's June 2025 “Trust Layers” update values authenticity.

Implementation Checklist

1. Backup robots.txt
2. Insert allow/disallow block
3. Test with curl -A "Applebot-Extended"
4. Monitor logs for Applebot-Extended hits
5. Add GA4 filter for utm_source=appleintelligence.apple.com
6. Audit schema coverage
7. Review server load after 14 days
8. Update SOP
9. Schedule quarterly audit
10. Book an expert SEO Audit to benchmark AI readiness

FAQs

What is Applebot-Extended?

Applebot-Extended is Apple's crawler for AI model training and generative answer cards. It's separate from Applebot, which powers classic Siri/Spotlight search.

Does allowing Applebot-Extended affect SEO?

No. Rankings rely on Applebot. Applebot-Extended only governs AI training and answer inclusion.

How do I block Applebot-Extended?

Add User-agent: Applebot-Extended plus Disallow: / to robots.txt. Applebot search remains unaffected.

Does Applebot honour crawl-delay?

Partially—anecdotal evidence suggests mixed compliance. Use 429 rate limits for certainty.

Where can I see Applebot traffic?

Filter server logs for “Applebot” strings or monitor Apple Search Console crawl stats.

Is Your Site Ready for AI Search?

Configuring robots.txt is just one piece of the puzzle. Our AI Search Optimization service ensures your site is structured, cited, and visible across Google AI Overviews, ChatGPT, Perplexity, and Gemini.

Get a Free AI Search Audit

Next Steps

Ready to capture Apple Intelligence visibility? Start with a holistic SEO Audit—our technicians benchmark crawl health, schema depth, and Applebot eligibility in two weeks. Need content that earns citations? Explore our data-driven SEO programs that turn insight into demand.