Docs HelpCrawling behind Cloudflare

Crawling behind Cloudflare

When Cloudflare blocks the Alpheris crawler, the knowledge base can't find any pages. Here's how to briefly let the crawler through, restart the crawl, and restore your protection afterwards.

In short: Cloudflare holds the crawler back via bot protection, WAF rules, "Under Attack" mode or rate limiting. Create an exception for the Alpheris crawler, re-crawl, and keep your normal protection for all other visitors.

How Alpheris crawls

During onboarding and on every re-crawl, Alpheris fetches the pages of your verified domain over HTTPS. The crawler follows your sitemap (e.g. sitemap.xml or wp-sitemap.xml) and reads out the page content along with product/OG metadata. The fetch runs server-side from Alpheris — not from your visitors' browsers. It is precisely these server-side requests that Cloudflare can classify as a "bot" and block.

The crawl requests come with one of these user agents:

User-Agent

# Sitemap, product, and API checks
AlpherisBot/1.0 (+https://alpheris.com)

# Fetching the actual page content (browser-like)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

Important: A rule that only matches the string AlpherisBot often isn't enough — the content fetch uses a normal browser user agent. It's more reliable to briefly relax protection or to allowlist by IP (see below).

Detecting whether Cloudflare is blocking

Typical signs that Cloudflare is the cause:

The crawl returns 0 pages or aborts immediately with an error.
When you open your page manually, a Cloudflare security challenge appears ("Checking your browser…", Turnstile/Captcha page) or an Error 1020 / 1010.
HTTP status 403 or 429 on pages that are actually public.
Cloudflare logs (Security → Events) show blocked requests targeting your sitemap or product pages.

Solution 1 — Temporarily relax bot protection (fastest)

Open the Cloudflare dashboard

Go to dash.cloudflare.com and select the affected domain.

Disable Under Attack mode

Under Security → Settings, make sure Under Attack mode is off. It blocks every non-browser request.

Pause Bot Fight Mode

Under Security → Bots, turn off or relax Bot Fight Mode (or Super Bot Fight Mode) for the duration of the crawl.

Start a re-crawl in Alpheris

In the dashboard, under Data Sources & Knowledge Base, click "Scan website" (see the "Trigger a re-crawl" section).

Turn protection back on

Once the crawl is done, set bot protection and, if applicable, Under Attack mode back to your usual values.

This approach is the simplest, but it briefly relaxes protection for everyone. For a permanent, targeted exception, use Solution 2 or 3 instead.

Solution 2 — Targeted WAF skip rule

Instead of relaxing protection globally, you can create a Custom Rule that lets the Alpheris crawler through and skips all remaining protection measures. Create the rule under Security → WAF → Custom rules → Create rule.

Variant A — Match on the user agent

Reliably matches the API/sitemap requests:

Expression

(http.user_agent contains "AlpherisBot")

Variant B — Match on the crawl IP (most robust)

Since the content fetch uses a browser user agent, a rule on your Alpheris account's egress IP is more reliable. You can get the crawler IP valid for your account from support at alpheris.com; enter it in place of the CRAWLER_IP placeholder:

Expression

(ip.src eq CRAWLER_IP)

Recommended settings for the rule:

Setting	Value
Rule name	`Allow Alpheris Crawler`
Action	`Skip`
Skip	All remaining custom rules, Rate Limiting, Managed rules, Super Bot Fight Mode
Place at	Very top (highest priority)

The user agent can be spoofed — so a UA-only rule is somewhat less secure than an IP rule. Combine both (UA and IP) if you want to leave it open permanently.

Solution 3 — IP allowlist

If you already work with an IP allowlist, add the Alpheris crawler's egress IP there:

Open Security → WAF → Tools → IP Access Rules.
Add the crawler IP provided by support with Action Allow for the affected zone.
Don't use fixed example IPs — only the address valid for your account.

Keep sitemap & robots.txt reachable

Even with relaxed protection, the crawler has to be able to find your structure. So check:

Your sitemap (/sitemap.xml or /wp-sitemap.xml) is publicly accessible without a captcha.
Your robots.txt doesn't lock the crawler out (no blanket Disallow: /).
Cloudflare features like Cache Everything or mobile redirects serve the crawler real HTML content instead of a placeholder page.

Trigger a re-crawl

After every Cloudflare change, you need to kick off the crawl again so the knowledge base re-ingests the pages:

Open the dashboard

Open Data Sources

Go to Data Sources & Knowledge Base.

Scan website

Under Scan website, click the "Scan your-domain.com" button. Alpheris then re-ingests all reachable pages.

Check the status

The crawl runs in the background; the source then shows the number of captured pages. If it stays at 0, protection is still active.

Tip: If you only need individual pages, you can add specific URLs under Individual pages instead of re-crawling the whole domain.

If it still doesn't work

If the crawl still loads no pages despite the exception, check other possible causes on the general help page — or contact support at alpheris.com. Have your domain and the approximate time of the crawl attempt ready so the logs can be matched up.

Note: Cloudflare is not an Alpheris connector — this page only helps with crawling websites that sit behind Cloudflare. The data is stored in Germany; for AI processing see Security & Privacy and the DPA).