How the Crawl Works
During onboarding and afterwards, Alpheris reads your domain in as a knowledge base: the sitemap (including wp-sitemap.xml), product and Open Graph metadata, as well as platform-specific patterns. The knowledge base that the chatbot uses to give grounded answers is built from these publicly reachable pages. If the crawl fails, it is almost always down to one of four causes: access blocked, no indexable content, wrong domain, or rate limit.
Quick Check (2 Minutes)
yourshop.com) – not a staging URL, a preview address, or a password-protected preview.https://your-domain/robots.txt and make sure that not everything is blocked via Disallow: /.Common Errors & Fixes
| Symptom / Error | Cause & Fix |
|---|---|
| 403 / challenge page | Cloudflare or a WAF is blocking the bot. Create an exception for our crawler – details under Crawl Behind Cloudflare. |
| 429 (rate limit) | The server is throttling requests. Loosen bot/IP rate limits or set an exception, then crawl again. |
| robots.txt blocking | A global Disallow: / blocks the crawl. Allow the relevant paths (products, FAQ, policies) and start a re-crawl. |
| 0 pages / empty knowledge base | No sitemap, a pure JavaScript page without server-side HTML, or only images without text. Add a sitemap or upload content manually. |
| Only login pages captured | Content is behind a login. Prioritize public pages or upload key content as a file. |
| Timeout / domain unreachable | DNS problem, maintenance mode, or wrong domain. Check in a private window whether the site loads without logging in. |
| Wrong / too few pages | Staging domain or missing internal linking. Enter the final domain and provide a complete sitemap. |
Pages Are Being Blocked (403, Challenge, Rate Limit)
If the site loads in the browser but the crawl fails, in most cases a protection mechanism is blocking automated access. Typical sources are Cloudflare, bot management, or aggressive rate limits on the web server.
- Cloudflare / bot protection: Create an allow rule or a "skip" rule for our crawler. The step-by-step guide is under Crawl Behind Cloudflare.
- Rate limit (429): Increase the limit for bot access or exclude the crawler from the rate limit. Then trigger the crawl again.
- Geo/IP block: Make sure your hosting does not block access from the EU. Alpheris processes and crawls from the EU (hosting at Hetzner, data centers in Nuremberg/Falkenstein).
- Maintenance mode / "coming soon": Such plugins serve all visitors only a placeholder page – the crawler included. Disable it first, then crawl.
Pages Are Empty or Have No Text
Sometimes access is allowed, but the crawl finds little or no usable text. Common reasons:
- Pure client-side rendering: If the content is only loaded afterwards via JavaScript and is not in the served HTML, the crawler is missing the text. Server-side rendering (SSR) or a sitemap with real content pages helps.
- No sitemap: Without a
sitemap.xmlorwp-sitemap.xml, the crawler only finds what is internally linked. Create a sitemap or link your most important pages from the homepage. - Content only as an image or PDF: Text in graphics or embedded documents is not read as text on the website. Upload such documents directly as a file – see below.
- Very thin pages: A pure landing page without FAQ, product, or policy text results in a thin knowledge base. Add the content or upload it as a file.
You can check this in the browser via "View page source": if there is barely any text there, the crawler also sees barely any text.
Restart the Crawl
After you have fixed the cause, you trigger the crawl again in the dashboard:
Alternative: Upload Content Manually
If the crawl simply cannot be unblocked – for example because the site is behind a login or is still being built – you can also fill the knowledge base entirely without a crawl using file uploads. This works on every plan and is its own source type (upload) alongside the website crawl.
In the dashboard under Data Sources, upload e.g. your FAQ, shipping and returns policies, product data sheets, or price lists. Supported file types:
| Category | Allowed Formats |
|---|---|
| Documents | .pdf .txt .md .rtf .log .doc .docx .odt |
| Tables / Data | .csv .tsv .json .xml .xls .xlsx .ods |
| Presentations | .ppt .pptx |
| Images | .png .jpg .jpeg .webp .gif .bmp .tif .tiff .heic .heif |
.html and .svg are rejected as uploads. Executable files (e.g. .exe, .msi, .sh, .jar, .bat) are also blocked. Every upload is checked with a virus scanner (ClamAV) before processing..html cannot be uploaded directly, copy the visible text of the page into a .txt or .md file, or print the page as a .pdf and upload that.Crawl sources and uploads can be combined freely: you can upload first and, later, once access is open, additionally crawl.
Shopify & WordPress Shops
With a connection via Shopify or WordPress & WooCommerce, the domain is verified automatically and – with Shopify – product and order data are additionally synced. If the plain website crawl finds little there, the platform connection still delivers the most important product content. In that case, check whether the integration is connected before repeating the crawl multiple times.
Still Stuck?
If the crawl still fails after all points have been addressed, send us the affected domain and – if available – the displayed error message. General diagnostic steps can be found under Troubleshooting; if you are protected by Cloudflare, the Cloudflare guide is the fastest way forward.