Knowledge & Content
Content not loading into Engage
June 2026
Quick answers
- Why are my website pages missing from Engage Indexed pages?
Firewalls and bot protection often return a challenge page instead of your real HTML. Engage then cannot index that content. Allow Scale2Rev crawl traffic per the instructions in your portal crawl errors view, then run a new crawl.
Also asked as: my content isn't loading; pages not in knowledge base.
- Cloudflare is blocking Engage from reading my site, what do I do?
Create a firewall or WAF rule that bypasses bot challenges for Scale2Rev crawl traffic. Your Engage portal crawl errors screen lists the exact allow-list token your security team should use.
Also asked as: Cloudflare bot fight mode; Just a moment page.
- How do I re-index my site after fixing the firewall?
Open Manage → Websites, select the site, and start a new crawl. When complete, check Manage → Indexed pages for full text previews on the URLs you care about.
Also asked as: re-crawl website; refresh knowledge base.
- Can I exclude pages from Engage?
Yes. Use do-not-crawl URL patterns in site crawl settings and turn off individual pages in Manage → Indexed pages when you do not want them used in answers.
Also asked as: hide page from chatbot; disable URL.
Related searches: pages not crawled · knowledge base empty · content not indexed · Cloudflare blocking crawl · team page missing from Engage
Engage learns from public pages on your site. If firewall rules block our crawler, indexed pages may show challenge pages or miss URLs entirely.
Check the portal first
- Open Manage → Indexed pages, are important URLs missing or showing thin “challenge” text?
- Open the site Crawl errors view for categorized failures.
- Fix access rules, then re-crawl the site.
Work with your security team
Share the allow-list guidance shown in your portal crawl errors panel with whoever manages Cloudflare, Akamai, or similar services. Do not guess allow-list values from third-party copies — use the values displayed for your customer account.
Optional path allow-listing
Some teams also allow /sitemap.xml and high-value paths (for example /team, /pricing) while tuning rules.
After access is fixed
Run a new crawl and confirm each priority URL shows meaningful text in Manage → Indexed pages.