Knowledge & Content
Allow crawl traffic through your firewall
June 2026
Quick answers
- Why does Engage need firewall access to my website?
Engage reads public pages you choose to onboard so the assistant can answer visitor questions from your own content. If bot protection returns a challenge page, answers will be incomplete until your team allows legitimate Scale2Rev crawl traffic.
Also asked as: why crawl my site; bot documentation.
- Where do I find allow-list instructions for my site?
Open Engage → Sites → Crawl errors (or the crawl status view for your site). The portal shows the allow-list token and steps your CDN or WAF administrator needs.
Also asked as: crawler user agent; WAF rule.
- Does Engage access private or login-only pages?
No. Only public pages are requested. Pages behind login, paywalls, or your do-not-crawl list are out of scope.
Also asked as: robots.txt; excluded URLs.
- Who do I contact for security review letters?
Use [/contact](/contact) and reference your customer name and site URL. We provide verification details to security teams under NDA when required.
Also asked as: Cloudflare verified bot; security questionnaire.
Related searches: allow Scale2Rev crawler · firewall allow list · WAF bot protection Engage · verified bot review
This page is for site owners and security reviewers, not a public specification of internal systems.
What to allow
Your CDN or WAF should let Scale2Rev crawl traffic reach public HTML for sites you onboarded to Engage. Exact allow-list values are shown in your portal crawl errors view so they stay accurate for your customer account.
What Engage does not do
- Access login-gated or paywalled content
- Ignore your do-not-crawl URL list
- Scrape sites you have not added in Engage
Typical symptoms when blocked
- “Just a moment” or bot challenge HTML in Engage → Content
- Missing team, pricing, or service pages visitors ask about
- Crawl errors citing bot protection or access denied
Fix workflow
- Share portal allow-list guidance with your security owner.
- Apply the rule on staging or production as appropriate.
- Re-crawl from Engage → Sites.
- Validate text previews in Engage → Content.
Enterprise security reviews
Contact /contact for vendor review packets. Detailed technical briefs are shared directly with your security team, not published on the open web.