Configure robots.txt for retrieval crawlers

Allow the search and retrieval crawlers you want, and treat training crawlers as a separate choice.

Scan your site All fix guides

Full guide coming soon

This placeholder is live so scan results can link here now. The detailed how-to, travel examples and pass/fail patterns will be added next.

What the scanner looks for

Allow the search and retrieval crawlers you want, and treat training crawlers as a separate choice.

What to do meanwhile

Use the Action required note from your scan result, then rescan after you make changes. Related articles below can help until the full guide is ready.

Useful articles while this guide is in progress