Identify the intended public content
List the exact URLs intended for public discovery and identify content that must remain private, restricted, or unpublished. Confirm the business owner’s intent before requesting any access change. A readiness exercise should never cause a team to expose customer records, internal documentation, or licensed material. Give a technical reviewer the list of public pages and the symptoms you observed, rather than a blanket request to allow every crawler everywhere.
Check the correct provider controls
OpenAI documents separate crawler roles: OAI-SearchBot supports search, while GPTBot concerns content that may be used for model training. A policy for one is not automatically a policy for the other. Have the authorized technical owner review the current official documentation and relevant site settings. For Google’s AI search features, consult its Search-specific guidance. Provider behavior changes, so record the date of the review and the documentation used. Do not copy an old configuration without understanding its scope.
Separate observations from unresolved checks
Ask the reviewer to report what was tested, how, and with what result for each URL or rule. Suprara fetches the public page’s initial response, robots.txt, and one sitemap, without rendering JavaScript or crawling the full site. The scan PDF records the same captured evidence as the results screen. Preserve unknown and not-applicable statuses with their reasons; neither belongs in the passed-plus-failed scoring denominator. A timeout or scanner challenge does not prove that Googlebot is blocked. A page that works in your browser does not establish indexing. Use a broader technical review when the bounded scan cannot resolve the question.
Close the loop without overpromising
After an approved website change, verify that intended public content remains accurate and restricted material remains protected. Rescan the same URL and scope; retain the before-and-after PDFs, timestamps, rubric versions, coverage, and actual findings. Record unresolved limitations and the responsible owner. Technical availability creates an opportunity for discovery, not an entitlement to a citation or recommendation. A missing optional canonical, sitemap, or JSON-LD declaration is not an automatic defect. If you later observe an AI mention, preserve that response separately. Keep access decisions, basic technical readiness, and external outcomes distinct.