Map the journey
A page can exist without appearing for a search. Google first discovers addresses, then crawls accessible resources, processes their content for indexing, and selects relevant results for a query. These are different stages, and inclusion is not guaranteed. Treat them as a diagnostic sequence rather than a promise that publishing produces traffic.
Imagine a training studio with a new public workshop page. Before rewriting the headline, record its exact address, intended audience, publication date, and where visitors can find a link to it. Ask a colleague to navigate from the homepage without receiving the direct address. If they cannot find it, you have identified a useful usability investigation. Build a small inventory with columns for page purpose, owner, intended search visibility, last change, and next check. Include pages you intentionally keep out of search so that successful exclusion is not mistaken for a failure.
Choose the right control
Robots.txt controls crawler access; it is not a reliable way to remove a known address from search. A noindex instruction must be accessible to Google so it can be read. Google does not support putting noindex in robots.txt. A sitemap helps communicate preferred URLs, but submission does not guarantee indexing. Do not use search controls as protection for private information; use appropriate access controls.
For our workshop scenario, write the intended outcome before proposing a configuration change. A public registration page and an internal attendee list have different requirements. Ask the responsible developer to explain the actual response, crawler rules, and indexing instructions for each. Record observed evidence separately from your hypothesis. If you lack access to diagnostic tools, say which checks are unverified instead of marking them as passed.
Make a testable repair plan
Start with one representative page, not a sitewide setting change. Keep a dated record of what you observed, what the page should do, and who may approve changes. A useful ticket names the affected address, describes the intended visitor journey, includes reproducible evidence, and defines a verification step. Avoid vague tickets such as improve SEO.
After an authorized repair, repeat the same checks and record the date. Do not call a change successful merely because the editing screen saved it. Distinguish a corrected configuration from a later search outcome. If the page is accessible and intended for indexing but remains absent, continue investigating content and indexing evidence rather than repeatedly resubmitting the same request. This habit turns SEO from guesswork into a sequence of observable decisions.
Apply it in your own notes
Create an audit record for a real public page or a clearly labeled practice page. Include its purpose, exact URL, discovery path, intended indexing state, three observed checks, and one uncertainty. Write a repair ticket with an owner and a repeatable verification step. Self-check: have you separated observed facts from assumptions, and protected private pages with access controls rather than search settings?
Get a worksheet →Practice questions
1. Which sequence best describes the search journey?
2. Google must read a noindex instruction. Which setup obstructs that?
3. What is the best completion evidence for a configuration repair?
Primary sources
Google Search Central: How Search works
Google Search Central: Block indexing with noindex