1. Decide what should stay crawlable
List the pages customers should find: your services, products, useful guides, and contact information. Then list areas that serve a different purpose, such as internal search or repeated filter combinations. Do not block those areas automatically. First establish whether crawling them is actually causing a problem and whether their content has search value. For a small business website, keeping a short, understandable policy is usually easier to maintain than copying a long file from an unrelated site.
2. Inspect the file that is actually published
Open /robots.txt on the same host and protocol as the page you are investigating. Rules published on a shop subdomain do not automatically govern the main website. Read the user-agent groups before interpreting individual paths. A Disallow rule is a crawling instruction, while a Sitemap line points to a discovery file. A missing robots.txt returning 404 is not automatically an SEO failure; Google treats that response as no crawl restrictions. A server error is a different finding and deserves separate investigation.
3. Review risky patterns with real examples
A rule covering the root path can block far more than a rule covering one directory. Before changing a pattern, prepare a few example URLs that should be allowed and a few that should be blocked. Include capitalization, trailing slashes, and parameters used by your site. Ask the person maintaining the website to test these cases against the intended crawler's rules. Keep a brief reason beside each decision in your change record, so the next person understands why the restriction exists.
4. Keep privacy and indexing controls separate
Robots.txt is public guidance, not access control. Confidential files need authentication or another genuine security boundary. A blocked page URL can still appear in Google when other pages link to it. If the goal is exclusion from search, consider noindex while allowing the crawler to fetch the page and see that instruction. Avoid blocking essential scripts or styles without understanding how the page depends on them. A rule that looks tidy in a text file can interfere with understanding the actual page.
5. Verify the change and preserve evidence
Save the previous version before editing. After publication, open the public file again and check representative pages in Search Console. Record the test date and exact URLs rather than writing only “robots fixed.” Suprara reviews public HTML alongside robots.txt and sitemap signals, helping you organize the next steps in a personalized PDF. Use its action plan and a rescan to compare the public signals after a change. That comparison does not establish what Google has crawled or indexed; verify those outcomes separately.
Your practical checklist
- Identify the exact host and intended crawler.
- Test important allowed and blocked example URLs.
- Check that private content has real access protection.
- Keep pages crawlable when Google must read noindex.
- Save the old file, publish carefully, and verify again.
Frequently asked questions
Apply what you learned
Practice this topic in Academy, then use the worksheet to document the evidence and the change you plan to make.
Sources and further reading
A clearer website starts with evidence.
Get a real public-page scan and a personalized PDF, then use the free worksheet to prioritize your next changes.