What Is robots.txt? A Plain-English Guide for Small-Business Website Owners
Learn what robots.txt does, what it cannot control, why it is not a privacy tool, and what small-business owners should check before changing it.
WebWise Management
9/25/20266 min read
A robots.txt file is one of those website files many business owners only notice when something goes wrong. You may see it mentioned in Google Search Console, in a website audit, inside hosting settings, or in a conversation with a developer. Because the name sounds technical, it is easy to assume it controls whether your website can appear in Google or whether certain pages are private.
That is not quite what it does.
In plain English, robots.txt is a public text file that gives automated crawlers instructions about which parts of a website they may crawl. Search engines such as Google use crawlers to visit pages and resources. A robots.txt file can tell those crawlers not to request certain URL paths, or can allow access where a broader rule would otherwise block it.
For a small-business owner, the important part is not memorising every rule. It is understanding the job of the file, recognising a few obvious warning signs, and knowing when not to make a change without checking the consequences.
Where is the robots.txt file?
The file normally lives at the root of a website. If your website address is https://www.example.co.za, the robots.txt file would normally be available at https://www.example.co.za/robots.txt.
That location matters. A robots.txt file applies to the protocol, host and port where it is published. In practical terms, rules on one host do not automatically control a different subdomain or another version of the site.
You can usually check whether a site has a robots.txt file by adding /robots.txt after the main domain in your browser. If a valid file is there, you will normally see plain text rather than a designed web page.
A website does not necessarily need a custom robots.txt file. Some content-management systems and website platforms create or manage one automatically, and a small site with no special crawl-management needs may not need custom rules at all.
What do the main lines mean?
A robots.txt file is made up of groups of rules. The most common lines a small-business owner may encounter are User-agent, Disallow and Allow.
User-agent identifies the crawler or group of crawlers the following rules apply to. An asterisk (*) is commonly used to refer to crawlers generally. Disallow identifies a path the named crawler should not crawl. Allow can make an exception inside a broader blocked area.
A very simple example could look like this:
User-agent: *
Disallow: /site-search/
In this example, the instruction asks crawlers covered by the rule not to crawl URLs under /site-search/. It does not make that area private, delete it, or guarantee that its URLs can never appear elsewhere.
What robots.txt can control
Its main job is crawler access. It can help prevent compliant crawlers from requesting selected URL paths or sections of a site. On some websites, that can be useful for avoiding unnecessary crawling of duplicate, low-value or machine-generated URL areas.
It can also apply different rules to different crawlers. That is useful in more advanced situations, but most small-business owners do not need to build complicated rule sets themselves.
The key word is crawling. robots.txt is about whether an automated crawler should request a URL. It is not a general switch for visibility, ranking, privacy or security.
What robots.txt cannot do
It cannot reliably keep sensitive information private
A robots.txt file is publicly accessible. Anyone can visit the file and read the paths listed there. The Robots Exclusion Protocol standard is explicit that these rules are not access authorisation, and that robots.txt is not a substitute for real security controls.
If a page or file contains information that should genuinely be private, it needs an appropriate access-control method, such as authentication or another security measure suited to the website. Do not rely on Disallow as a privacy feature.
It does not reliably keep a URL out of Google
Blocking a page in robots.txt stops Google from crawling the blocked content, but Google may still discover the URL through links or other signals and may show the URL in search results without a normal description. That is why robots.txt should not be treated as an indexing-control tool.
There is another important interaction here: if Google is blocked from crawling a page, it cannot fetch that page to see page-level instructions placed on it. This is one reason crawling and indexing controls should not be mixed casually.
If your real problem is that a page or website is not appearing in Google, that is a broader discovery and indexing question. WebWise Management covers that separately in “Why Is My Small Business Website Not Showing on Google Search?”
It does not improve rankings by itself
Adding or editing a robots.txt file is not a shortcut to better search rankings. A sensible file can support crawler management, but it does not replace useful content, clear website structure, sound technical setup or broader SEO work. For that wider context, see “AI Search and SEO: What Small Businesses Should Do Now.”
Why might a small-business owner encounter robots.txt?
You may come across it for several ordinary reasons. A developer may mention it during a website build or redesign. A host or website platform may generate it automatically. Google Search Console may report that a URL is blocked by robots.txt. An audit may flag a rule that appears too broad. A site migration, staging setup or previous troubleshooting change may also leave rules behind.
Seeing robots.txt mentioned does not automatically mean something is wrong. The question is whether the current rules match what the website is meant to allow crawlers to access.
Seven sensible checks before changing robots.txt
1. Open the current file before assuming it needs editing
Visit the root robots.txt address and read what is already there. Save a copy or record the existing rules before making any change. If your platform manages the file through settings rather than direct file access, use the platform's supported method.
2. Check which crawler each rule applies to
Do not read a Disallow line in isolation. Look at the User-agent group above it. A rule aimed at one crawler is not necessarily a rule for every crawler.
3. Look for unusually broad blocks
A line such as Disallow: / under a general User-agent: * group is a strong signal to stop and verify the intention, because it asks the covered crawlers not to crawl the site. Broad directory blocks can also affect more pages than expected.
4. Check the exact path
robots.txt path matching is specific, and path values are case-sensitive in Google's interpretation. A small wording, slash or path change can alter which URLs are affected. Avoid editing rules from memory.
5. Find out whether your CMS, host or developer manages the file
Some website platforms do not expect owners to edit robots.txt directly. Before uploading or replacing a file, check whether the platform generates it or provides a search-visibility setting. Replacing a managed file can create conflicts or be overwritten later.
6. Use an appropriate test before and after a change
Google Search Console includes robots.txt reporting and URL inspection tools that can help show whether Google can process the file and whether a particular URL is blocked from crawling. Use the evidence from those tools rather than guessing from a search result alone.
7. Do not expect an instant crawler response
Crawlers can cache robots.txt. The Robots Exclusion Protocol allows caching and says a cached copy should generally not be used for more than 24 hours unless the file is unreachable. In other words, a correct update may not be reflected by every crawler immediately.
Common mistakes to avoid
The most common owner-level mistakes are usually not complicated coding errors. They are changes made without enough context.
Avoid copying a robots.txt template from another website, blocking folders simply because their names look technical, using robots.txt to protect confidential material, deleting rules you do not understand, or making a broad site-wide block to solve a single page problem.
Also avoid treating every robots.txt warning as an SEO emergency. A blocked URL may be intentionally excluded from crawling. The useful question is whether an important public page or resource is being blocked when it should be accessible.
Does every small-business website need robots.txt?
No. Google states that a small site may not need a robots.txt file at all. If there is no file and the server correctly returns a normal not-found response for /robots.txt, that does not automatically mean the site is broken.
A custom robots.txt file becomes useful when there is a real crawler-management reason for it. That reason should be clear before a business owner starts adding rules.
A practical rule of thumb
Think of robots.txt as a sign for cooperative automated visitors, not a locked door.
It can tell compliant crawlers where they should and should not go. It cannot secure confidential information, guarantee that a URL disappears from search results, or replace broader SEO and indexing decisions.
If you find a robots.txt rule and are not sure why it exists, the safest owner-level approach is to check what it currently does, identify who or what manages it, confirm whether important pages are affected, and make changes only when the intended outcome is clear.
If you want a second set of eyes before changing a technical website setting, WebWise Management can help review your website setup and explain the next sensible step.
Contact
Reach out anytime for personalized support.
Email:
Phone:
+27 72 709 7557
© 2026. All rights reserved.