What is robots.txt?
Robots.txt is a text file that you place in the root directory of your website to tell search engines which pages they may and may not crawl. For example, when Googlebot visits your site, it automatically checks the file at https://ditdomæne.dk/robots.txt before it begins indexing your content. This happens because the file functions as a set of instructions—not as a technical block. The search engine voluntarily chooses to respect the instructions based on the common “Robot Exclusion Protocol.”
This file is a de facto standard in search engine optimization, and it plays a central role in determining which pages receive attention in search results. In practice, you can use it to prevent irrelevant pages—such as internal resources, test environments, or duplicate content—from consuming your crawl budget. In other words, it’s about managing how search engines interact with your digital infrastructure.
How do you use robots.txt?
When working with SEO or technical optimization, you use robots.txt to control access for different bots. You can provide general instructions for all crawlers or customize the rules for specific user-agents. The User-agent directive defines which bot the rule applies to; Disallow specifies which paths it is not allowed to access; and Allow can override a block for specific folders or files. You can also add a Sitemap directive that guides search engines directly to your XML sitemap, making indexing more efficient.
Example of a simple structure:
User-agent: *
Disallow: /admin/
Allow: /public/
Sitemap: https://eksempel.dk/sitemap.xmlSearch engines like Google follow the latest standard (RFC 9309), so it's important that you format the file correctly. If you make mistakes in the structure, the crawlers may misinterpret your rules and potentially exclude your entire site from indexing. That's why you should always test your file before uploading it.
That's why you should use robots.txt
By controlling where search engines crawl, you help both your server and your visibility. When you block irrelevant or technical content, you save crawl resources so that the bots spend their time on the pages that actually have value for searches. This matters for your SEO strategy because search engines operate with a limited crawl budget. If you let bots waste time on irrelevant subpages, you risk your most important pages not being updated as often.
In some cases, it’s also a matter of security and performance. For example, you can prevent file pages in /backup/ or script folders from being crawled. However, robots.txt does not protect against access by malicious actors, since anyone can view and ignore the file. It serves solely as a guideline, not as a technical safeguard. Therefore, you should combine it with other solutions, such as the proper use of noindex tags, login protection, or access restrictions via your web server.
What types and varieties are available?
Robots.txt uses a few key directives. The most common ones are:
- User-agent: Specifies which bot a directive applies to, e.g., * for all bots or Googlebot for Google's crawler.
- Disallow: Prevents access to a path or file.
- Allow: Allows access even if an overarching Disallow directive is in effect.
- Sitemap: Links to your sitemap, which helps search engines better index your content.
When working across subdomains or protocols (http/https), each version requires its own robots.txt file. If you don’t have a file, search engines will interpret this as full access permission. It is therefore better to have an explicit configuration, even if you grant unrestricted access—this way, you avoid misunderstandings.
How do you use robots.txt in practice?
Create the file in a text editor and save it as “robots.txt” in your root directory. Place it so that it’s accessible directly at the domain root, not in a subfolder. Then upload it via FTP or your CMS’s file manager. Many people choose to combine working with robots.txt with structured data and sitemap optimization, because together these elements improve search engines’ overall understanding of your site.
To ensure that the file works as expected, you can test it in Google Search Console using the robots.txt tool. Here, you can see which lines Google follows and whether any rules conflict. If you work with campaign sites, subdomains, or international versions, you can customize your directives by country or language version. This ensures consistency between your SEO setup, paid campaigns in Google Ads, and your overall digital strategy.
What should you keep in mind?
Robots.txt is a simple tool, but even small mistakes can cause major problems. A single extra slash or incorrect capitalization can completely change the meaning, because the language in the file is case-sensitive. Therefore, make sure you always test your rules in a test environment before publishing them. And remember that it may take some time for search engines to detect the changes.
You should also keep in mind that robots.txt only controls crawling—not indexing. If you want to prevent pages from appearing in search results, you need to use a noindex tag in the HTML or the HTTP header. If you combine the two strategically, you’ll have much better control over how your content appears in search results. In other words, robots.txt is closely tied to SEO, website structure, and server administration when you’re building a secure, efficient, and well-indexed website.
