What is robots.txt?
Robots.txt is a text file you place in the root of your website to tell search engines which pages they may and may not crawl. When Googlebot, for example, visits your site, it automatically checks the file at https://ditdomæne.dk/robots.txt before it starts indexing your content. That's because the file works as a set of instructions – not as a technical barrier. The search engine voluntarily chooses to respect the directions based on the shared, so-called Robot Exclusion Protocol.
The file is a de facto standard in search engine optimisation, and it plays a central role in how you control which pages get attention in the search results. In practice, you can use it to stop irrelevant pages – such as internal resources, test environments or duplicate content – from eating up crawl budget. In other words, it's about controlling how search engines interact with your digital infrastructure.
How do you use robots.txt?
When you work with SEO or technical optimisation, you use robots.txt to control access for different bots. You can give general instructions to all crawlers or tailor the rules to specific user-agents. The User-agent directive defines which robot the rule applies to, Disallow tells it which paths it may not access, and Allow can override a block for specific folders or files. You can also add a Sitemap directive that guides search engines directly to your XML sitemap, making indexing more efficient.
Example of a simple structure:
User-agent: *
Disallow: /admin/
Allow: /public/
Sitemap: https://eksempel.dk/sitemap.xmlSearch engines such as Google follow the latest standard (RFC 9309), so it's important that you format the file correctly. If you make mistakes in the structure, the robots can misread your rules and potentially exclude the entire site from indexing. That's why you should always test your file before you upload it.
Why you should use robots.txt
By controlling where search engines crawl, you help both your server and your visibility. When you block irrelevant or technical content, you save crawl resources, so the robots spend their time on the pages that actually matter for searches. This matters for your SEO strategy, because search engines work with a limited crawl budget. If you let bots waste time on unimportant subpages, you risk your most important pages not being updated as often.
In some cases, it's also about security and performance. For example, you can prevent file pages in /backup/ or script folders from being crawled. However, robots.txt doesn't protect against access from malicious actors, as anyone can see and ignore the file. It works purely as a guideline, not as a technical safeguard. That's why you should combine it with other solutions, such as correct use of noindex tags, login protection or access restrictions via your web server.
What types and variants are there?
Robots.txt works with a few, but important, directives. The most common are:
- User-agent: Specifies which bot an instruction applies to, e.g. * for all or Googlebot for Google's crawler.
- Disallow: Prevents access to a path or file.
- Allow: Allows access, even if a higher-level Disallow applies.
- Sitemap: Points to your sitemap, which helps search engines cover your content better.
When you work across subdomains or protocols (http/https), each version requires its own robots.txt file. If you don't have a file, search engines treat it as full permission to access. It's therefore better to have an explicit configuration, even if you allow full access – that way, you avoid misunderstandings.
How do you use robots.txt in practice?
You create the file in a text editor and save it as "robots.txt" in your root directory. Place it so it's accessible directly at the domain root, not in a subfolder. Then upload it via FTP or your CMS's file manager. Many choose to combine their work on robots.txt with structured data and sitemap optimisation, because together they strengthen search engines' overall understanding of your site.
To make sure the file works as expected, you can test it in Google Search Console using the robots.txt tool. Here you can see which lines Google follows and whether any rules conflict. If you work with campaign sites, subdomains or international versions, you can tailor your directives per country or language version. This ensures consistency between your SEO setup, the paid campaigns in Google Ads and your overall digital strategy.
What should you watch out for?
Robots.txt is a simple tool, but small mistakes can cause big problems. A single extra slash or the wrong case can change the meaning completely, because the language in the file is case-sensitive. So make sure you always test your rules in a test environment before you publish them. And remember that changes can take time before the search engines pick them up again.
You should also remember that robots.txt only controls crawling – not indexing. If you want to prevent pages from appearing in search results, you need to use a noindex tag in the HTML or the HTTP header. If you combine the two strategically, you get much better control over how your content appears in searches. In other words, robots.txt is closely linked to SEO, site structure and server administration when you want to build a secure, efficient and well-indexed website.
