What is Screaming Frog?
Screaming Frog is a program that finds errors and reveals technical problems on your website. It works as a spider and goes through your site in much the same way as Google crawls it. In other words, you see your website through the search engine's eyes. Then you can fix the problems and move the site in the right direction.
You can download the program for PC (Windows), Mac and Linux. It comes in a free and paid version. The free version crawls a maximum of 500 URLs, and a number of technical features are missing. For example, you can't see which pages have structured data, and you can't pull in data from Google Analytics.
Not familiar with structured data? We've written an introduction to the topic right here.
Back to Screaming Frog. The premium version costs £149 a year, approx. DKK 1,235, and gives access to all features. Feel free to test the free version for a while, and then decide whether you need a licence. Just remember that the free version only gives access to some of the features.
Why Screaming Frog is worth using
Screaming Frog has many features you can't do without when you want to optimise your website for both user experience and search engines. For example, the program shows you:
- Whether your images are too large
- Whether your meta tags are too long to display in the SERP
- Whether there are 404 links, both internal and external
- Whether your headline tags (H1, H2, H3, etc.) are correct
- Whether there are canonical links
- … and much more
Seven tips that can improve your website
Screaming Frog can do a great many things. Some features are hard to use, and others are much easier to get to grips with. We've gathered seven tips that are mainly easy to use. Of course, it depends on which CMS you use. If you use WordPress, for example, you're already in a good position when you want to improve your website's SEO.
1. Check whether your meta tags are the right length and grab the reader
2. Get an overview of your image sizes and alt texts
3. Find broken links (404 pages)
4. Analyse redirects when you migrate your site
5. Integrate G Suite products and other SEO tools
6. Crawl your site the way Google would
7. Verify your schema markup
1) Check whether your meta tags are the right length – and engaging!
Your meta tags need to be the right length and engaging at the same time. When Google shows your website for a particular search, you only have a certain number of characters to work with. Here's an example:
The example above shows a meta description that is too long and needs fixing. But what if you have several pages with meta descriptions or titles that are too long? This is where we turn to Screaming Frog. Enter the domain in the program, and go to Overview -> Meta Descriptions. You'll then see which meta descriptions are too long, too short or duplicated, and which other critical things you need to improve.
Once you have an overview of which pages you need to optimise, you find them in your website's backend. If you use WordPress, it's a good idea to install Yoast SEO. You've probably already done so. Go to SEO -> Tools -> Bulk editor. Here you can edit your meta titles and descriptions in bulk, which is easier than clicking into each individual page.
At Amplify, we've built a small application that helps you write engaging meta titles and descriptions. It can increase your CTR (click-through rate) and with it the number of visitors from Google's organic search results. You'll find our CTR tool right here.
2) Get an overview of your image sizes and alt texts
Your website needs to load quickly. That's why there are requirements for how your content loads, especially images. Google is the most popular search engine and primarily analyses your website via its mobile version. Google calls this Mobile-first indexing. What does that mean for you? You need to keep an eye on how your images load, how much space they take up, and which format they're in.
Alt texts also play a role here. In short, an alt text describes your image in words. It especially helps users who are blind or partially sighted, users with screen readers and users with a poor internet connection. They get a description of what the image shows and means when they can't see it.
Google recommends a number of best practices for optimising images. You can read much more about them right here.
So an image on a website takes more than a party photo on your favourite social network. You need to consider both the image's file size and its dimensions. In Screaming Frog, you can see how much space your images take up and which ones are missing alt texts. You'll find this under Overview -> images.
Once you've found the list of images, you can sort them by size and click ‘Inlinks’ at the bottom of the program. You'll then see which images take up the most space and where they are on your website.
Next, you need to find a way to crop and/or compress your images. You can do this in Photoshop, for example. If you only need to compress your images, you can use online tools such as Compress PNG or TinyPNG.
3) Find broken links (404 pages)
Screaming Frog also finds the broken links on your site. That applies both to links to external sources and to links to subpages you've had in the past. Fix them so they send visitors on to the right new page or to content that matches what the now ‘broken’ page showed. That helps the search engines, and it helps your users.
If a visitor lands on a page without getting through to the right content, it's tempting to close the page again. That makes for a poor user experience. That's why you need to get those links under control. Here's how:
Just as with the list of images, you can see where the links appear on your site and which anchor text they may have.
Besides 404 pages, you can see which redirects actually work, which links robots.txt blocks, and much more. In general, keep an eye on which response codes (HTTP status codes) your server sends to visitors to your website. They can affect how a search engine indexes it.
4) Analyse redirects when migrating your site
If you've got a new website, your redirects need to be spot on. That way, you carry the SEO value you already have over to the new site. Otherwise, you risk losing SEO value. That happens if Google ranks one of your old landing pages highly and you don't tell the search engine that the content may have moved.
With Screaming Frog, you can check that the migration between the old and the new site has gone properly and that all your redirects are in place.
First, switch mode from Spider to List. You do this under Mode in the menu:
Then upload the list of your old URLs. There are several ways to do this. We've chosen to do it manually:
Next, set the program to follow your redirects. You do this under Spider -> Advanced -> Always Follow Redirects.
When the crawl is finished, select Reports -> Redirects -> All Redirects. Here you download a .CSV file, which you import into e.g. Excel or Google Sheets. That's where you analyse the results.
Below is an excerpt from an example from one of Amplify's clients. The client's redirects are working as they should and send the traffic on to the intended URL.
5) Integrating G Suite products and other SEO tools
Screaming Frog lets you combine several types of data. That lets you analyse how to optimise your website. Here's an example:
Landing pages without meta tags (according to Screaming Frog) often have a low CTR (click-through rate). The user is given no reason to click through to the pages. At the same time, look at the traffic and the average position for each page. Then you know where optimising pays off. In other words, you use Screaming Frog to find pages without meta tags and combine the findings with data from Search Console (average position and clicks) and Analytics (traffic to the page).
Screaming Frog can connect to these APIs:
- Google Analytics
- Google Search Console
- PageSpeed Insights
- Majestic
- Ahrefs
- Mozscape
You connect to the APIs before you start crawling your website. You do this via the API feature on the right-hand side:
In this example, we've connected to three: Google Analytics, Google Search Console and Ahrefs. Afterwards, we can compare data from the three sources and analyse each individual landing page and how we can optimise it.
6) Crawl your site the way Google would
Screaming Frog crawls your website in its own way when the program shows you how you can optimise it. You can also get the program to crawl the website the way Google would. Why does that matter, you might ask?
It matters because Google decides how your website gets into the index, and therefore where it ranks. Google also indexes your website on a mobile-first basis, as we mentioned earlier in the blog. This means Google crawls your website as if it were a mobile device. The setting in Screaming Frog is important for two reasons:
1. When you imitate the Googlebot Smartphone crawler, you quickly find the problems Google may run into when it crawls and renders your website on a mobile-first basis.
2. When you use the Google Smartphone User-Agent in the program, you can compare your own crawls with Google's when you analyse the server's log files. Then you can see where the differences lie.
The setting is easy to change. Select Configuration -> User-Agent in the menu, and change it there.
7) Verifying your schema markup
Many SEO specialists call structured data the new black. The reason is simple: it can make your website more visible in the organic search results. With structured data, you can do the following:
- Show the number of reviews, the price and stock status directly in the SERP for products.
- Show your frequently asked questions and the answers directly in the SERP. The user gets the answer straight away and can click through to your site to find out more.
- Show the user which events you're holding in the near future.
- … and much more. Google actually has a list of what you can have displayed, when you structure your data. Remember that the search engine decides for itself whether to show it.
Your data needs to be validated so you know it's been set up correctly. Screaming Frog takes care of that. Before you start crawling the website, go to Spider -> Extraction in the menu. Here you can choose JSON-LD, Microdata and RDFa. Your choice depends on how you've structured your data. To be on the safe side, we've selected all of them. Also select Schema.org Validation and Google Rich Result Feature Validation. That way, you don't have to go into Google's own validation tool to see whether the search engine can show your structured data.
Once the program has done its job, look under Overview -> Structured Data. Here you can see whether anything needs fixing:
Those were our seven tips. Screaming Frog can do much more, and with these tips you're well equipped to optimise the basics. That's especially true if you've only just started with search engine optimisation and your opportunities for more organic traffic.
Do you have questions about Screaming Frog or SEO in general?
Find out which keywords your biggest competitors are targeting, and get inspiration for your own keyword research.
Are they ranking for keywords you should be ranking for too? And are there gaps in the market?