What is Screaming Frog?
Screaming Frog is an application that can find and visualise technical issues on your website. The program works as a spider, somewhat like the way a search engine such as Google crawls your website. That's handy, because it lets you fix problems on your website and optimise them in a positive direction.
The program is available for PC (Windows), Mac and Linux, and comes in a free and paid version. In the free version you can crawl a maximum of 500 URLs, and various technical features aren't available. For example, you can't see which pages have structured data implemented, or integrate data from Google Analytics.
If you're not familiar with structured data, I've written an introduction to the topic right here.
Anyway, back to Screaming Frog! The premium version of the program costs £149 (approx. DKK 1,235) a year and gives you access to all features. That said, it's a good idea to test the free version for a while and then decide whether you need a licence. It's worth noting again, though, that you only have limited access to various features in the free version.
Why should I use Screaming Frog?
The program has many indispensable features that make it possible to optimise your website for user experience and search engines. For example, you can find out the following about your site:
- Whether any images are too large
- Whether any meta tags are too long to display in the SERP
- Whether there are 404 links, both internal and external
- Whether you have correct headline tags (H1, H2, H3 etc.)
- Whether there are canonical links
- … and much more
Seven tips that can improve your website
Screaming Frog can do a great many things. Some of the features are hard to use, while others are much easier to get to grips with. I've put together seven tips that are mainly easy to use. Of course, it depends on which CMS you use. If you use WordPress, for example, you're already well placed to improve your website SEO-wise.
1. Check whether your meta tags are the right length – and engaging!
2. Get an overview of your image sizes and alt texts
3. Find broken links (404 pages)
4. Analyse redirects when migrating a site
5. Integrate G Suite products and other SEO tools
6. Crawl your site the way Google would
7. Verify your schema markup
1) Check whether your meta tags are the right length – and engaging!
It's important that your meta tags are the right length, but also engaging. When your website appears on Google for a specific search, you only have a certain number of characters to work with. Here's an example:
The example above shows that the meta description is too long and needs fixing. But what if you have several pages where the meta description or title is too long? To find out, this is where we bring in Screaming Frog. By entering the domain in the program and then navigating to Overview -> Meta Descriptions, we get an insight into which meta descriptions are too long, too short, duplicated or have other critical issues that need improving.
Once you have an overview of which pages need optimising, it's simply a case of finding them in your website's backend. If you use WordPress, it's a good idea to install Yoast SEO (I bet you already have? ;)) and navigate to SEO -> Tools -> Bulk editor. Here you can edit your meta titles and descriptions in bulk, which is easier than clicking into each page.
At Amplify, we've even built a small application that can help you write engaging meta titles and descriptions. This can help increase your CTR (click-through rate) and thereby the number of visitors from the organic search results on Google. You'll find our CTR tool right here.
2) Get an overview of your image sizes and alt texts
A website needs to load quickly. That's why there are requirements for how your content is loaded, especially images. The most popular search engine, Google, mainly analyses your website via the mobile version, also known as Mobile-first indexing. So what does that mean for you? It means you need to focus on how images are loaded, how much space they take up, and what format your images are in.
In this context, alt texts are also important. In short, alt texts describe your images in writing. This is especially relevant for users who are blind or partially sighted, as well as users who rely on a screen reader or have a poor internet connection. It gives them a description of what the image contains and means if they can't see it.
You can read much more about what Google recommends as best practice for optimising images right here.
Adding an image to a website is therefore not quite as simple as uploading a party photo to your favourite social media platform. You need to take into account the file size of the image as well as its dimensions. In Screaming Frog, you can see both how much space your images take up and which ones are missing alt texts. You'll find this under Overview -> images.
Once you've found the list of images, it's a good idea to sort them by size and then click ‘Inlinks’ at the bottom of the program. This gives you an insight into which images take up the most space and where they're placed on your website.
Next, you need to find a way to crop and/or compress your images. You can do this in Photoshop, for example. If you only need to compress your images, online services such as Compress PNG or TinyPNG are a good option.
3) Find broken links (404 pages)
Screaming Frog also lets you find broken links that point either to external sources or to former subpages on your site. It's generally good practice to fix these links so they redirect to the correct new page or to equivalent content to what appeared on the now ‘broken’ page. Not only for the search engine's sake, but also for the user experience.
For example, it's very tempting to close the page if you aren't redirected to the right content, which will inevitably create a poor user experience. That's why it's important to get these links sorted. Here's how:
Just as with the list of images, you can find where the links are placed on your site, and with which anchor text, if any.
Besides 404 pages, you can also see which redirects actually work, which links are blocked by robots.txt and much more. In general, you should keep an eye on which response codes (HTTP status codes) your server sends to the client visiting your website, as this can affect how a search engine indexes it.
4) Analyse redirects when migrating a site
If you've had a new website built and want to make sure your existing SEO value is ‘transferred’, it's important that your redirects are spot on. Otherwise, you could lose SEO value if Google ranks one of your old landing pages highly and you don't tell the search engine that the content may have moved.
With Screaming Frog, you can make sure the migration between the new and the old site has been done properly and that all redirects are in place.
First, you need to switch mode from Spider to List. You do this under Mode in the menu:
Once that's done, you need to upload the list of your old URLs. There are several options, but I've chosen to do it manually:
Then you need to set the program to follow your redirects. You do this under Spider -> Advanced -> Always Follow Redirects.
Once the crawl is complete, select Reports -> Redirects -> All Redirects, after which you can download a .CSV file that you can import into Excel or Google Sheets, for example, and analyse the results there.
For good measure, I've included an excerpt from an example where you can see that one of Amplify's clients' redirects have been set up as they should and send the traffic on to the intended URL.
5) Integrate G Suite products and other SEO tools
Screaming Frog actually lets you combine several different types of data and use them to analyse how you can optimise your website. Here's an example:
Landing pages with missing meta tags (according to Screaming Frog) often have a low CTR (click-through rate), because users aren't prompted to visit the pages. By also looking at the amount of traffic and the average position of the specific page, you can optimise for this. In that case, you use Screaming Frog to find pages with missing meta tags, while working with the data from Search Console (average position and clicks) and Analytics (traffic to the page).
Screaming Frog lets you connect to the following APIs:
- Google Analytics
- Google Search Console
- PageSpeed Insights
- Majestic
- Ahrefs
- Mozscape
To connect to the various APIs, you need to do so before you start crawling your website. You do this via the API feature on the right-hand side:
In this example, I've done it with three different ones: Google Analytics, Google Search Console and Ahrefs. I can then compare data from the three sources and analyse each individual landing page and how it can be optimised.
6) Crawl your site the way Google would
Screaming Frog has its own way of crawling a website when it visualises and tells you how you can optimise it. But you can actually also get the program to crawl the website as if it were Google. Why does that matter, you ask?
It matters because it's Google that decides how your website is indexed, and therefore how it's ranked. The search engine also indexes your website on a mobile-first principle (as mentioned earlier in this blog), meaning it crawls your website as if it were a mobile device. Being able to change this setting in Screaming Frog is important for two reasons:
1. By imitating the Googlebot Smartphone crawler, you can quickly find out what problems Google might have when it crawls and renders your website on a mobile-first principle
2. By using the Google Smartphone User-Agent in the program, you can compare your crawls with Google's own when analysing server log files, to see where the differences lie.
This is relatively easy to do. By selecting Configuration -> User-Agent in the menu, you can easily change it.
7) Verify your schema markup
Structured data is the new black, if you ask many SEO specialists. That's because it can pave the way for your website to become more visible in the organic search results. With structured data, you can do the following:
- Show the number of reviews, the price and stock availability directly in the SERP for products.
- Show your frequently asked questions and the answers directly in the SERP. That way, users can get an answer straight away and click through to your site if they want to know more.
- Let users see which events you're holding over the coming period.
- … and much more. In fact, Google has a list of what you can display, if you structure your data. Remember, though, that it's up to the search engine itself whether it shows it or not.
This data does need validating, however, to make sure it's set up correctly, and you can use Screaming Frog for that. Before you start crawling the website, navigate to Spider -> Extraction in the menu. Here you can choose JSON-LD, Microdata and RDFa. It depends on how you've chosen to structure your data. For good measure, I've selected all of them. It's also a good idea to select Schema.org Validation and Google Rich Result Feature Validation, as you then don't need to go into Google's own validation tool to see whether the data you've structured will be shown in the search engine.
Once the program has done its job, you can take a look under Overview -> Structured Data to see if anything needs fixing:
Those were the seven tips I had to offer. You can use Screaming Frog for much, much more, but with these tips you're well equipped to optimise the most basic areas, especially if you've only just started looking into search engine optimisation and your options for more organic traffic.
Do you have questions about Screaming Frog or SEO in general?
By looking into which keywords your biggest competitors are going after, you can get inspiration for your own keyword analysis.
Are they ranking for keywords you should be ranking for too? And are there gaps in the market?
