Seven tips to improve your SEO with Screaming Frog

It can often be difficult to get an overview of which SEO measures can be improved on your website. In fact, it can be overwhelming to think about where to start with the technical and visual analysis of your website.

In this guide, you'll get an introduction and of course tips to the SEO program that is loved and appreciated by SEO specialists, generalists and digital marketing consultants all over the world - Screaming Frog. After reading this guide, you'll be ready to improve your website - and influence search engine optimization in a positive direction. Welcome to Screaming Frog!

screaming-frog

What is Screaming Frog?

Screaming Frog is an application that can troubleshoot and visualize technical issues on your website. The program works like a spider, similar to the way a search engine such as Google crawls your website. This is smart, as it allows you to improve challenges on your website - and optimize them in a positive direction.

The program is available for PC (Windows), Mac and Linux and is available in a free and paid version. In the free version, you can crawl a maximum of 500 URLs, and various technical features will not be available - for example, you cannot see which pages have structured data implemented or integrate data from Google Analytics.

If you're not familiar with structured data, I've written an introduction to the topic right here

Anyway, back to Screaming Frog! The premium version of the program costs £149 (approx. 1235 dkk) per year, where you get access to all features. However, you may want to test the free version for a period of time and then decide if a license is necessary. It's worth noting that you only have limited access to various features in the free version.

Why should I use Screaming Frog?

The program has many indispensable features that make it possible to optimize your website in terms of user experience and search engines. For example, you can find out the following about your website:

  • Whether images are too large
  • Whether there are meta tags that are too long in relation to display in the SERP
  • Whether there are 404 links - both internal and external
  • Whether you have correct headline tags (H1, H2, H3 etc)
  • Whether there are canonicals links
  • ... and much more

Seven tips to improve your website

Screaming Frog can do a lot of things. Some of the features are difficult to use, while others are much easier to use. I've put together seven tips that are mainly easy to use. Of course, it depends on which CMS you use. If you're using WordPress, for example, you're already well placed to improve your website SEO-wise.

1. Check if meta tags are the correct length - and catchy!

2. Get an overview of your image sizes and alt texts

3. Find broken links (404 pages)

4. Analyze redirects during page migration

5. Integration of G Suite products and other SEO tools

6. crawl your site the way Google will

7. Verification of your schema markup

1) Check if meta tags are the correct length - and catchy!

It's important that your meta tags are the right length, but also catchy. When your website is exposed on Google based on a specific search criterion, you only have a certain number of characters to work with. An example is the following:

too far meta tag screaming frog

The above example shows that the meta description is too long and needs to be fixed. But what if you have multiple pages where the meta description or title is too long? To find out, this is where we put Screaming Frog to work. By entering the domain into the program and then navigating to Overview -> Meta Descriptions, we get an insight into which meta descriptions are too long, too short, duplicated or other critical things that need improvement.

screaming frog overview of meta

Once you have an overview of which pages to optimize, just find them in the backend of your website. If you're using WordPress, you can install Yoast SEO (I wonder if you've already done that? ;)) and navigate to SEO -> Tools -> Bulk editing. Here you can edit your meta titles and descriptions in bulk - which is easier than clicking into each page.

At Amplify, we've even created a small application that can help you write catchy meta titles and descriptions. This can help increase your CTR (click-through-rate) and thus increase the number of visitors via the organic search results on Google. You can find our CTR tool right here.

2) Get an overview of your image sizes and alt texts

A website needs to load quickly. Therefore, demands are placed on how your content is loaded - especially images. The most popular search engine, Google, primarily analyzes your website via the mobile version - also known as mobile-first indexing. So what does this mean for you? It means you need to focus on how images are loaded, how much space they take up and what format your images are in.
In this context, alt texts are also important. In short, alt texts describe your images in a written context. This is especially relevant for users with partially or fully impaired vision and users who use a screen reader or have a poor internet connection. This allows them to get a description of what the image contains and means if they can't see it.

You can read more about what Google recommends as best practices for image optimization right here.

Putting an image on a website is therefore not as simple as uploading a party photo on your favorite social media. The size and dimensions of the image itself must be taken into account. Via Screaming Frog, you can see both how much space your images take up and which ones are missing alt texts. This can be found under Overview -> images.

screaming frog image size

Once you have found the list of images, you can sort them by size and then click "Inlinks" at the bottom of the program. This will give you an insight into which images take up the most space and where they are placed on your website.

Next, find a way to crop and/or compress your images. You can do this in Photoshop, for example. If you only need to compress your images, you can use online services such as Compress PNG or TinyPNG.

3) Find broken links (404 pages)

With Screaming Frog, you also have the option to find broken links that either point to external sources or previous subpages on your site. It's generally good practice to fix these links so that they are redirected to the correct new page or equivalent content that appeared on the now 'broken' page. Not only for the sake of a search engine, but also for the user experience.

For example, if you're not redirected to the right content, it's very obvious to shut down the site - it will create a poor user experience. That's why it's important that you get a handle on these links. Here's how you do it:

Broken links - 404 pages

As with the list of images, you have the option to find the links' location on your website - and in which anchor text, if any.

Besides 404 pages, you can also see which redirects actually work, which links are blocked by robots.txt and much more. In general, you should be aware of what response codes your serveris sending out to the client visiting your website, as this can affect how a search engine indexes it.

4) Analyze redirects during page migration

If you have a new website and you want to ensure that the existing SEO value will be 'transferred', it's important that your redirects are spot on. Otherwise, you will potentially lose SEO value if Google indexes one of your old landing pages highly and you don't tell the search engine that the content may have changed location.

With Screaming Frog, you can ensure that the migration from the new to the old site is done properly and that all redirects are done.

First you need to switch the mode from Spider to List. This is done under Mode in the menu:

screaming fro former urls

Once this is done, you need to upload the list with your old URLs. There are several options, but I chose to do it manually:

Next, you need to set the program to follow your redirects. This is done under Spider -> Advanced -> Always Follow Redirects.

screaming frog all redirects

Once the crawl has been completed, select Reports -> Redirects -> All Redirectsand you can download a .CSV file that you can import into e.g. Excel or Google Sheets and analyze the results there.

For the record, I've created a snippet of an example where you can see that one of Amplify's customers' redirects is set up correctly and sends traffic to the desired URL.

screaming frog redirect analysis

5) Integration of G Suite products and other SEO tools

Through Screaming Frog, you actually have the opportunity to combine several different types of data to analyze how you can optimize your website. An example could be the following:

Landing pages with missing meta tags (see Screaming Frog) often have a low CTR (click through rate) as the user is not influenced to visit the pages. By also looking at the amount of traffic and the average ranking on the specific page, you can optimize for this. In this case, you use Screaming Frog to find pages with missing meta tags while interacting with data from Search Console (average ranking and clicks) and Analytics (traffic to the page).

Via Screaming Frog you have the option to connect with the following APIs:

  • Google Analytics
  • Google Search Console
  • PageSpeed Insights
  • Majestic
  • Ahrefs
  • Mozscape

To connect to various APIs, you need to connect to them before you start crawling your website. This is done via the API function on the right side:

screaming frog api

In this example I have done this with three different ones; Google Analytics, Google Search Console and Ahrefs. Then I have the opportunity to compare data from the three sources and analyze each landing page and how they can be optimized.

6) Crawl your page the way Google will

Screaming Frog has its own way of crawling a website to visualize and tell you how to optimize it. But actually, you can also have the program crawl the website as if it were Google. Why is that relevant, you ask?

That's because Google decides how your website should be indexed - and thus how it should be ranked. In addition, the search engine also indexes your website based on a mobile-first principle (as mentioned earlier in this blog) - that is, it crawls your website as if it were a mobile device. The fact that you can change this setting in Screaming Frog is important because of two factors:

1. By emulating the Googlebot smartphone crawler, you can quickly find out what problems Google may have when crawling and rendering your website from a mobile-first perspective

2. Using the Google Smartphone User-Agent in the application, you can compare your crawls and Google's own when analyzing server logs - to see where the difference lies.

This is done relatively easily. By selecting Configuration -> User-Agent in the menu, you can easily change this.

7) Verification of your schema markup

Structured data is the new black if you ask many SEO specialists. This is because it can pave the way for your website to become more visible in organic search results. With structured data, you can do the following:

  • See the number of reviews, price and stock availability directly in the SERP for products.
  • Display your FAQs and the answers directly in the SERP. This way, the user can get an answer directly and click through to your site if they want to know more.
  • Allow the user to see what events you are hosting in the near future.
  • ... and much more. In fact, Google has a list of what you can display if you structure your data. However, keep in mind that it's up to the search engine itself whether it wants to show it or not.

However, this data needs to be validated so that it is set up correctly - and you can use Screaming Frog to do this. Before you start crawling the website, navigate to Spider -> Extraction in the menu. Here you can select JSON-LD, Microdata, and RDFa respectively. However, it depends on how you have chosen to structure your data. For the sake of good order, I have chosen them all. You can also choose Schema.org Validation and Google Rich Result Feature Validation, as you don't have to use Google's own validation tool to see if the data you have structured will be displayed on the search engine.

After the program has done its work, you can take a look under Overview -> Structured Data to see if there is anything that needs fixing:

Those were the seven tips I had to offer. You can use Screaming Frog for so much more, but with these tips you are at least well equipped to optimize the most basic options - especially if you are just now starting to explore search engine optimization and your options for increasing organic traffic.

Do you have questions about Screaming Frog or SEO in general?

By researching which keywords your biggest competitors are targeting, you can get inspiration for your own keyword analysis.

Are they ranking on keywords you should also be ranking on? And are there gaps in the market?

Gain deeper insights

Whether you're a generalist or a marketing specialist, our specialists have put together some great advice for you on our blog.

Write a reply