Seven Tips for Improving Your SEO Using Screaming Frog
It can often be difficult to get a clear picture of which SEO measures on your website could be improved. In fact, it can be overwhelming to figure out where to start with the technical and visual analysis of your website.
In this guide, you’ll find an introduction and, of course, tips for the SEO tool that’s loved and valued by SEO specialists, generalists, and digital marketing consultants around the world—namely, Screaming Frog. Once you’ve read this guide, you’ll be ready to improve your website—and thereby positively impact your search engine optimization. Welcome!
What is Screaming Frog?
Screaming Frog is an application that can troubleshoot and visualize technical issues on your website. The program acts as a spider, functioning in a way that’s similar to how a search engine like Google crawls your website. This is useful because it allows you to address issues on your website—and optimize them for better performance.
The program is available for download on PC (Windows), Mac, and Linux, and comes in both a free and a paid version. In the free version, you can crawl a maximum of 500 URLs, and various technical features will not be available—for example, you cannot view which pages have structured data implemented or integrate data from Google Analytics.
If you're not familiar with structured data, I've written an introduction to the topic right here.
Anyway, back to Screaming Frog! The premium version of the program costs £149 (approx. 1,235 DKK) per year, which gives you access to all features. However, it’s a good idea to try the free version for a while and then decide whether you need a license. It’s worth noting, however, that you only have limited access to various features in the free version.
Why should I use Screaming Frog?
The program offers many essential features that allow you to optimize your website for both user experience and search engines. For example, you can find out the following about your website:
- Whether the images are too large
- Whether there are meta tags that are too long to be displayed in the SERP
- Whether there are any 404 links —both internal and external
- Whether you have the correct headline tags (H1, H2, H3, etc.)
- Whether there are canonical links
- … and much more
Seven Tips to Improve Your Website
Screaming Frog can do a lot of things. Some of its features are difficult to use, while others are significantly easier to navigate. I’ve put together seven tips that are primarily easy to use. Of course, it depends on which CMS you use. If, for example, you use WordPress, you’re already in a good position to improve your website’s SEO.
1. Check that the meta tags are the correct length—and catchy!
2. Get an overview of your image sizes and alt text
3. Find broken links (404 pages)
4. Analyze redirects during website migration
5. Integration of G Suite products and other SEO tools
6. Crawl your site the way Google would
7. Verifying Your Schema Markup
1) Check that the meta tags are the correct length—and catchy!
It’s important that your meta tags are the right length, but also catchy. When your website appears in Google search results based on a specific search query, you only have a certain number of characters to work with. Here’s an example:
The example above shows that the meta description is too long and needs to be corrected. But what if you have multiple pages where the meta description or title is too long? To find out, this is where we bring Screaming Frog into play. By entering the domain into the program and then navigating to Overview -> Meta Descriptions, we get an overview of which meta descriptions are too long, too short, duplicated, or have other critical issues that need to be addressed.
Once you’ve gotten an overview of which pages need to be optimized, all you have to do is find them in your website’s backend. If you use WordPress, I recommend installing Yoast SEO (Don’t tell me you haven’t already? ;)), and navigating to SEO -> Tools -> Bulk Edit. Here you can edit your meta titles and descriptions in bulk—which is easier than clicking into each page individually.
At Amplify, we’ve even created a small tool that can help you write compelling meta titles and descriptions. This can help increase your CTR (click-through rate) and, as a result, boost the number of visitors coming from Google’s organic search results. You can find our CTR tool right here.
2) Get an overview of your image sizes and alt text
A website needs to load quickly. That’s why there are requirements for how your content is loaded—especially images. The most popular search engine, Google, primarily analyzes your website via the mobile version—also known as “mobile-first indexing.” So what does that mean for you? It means you need to focus on how images load, how much space they take up, and what format your images are in.
In this context, alt text is also important. In short, alt text describes your images in written form. This is especially relevant for users with total or partial vision impairment, as well as users who rely on a screen reader or have a poor internet connection. This way, they can receive a description of what the image contains and means if they cannot see it.
You can read much more about Google's best practices for image optimization right here.
Adding an image to a website is therefore not as simple as uploading a party photo to your favorite social media platform. You need to take into account the size of the image itself as well as its dimensions. Using Screaming Frog, you can see both how much space your images take up and which ones are missing alt text. You’ll find this under Overview -> images.
Once you’ve found the list of images, it’s a good idea to sort them by size and then click “Inlinks” at the bottom of the program. This will give you an idea of which images take up the most space and where they’re located on your website.
Next, you'll need to find a way to crop and/or compress your images. You can do this in Photoshop, for example. If you only need to compress your images, you might want to use online services such as Compress PNG or TinyPNG.
3) Find broken links (404 pages)
With Screaming Frog, you can also find broken links that either point to external sources or to former subpages on your website. It’s generally recommended that these links be fixed so they redirect to the correct new page or equivalent content that was on the now “broken” page. This is important not only for search engines but also for the user experience.
For example, it makes perfect sense to close the page if you aren’t redirected to the correct content—all things being equal, that will result in a poor user experience. That’s why it’s important to keep these links under control. Here’s how to do it:
Just as with the list of images, you can find where the links are located on your website—and, if applicable, what anchor text they use.
In addition to 404 pages, you can also see which redirects are actually working, which links are blocked by robots.txt, and much more. In general, you should pay attention to the response codes your server sends to clients visiting your website, as this can affect how a search engine indexes it.
4) Analyze redirects during site migration
If you’ve had a new website built and want to ensure that the existing SEO value is “transferred,” it’s important that your redirects are set up correctly. Otherwise, you could potentially lose SEO value if Google ranks one of your old landing pages highly and you don’t tell the search engine that the content may have moved to a new location.
With Screaming Frog, you can ensure that the migration from the new site to the old one is done properly and that all redirects are set up.
First, switch the mode from Spider to List. You can do this under " Mode " in the menu:
Once you've done that, upload the list of your old URLs. There are several options, but I've chosen to do it manually:
Next, you need to set the program to follow your redirects. You can do this under Spider -> Advanced -> Always Follow Redirects.
Once the crawl is complete, select Reports -> Redirects -> All Redirects. You can then download a .CSV file, which you can import into Excel or Google Sheets, for example, and analyze the results there.
Just to be clear, I've included a screenshot of an example where you can see that one of Amplify's customers has set up their redirects correctly, and they're sending traffic to the desired URL.
5) Integration of G Suite products and other SEO tools
With Screaming Frog, you can actually combine several different types of data to analyze how to optimize your website. Here’s an example:
Landing pages with missing meta tags (according to Screaming Frog) often have a low CTR (click-through rate), since users aren’t motivated to visit those pages. By also examining the amount of traffic and the average ranking for that specific page, you can effectively optimize it. In this case, use Screaming Frog to identify pages with missing meta tags while also analyzing data from Search Console (average ranking and clicks) and Analytics (traffic to the page).
Using Screaming Frog, you can connect to the following APIs:
- Google Analytics
- Google Search Console
- PageSpeed Insights
- Majestic
- Ahrefs
- Mozscape
To connect to various APIs, you must connect to them before you start crawling your website. This is done using the API feature on the right-hand side:
In this example, I’ve done this using three different tools: Google Analytics, Google Search Console, and Ahrefs. Next, I can compare data from these three sources and thus analyze each landing page and determine how they can be optimized.
6) Crawl your site the way Google would
Screaming Frog has its own way of crawling a website when it needs to visualize and explain how you can optimize it. But you can actually also have the program crawl the website as if it were Google. Why is that relevant, you might ask?
That’s because Google determines how your website is indexed—and thus how it’s ranked. In addition, the search engine also indexes your website based on a “mobile-first” principle (as mentioned earlier in this blog)—meaning it crawls your website as if it were a mobile device. The fact that you can change this setting in Screaming Frog is important for two reasons:
1. By simulating the Googlebot Smartphone crawler, you can quickly identify any issues Google might encounter when crawling and rendering your website based on a mobile-first approach
2. By using the Google Smartphone User-Agent in the program, you can compare your crawls with Google’s own when analyzing server log files—to see where the differences lie.
This is relatively easy to do. By selecting Configuration -> User-Agent from the menu, you can easily change this.
7) Verifying Your Schema Markup
Structured data is the new trend, if you ask many SEO specialists. This is because it can pave the way for your website to become more visible in organic search results. With structured data, you can do the following:
- View the number of reviews, price, and stock availability directly in the SERP for products.
- Display your frequently asked questions and answers directly in the SERP. This allows users to get an answer right away and click through to your site if they want to learn more.
- Let users see what events you're hosting in the near future.
- … and much more. In fact , Google has a list of what you can have displayed if you structure your data. Keep in mind, however, that it’s up to the search engine itself whether or not it will display it.
However, this data needs to be validated to ensure it’s set up correctly—and you can use Screaming Frog for that. Before you start crawling the website, navigate to Spider -> Extraction in the menu. Here, you can select JSON-LD, Microdata, or RDFa, depending on how you’ve chosen to structure your data. Just to be on the safe side, I’ve selected all of them. In addition, you should select Schema.org Validation and Google Rich Results Feature Validation, as this will save you from having to use Google’s own validation tool to check whether the data you’ve structured will appear in the search engine results.
After the program has run its course, you can check under Overview -> Structured Data to see if there's anything that needs to be corrected:
Those were the seven tips I had to offer. You can use Screaming Frog for much, much more, but with these tips, you’ll definitely be well-equipped to optimize the most basic features—especially if you’ve only just started exploring search engine optimization and your options for increasing organic traffic.
Do you have any questions about Screaming Frog or SEO in general?
By researching which keywords your biggest competitors are targeting, you can gain inspiration for your own keyword analysis.
Are they ranking for keywords that you should also be ranking for? And are there gaps in the market?
Learn Even More
Whether you're a generalist or a specialist in marketing, our experts have put together a number of helpful tips for you on our blog.
