Screaming Frog is a website crawler that helps you improve onsite SEO by extracting data and auditing for common SEO issues like broken links and redirect chains. This desktop application acts as a bridge between your local machine and a website’s server, providing a granular view of how search engine bots interpret your site’s structure and content. By using this tool, SEO practitioners can quickly identify technical barriers that prevent efficient indexing and ranking in search engine results pages.
Setting up your first crawl in Screaming Frog
To begin your technical audit, you must first configure the software to handle the specific scale of your website. After installation, the default mode is the Spider mode, which allows you to enter a root URL and let the software follow every internal link it finds. For a typical e-commerce site with thousands of product pages, you should first check your memory settings. Navigate to Configuration > System > Memory to ensure the software can utilize enough RAM to store the crawl data without crashing. Most modern computers can handle 4GB to 8GB for medium-sized sites, but larger domains may require 16GB or more to finish the process.
Once your memory is allocated, enter your website address into the ‘Enter URL to spider’ box at the top of the interface. Before clicking start, it is a common practice to review the Spider configuration under the Configuration menu. Here, you can decide whether you want to crawl images, CSS, and JavaScript files. While crawling these assets provides a more complete picture of the page load process, it also increases the time and memory required for the crawl. If your goal is a quick check of status codes and titles, you might disable these to speed up the initial data gathering phase.
How do you configure the spider for a full crawl?
A full technical audit requires the spider to look beyond simple HTML links. In the Configuration > Spider menu, you should enable the ‘Crawl All Subdomains’ option if your site uses various prefixes like ‘blog.example.com’ or ‘shop.example.com’. This ensures that the spider does not stop at the primary domain boundary. Additionally, you should check the ‘Crawl Outside Start Folder’ box if you are starting the crawl from a specific subfolder but want to see how it connects to the rest of the site’s architecture. These settings are fundamental for capturing the complete scope of a domain’s internal linking environment.
Another advanced setting involves JavaScript rendering. Many modern websites built on frameworks like React or Angular do not display content until JavaScript is executed. If you crawl these sites using the default ‘Text Only’ mode, the spider will see an empty page. To fix this, go to Configuration > Spider > Rendering and select ‘JavaScript’ from the dropdown menu. This tells Screaming Frog to use an integrated Chrome browser to render the pages, allowing it to find links and content that would otherwise remain hidden from a basic crawler. This is often the difference between a successful audit and a failed one for modern web applications.
Identifying broken links and redirect issues
One of the primary uses of this software is identifying 404 errors and redirect chains. Once the crawl is complete, or even while it is running, you can click on the ‘Response Codes’ tab to see a list of every URL and its status. Filtering by ‘Client Error (4xx)’ will immediately show you every broken link on the site. However, seeing the broken URL is only half the battle; you need to know where those links are located. By selecting a 404 URL in the top window and clicking the ‘Inlinks’ tab at the bottom, the software displays every page that links to that broken resource, making it easy to find and update the source code.
Redirect chains are another common issue that impacts both user experience and crawl efficiency. A redirect chain happens when URL A points to URL B, which then points to URL C. This creates unnecessary latency and consumes more crawl budget. To find these, go to Reports > Redirects > Redirect Chains after your crawl is finished. This report provides a clear map of every ‘hop’ in a redirect sequence. At Alien Road, we suggest resolving these by updating the initial link to point directly to the final destination, which simplifies the path for both users and search engine crawlers.
What are the most common metadata errors to find?
- Duplicate page titles that confuse search engines about which page is most relevant.
- Missing meta descriptions which lead to auto-generated snippets in search results.
- Page titles over 60 characters or 568 pixels that get truncated in the SERPs.
- Multiple H1 tags on a single page, which can dilute the primary topical focus.
- Missing alt text on images, reducing the accessibility and image search potential.
The ‘Page Titles’ and ‘Meta Description’ tabs in the software are designed to highlight these issues through built-in filters. For instance, you can use the ‘Over 60 Characters’ filter in the Page Titles tab to identify every page where your title might be cut off in Google search results. Similarly, the ‘Duplicate’ filter helps identify instances where you might have accidentally used the same title for different product variations. Fixing these metadata issues is a high-priority task because it directly influences the click-through rate from search engines to your website.
Analyzing site architecture and crawl depth
Crawl depth refers to the number of clicks required to reach a page from the homepage. In the ‘Internal’ tab, the ‘Crawl Depth’ column provides this data for every URL. Generally, important content should be accessible within three clicks or fewer. If you find that critical conversion pages have a crawl depth of five or six, it indicates a poor internal linking structure. This can lead to those pages being crawled less frequently or being perceived as less important by search engine algorithms. You can use the ‘Site Structure’ tab in the right-hand window to see a visual breakdown of how your pages are distributed across different levels of the site.
Visualizations are also a powerful way to explain architectural problems to stakeholders. Under the ‘Visualisations’ menu, you can choose ‘Crawl Tree Graph’ or ‘Force-Directed Directory Tree Diagram’. These interactive maps show the relationships between folders and pages. A messy or overly complex diagram often highlights a lack of logical categorization. For example, if a blog post is buried under five subdirectories, the visualization will show a long, thin branch stretching away from the center. This visual evidence is often more persuasive than a spreadsheet when recommending a site structure overhaul.
How can you use Screaming Frog for a site migration?
During a site migration, ensuring that old URLs redirect correctly to new ones is the most important task for maintaining SEO rankings. You can use the ‘List Mode’ in Screaming Frog to test your redirect map. Instead of crawling a live site, change the mode to ‘List’ under the Mode menu and upload a text file or CSV containing your old URLs. Before you start the crawl, go to Configuration > Spider > Advanced and ensure ‘Always Follow Redirects’ is checked. Once you run the crawl, the software will show you the final destination for each old URL and the status code encountered along the way.
This process allows you to verify hundreds of redirects in a matter of seconds. You can compare the ‘Address’ column (the old URL) with the ‘Redirect URL’ column (the new URL) to ensure they match your intended redirect map. If you see 404 errors in the list, it means the redirect was not implemented correctly or the destination page is missing. This proactive testing prevents the loss of organic traffic that typically occurs when a migration is handled without rigorous technical verification. It is a standard procedure for any agency managing a domain transition or a major URL structure change.
Integrating API data for comprehensive audits
Screaming Frog becomes even more effective when you connect it to external data sources. By navigating to Configuration > API Access, you can link the software to Google Analytics, Google Search Console, and PageSpeed Insights. This allows you to see crawl data alongside performance metrics. For example, you can identify pages that have high crawl depth but also receive a significant amount of organic traffic from Search Console. These are prime candidates for better internal linking because they are already performing well despite being buried in the site architecture.
Connecting the PageSpeed Insights API is particularly useful for auditing Core Web Vitals. The software will fetch metrics like Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS) for every URL in your crawl. Instead of checking pages one by one in a browser, you get a bulk report showing which sections of your site fail to meet the 2.5-second LCP threshold. This data-driven approach allows you to prioritize technical fixes for the specific templates or page types that are dragging down your overall site performance and user experience scores.
Custom extraction for specific data needs
Sometimes you need to audit data that is not part of standard SEO tags, such as stock status, price, or author names. The Custom Extraction feature allows you to use XPath, CSS Path, or Regex to pull specific text from the HTML of your pages. For instance, if you want to ensure all blog posts have a specific author markup, you can identify the CSS class used for the author name and set up an extractor in Configuration > Custom > Extraction. During the crawl, the software will create a new column for this data, allowing you to see at a glance which pages are missing information.
This is also helpful for auditing structured data implementation. While there is a dedicated structured data tab, custom extraction can be used to double-check that specific values within your Schema.org markup are correct across thousands of pages. If an e-commerce site has a ‘Price’ field in its JSON-LD, you can extract that value and compare it against the visible price on the page to ensure consistency. This level of detail helps prevent issues with Google Merchant Center or rich snippet displays that rely on accurate and consistent data across the entire domain.
Where to start this week
- Download the free version or license the full version of Screaming Frog and run a basic crawl of your domain.
- Check the ‘Response Codes’ tab and export all 4xx errors to identify broken links that need immediate fixing.
- Navigate to the ‘Page Titles’ tab and filter for duplicates to resolve internal competition between similar pages.
- Connect the Google Search Console API to identify pages that are indexed but receive zero clicks over the last 90 days.
- Use the ‘Crawl Analysis’ feature under the ‘Crawl Analysis’ menu to calculate link score and internal PageRank.
By following these steps, you will move from a surface-level understanding of your site to a deep technical knowledge of how it functions. Regular crawling is a habit that ensures small errors do not grow into major indexing problems. Start with the most critical errors—broken links and redirect loops—before moving on to more complex optimizations like site architecture and Core Web Vitals improvements.
Keep reading: SEO (Search Engine Optimization) · SEO