Move Over Xenu: Make Way for Screaming Frog’s SEO Spider [Review]
For a long time, Xenu’s Link Sleuth has been a key secret weapon in the SEO’s arsenal. Originally designed for finding bad links in a site, the simple crawler, especially combined with a bit of Excel wizardry, turned out to be useful for all sorts of site audit problems.
However, Xenu’s always had its limitations. That’s fine, it was designed as a simple tool after all. But there’s a new contender on the block, and that contender is Screaming Frog’s SEO Spider. Xenu on crack.
At its simplest level, SEO Spider is similar to Xenu. It’s a desktop application that will crawl any site you give to it. However, it’s a lot more configurable to Xenu, and returns a lot more information.
Critically, unlike Xenu’s static reports, SEO Spider’s report format is effectively a spreadsheet itself, allowing you to filter and reorder results at will. The standard export options are all there, of course.
In a small touch of genius, although SEO Spider has a lot of options, all the defaults are set up to do what you most often want to do: simply crawl a site. Loading up the package, giving it a URL and hitting “Start” will do pretty much the same as what Xenu does out of the box, but returning a lot more information. It also seems to crawl a lot faster than Xenu, although this can be bottle-necked by the server itself.
For each asset on your site, you can drill down into an even deeper level of data, including the number of external links on the page, and the number of internal links pointing to it.
Like Reading Search Engine Watch?
Overview
![screaming-frog-logo screaming-frog-logo](http://cms.searchenginewatch.com/IMG/614/229614/screaming-frog-logo.jpg?1343856841)
Data Returned
SEO Spider returns information in 12 separate tabs:- Internal pages: an overview of all your pages and other resources (images, etc)
- External links: any external links on the site
- Response codes: the response code and redirected URL (if appropriate) for every resource
- URI: details on your site’s URLs
- Page titles: detail on all the page titles on the site
- Meta description: detail on all the meta descriptions on the site
- Meta keywords: detail on all the meta keyword tags on the site (although they’re not used any more)
- H1: detail on your site’s H1 tags
- H2: detail on your site’s H2 tags
- Images: information on all the images found
- Directives: information on your site’s meta tag directives, such as robots and refresh
- Custom: your own custom source code searches
![seo-spider-overview seo-spider-overview](http://cms.searchenginewatch.com/IMG/613/229613/seo-spider-overview.jpg?1343856749)
![seo-spider-deeper seo-spider-deeper](http://cms.searchenginewatch.com/IMG/611/229611/seo-spider-deeper.png?1343856649)
Dynamic Results
These filters are one of the key differences between Xenu’s Link Sleuth and SEO Spider. The latter is specifically designed for SEOs, and allows you to work with the data directly and dynamically. For example, some use cases using filters are:- The “Titles” report allows you to find duplicate tags, pages where the title and H1 tag are the same and overly long or missing titles.
- The “URI” report allows you to find duplicate pages (they have to be exact duplicates, but this is still very useful), dynamic URLs, URLs containing non-ASCII characters, etc.
- The “Directives” report allows you to find all the pages set to noindex, etc.
- The “Custom” report could allow you to find all pages missing Google Analytics code (hands up everyone who has had a developer break the code on some, but not all, of a client’s site).
- The “Images” report lets you find images missing alt text or with overly long alt text.
- The “External links” report is great for finding broken, malformed or redirected external links, helping you to be a good net citizen.
![seo-spider-context-menu seo-spider-context-menu](http://cms.searchenginewatch.com/IMG/610/229610/seo-spider-context-menu.png?1343856449)
Limitations
The only real limitation is that SEO Spider comes in both free and premium versions. The free version is limited to crawling 500 URLs on a given domain, and disables many of the advanced options. In this form, it does everything that Xenu does, just with the URL limit, so if you only ever work with smaller sites, this will suit you fine. If you want to get into more advanced scanning (the custom searches are only available in the premium version, for example) or work with larger sites, the £100 per year price tag is pretty trivial.Like Reading Search Engine Watch?
Then you'll love attending SES San Francisco 2012 where you can meet, interact and learn from all your favorite industry experts. Immerse yourself in 5 days of unparalleled education with:
- 55+ sessions on PPC, SEO, social media, link building, mobile, and more
- Presentations from Facebook and LinkedIn
- 75+ exhibitors
- Abundant networking events.
- New session formats: SES Talkback and Meet the Experts
Register before August 13 to save $ 200!
![](http://pixel.quantserve.com/pixel/p-89EKCgBk8MZdE.gif)