The Crawl Tool Blog

The Crawl Tool is not only the best SEO tool for small to medium websites and web professionals working with them, it also regularly adds new features and functionality. For that reason, we've decided to do a weekly update so you can assess whether you want to use the new features and learn a bit about how to use them.

Favicons and Show All Button

We've added favicons into the projects list. It's a small thing but it's not just a nice visual touch, it helps you find the project you want to look at quicker.

With the SEO Dashboard for a project you get a lot of information. But perhaps you want to simplify it? If there are no broken links then a broken links box that says 0 may be re-assuring but it doesn't tell you much. At the top of the SEO dashboard there is now a "Show All" toggle. This lets you decide for yourself if you want to see all the boxes, or only those with actionable information.

Fresh

The Crawl Tool's crawler can now remember when it first found a URL. That's enabled us to add a new tab to the projects called "Fresh". This lists the pages found in the last 7 days for a site. If you're the SEO for a site but not necessarily the content producer, or don't produce all of the content, then this is useful to see what new content has been placed on the site. It's also useful to watch competitors to see what new pages they are producing.

It's helpful to use this with the Scheduled Crawl feature to make sure the site is crawled every week. If it's a competitor you may want to set the crawl speed to "slow" in the project settings. We've also added an option to only crawl on-site pages which doesn't check if offsite links are broken or the relevance of offsite links, but it's handy for saving credits if you just want to monitor a competitor for basic information and fresh pages.

The first time a site is crawled everything will be "fresh", of course. But after a week it will show only pages found in the last 7 days.

Page Information

When you click on a link in any report on The Crawl Tool it has taken you to the page. Now we've introduced a step in between. If we have information on that page then it presents a "Page Information" page. You can still click the "Open Page" link to get to the page, or you can check out the wealth of information about the page, it's problems, search data, traffic.

We also use vector analysis from the crawls to find and show similar pages here. These are the pages you might want to consider linking to this page or linking from this page. It's a great opportunity to discover internal linking opportunities.

World's Smallest AI Dectector

We've talked about the world's smallest ai text detector in a blog post before. It's a tiny AI text detection model that can run in browser, meaning there's no costs to us to run it, which in turn means you can use it as much as you like. It may not be as accurate as its bigger brothers, but it's pretty good and certainly useful. Please check the previous blog post for the reasoning and what it tells us about various myths about AI text and SEO in general.