The Crawl Tool Blog

It's the second of our new weekly videos showcasing the improvements to The Crawl Tool in the last week. It seemed like there were a million things in the first update, and only two in this second one. But they are big things.

Webpage Design

The crawling and reporting aspects of The Crawl Tool are the exciting part and so they've been the focus of most of the work. This left things like the home page, the guide, and the blog, looking a bit dated. That's more noticeable than before since the beginning of these weekly posts, that weren't that easy to read.

So all those pages have had a fresh, new, more modern, and clean redesign that will make everything easier to see, read, and digest.

Backlinks

A clear request from users is for backlink information. There's lots of ways we can use that - from a simple report that you can search or export if you wish, to backlink info by page on the backlink info pages, to a backlink research tool. We've been unable to do these because access to backlink databases is expensive and it would compromise our low cost approach.

Until now...we've started building our own backlinks database. This database building process will last a couple of weeks more before we can start adding the features we just mentioned. Except for one - we currently have a partial database. When you crawl a website it'll query that partial database and fill in a "Back Links" report for you on the dropdown. Until we've built the database, it's just a fraction of the backlink information that we'll be able to provide users with but we figured we may as well give what we have now.

We're considering opening an API to this data and are interested in hearing any ideas for potential use cases, requirements, etc, to sound out if that's a worthwhile effort.