Tracking Hits to robots.txt, sitemap.xml, and llms.txt for free
What are robots.txt, sitemap.xml, and llms.txt These files are used by search engines and bots to discover content and to le...
Expert insights, guides, and tips to improve your website's SEO performance
Not everyone likes, or needs, to do a deep spreadsheet style analysis every time they check their site with The Crawl Tool. For this reason a lot of work has been put into improving the project dashboards.
The idea of the project dashboard is to put the most important summary information in an easy to view and read report. This should save time for those times when you just want to get an idea of the current state of the site.
We're calling this version 1 of the new Projects Dashboard as we've still got plenty of ideas and enhancements we'd like to do to it. But it's so useful we've made the first version live now.
If you have any suggestions, get in touch: [email protected]
Start with a free crawl of up to 1,000 URLs and get actionable insights today.
Try The Crawl Tool FreeWhat are robots.txt, sitemap.xml, and llms.txt These files are used by search engines and bots to discover content and to le...
AI Crawlers and Citing Sources The rise of AI, rather than search, crawlers visiting websites and "indexing" information is ...
Why we need our pages faster Unless you've been hiding under a rock, you'll know that speed is an important part of user exp...
What is it? AI is getting better and better and increasing AI systems like Google's Gemma 3 can be used for free. Whilst I m...
What Problem is Being Solved Here? In the SEO world there's a lot of discussion about whether Search Engines like Google det...
How Backlinks Apply to SEO There's no question that backlinks are important in SEO. If you've read anything on the subject t...