How To Keep a Thematic Focus To Your Site
It's no secret that sites that tend to do well are those focused around a particular theme. But how can you tell when pages are going off theme? With The Crawl Tool Theme Report.
But how do you know the theme?
At The Crawl Tool we started using an AI model to create a representation of the theme of each page. We then use those to create a representation of the theme of a site.
Because they're AI representations, they're not really human interpretable - so we can't say, for example, that a site is about trucks. But what we can determine is how different the theme representation of a page is from another page, or indeed from the site's theme representation.
We can measure that on a scale that goes up to 1, where 1 means the theme representation perfectly matches (which means it is a virtual duplicate).
How to use the theme representation?
Firstly go into the Theme Report in The Crawl Tool. Then sort by "Relevance" by clicking on the column title. You'll want to sort this so the lowest values are first.
What this means is that the top items are the ones least related to the website's theme. If remember the top relevance score and scroll to the bottom and look at the last one, you can get an idea of the spread - or how focused on the topic your site is. This will help you make a judgement of what level of relevance you're willing to accept. Our general observation is that in practice anything less than a 0.8 can do with some improvement.
You might see some pages with lower scores that you think are relevant. What this means is that there is an opportunity to go to those pages and look at the wording. How does it describe itself and how can it describe itself better to represent the theme of the site. There are, perhaps, low scoring ones that you agree aren't relevant. Again this is an opportunity to update them, adjust the wording, and perhaps focus on some new keywords.
In general a low score should be viewed as an opportunity to improve the site by editing the page and getting a tighter site focus on a theme. You may see some pages in there that are part of the functionality of the site - like a login page - you don't need to worry about them, it's not the end of the world. But if you can improve them then go ahead - why not?
