You've set up your website, worked on your cookie and privacy policies, added a cookie permissions script, and you're all set and compliant with cookie regulations right?

European regulation, in particular, is strict and sites and tools often tell you they are GDPR compliant when they aren't. A common thought amongst web analytics tools, for example, is that if they just store identifications in a local database then it's compliant. Not true.

The same temptation occurs with cookies. If you just add a cookie permission script then are you not pushing cookies anymore?

Many sites use, for example, a CDN like Cloudflare to speed up their site. It might surprise you to learn, as it did us, that CDNs like Cloudflare can add cookies to a user's browser that you might be unaware of. These cookies are to enable various Cloudflare features and services, which require tracking of users. Under EU law, these should show in your cookie policy.

But because, in this case, it's your CDN that's doing it - these cookies have no awareness of whether a visitor has agreed to them or not. That's just one example, there could be other cookies sailing past your cookie permissions script without you knowing.

When The Crawl Tool crawls a site it starts by not knowing any cookies. You can think of it as a browser that has just been installed or cleared to its default state without cookies, that then visits every page on the site.

To help with the problem we've just described, one feature of The Crawl Tool is that we store a list of each cookie that something tries to install into the browser's cookies during our crawl. The crawler doesn't agree to cookie permissions, so this is a list of cookies that are being pushed anyway.

The Cookies Report in The Crawl Tool them shows these.

In some cases, such as XSRF tokens (for security) and session cookies for operation, we can consider these essential cookies for site operations. But in an ideal world this would be empty. In any case it allows you to look at the cookie names, the data pushed, for what domains and to decide if these are essential cookies that should've been set or not. If they shouldn't then the domain column should give you information about where to start looking if it is an external script that has caused this. The Offsite JS Scripts Report may useful in combination with this to isolate which script.