The Crawl Tool Blog

Why we need our pages faster

Unless you've been hiding under a rock, you'll know that speed is an important part of user experience and SEO (it's one of the few things that Google is unambiguous about). But as a small business trying to keep costs and prices low that's a difficult challenge. Our site is custom coded and whilst we can code it to be as fast as possible, we soon hit the limit of infrastructure which is bound to a single point in the world - where the web server is. The issue then is that site is fastest for those near the server but slow for those further away. If you're focus is a particular geographic area then having your web servers there is a satisfactory solution, but if your visitors are world wide then you can't fix it for everyone that way. If your site is optimized then the problem is simply the distance the data must travel.

Enter Edge Caching

Edge caching is interesting because it tends to be well known amongst smaller sites and not known about enough amongst mid-level sites. Only small sites know it as CDN. A number of infrastructure companies have servers worldwide - for example Microsoft or Amazon. Theoretically you could then run your website in various locations around the globe and be closer to every single user. You just need to run 130 copies of your website!

But some things don't change very much on your website. With The Crawl Tool, for example, the things behind the scenes when you've logged in change but the things before log in like the home page, the blog, and the privacy policy change much less. There can be tens of thousands, or more, requests before they change.

Rather than having 130 web servers generate the pages each time, wouldn't it be great is we could just have one web server generate those pages every so often and store it in 130 locations around the world. Users can then always get given a version that's close to them and those copies can update at a rate that means they don't miss information but also don't stress the server. That's edge caching!

Enter Cloudflare

Cloudflare has locations in 335 cities around the world and offers caching services. To be honest, they offer so much that it's hard to keep track of. But the basic core-offering is edge caching but another useful feature is a web application firewall to keep the bad guys out. There are low cost basic offerings that are suitable for many sites.

The Crawl Tool sits in a market where we need to compete with some big boys though. Our market may be small to medium websites at an affordable price point, but that's no excuse for us to think small. Competing with the big boys is difficult though, but we're pleased to say that Cloudflare have given us an enterprise subscription and $5000 free credits under their startup program. So big boys, watch out!

Getting Faster

We've been working hard on changes. At the moment these changes mostly take place before login, but we'll be reviewing the after login and API experience in due course too.

A major focus has been images. The temptation with modern internet connections is to forget that users are sometimes on slower or poor connections and simply resize larger images in browser. If you always pick a larger image than necessary then it always resizes nicely to fit with modern CSS. That's a fair visual experience idea but it's not a great experience for those users. So the first thing we did was auto-resize images down to a minimum necessary size - every image is no bigger than it needs to be.

Secondly, here and there, we've made pages before login a bit less dynamic. Most of the time these are obvious things to do, sometimes they take a lot more thinking about. For example, we've removed the login form from the home page and there's just a login button. At first thought, adding an extra click to the login isn't really bad but it doesn't feel like an improvement to the user experience.

However, after a crawl (often a scheduled crawl) we send our users a website report email and a lot of logins are click throughs from that. Similarly users will tend to bookmark a logged in page. If you try to visit those links and are logged in you go straight through and if not you go to the login page. It's not so common for users to actually log in on the front page.

Meanwhile the functionality for logging in was responsible for about half the page speed, or slowness in the way we're looking at it. Having a login button rather than a slower form means we're providing the slower functionality only when it's needed and giving an overall improvement in UX and performance to everyone.

Cloudflare Features

It's worth us going through the features we're using in Cloudflare as combined with the changes above our page speeds (before login) are half or better than they were previously. Though we have the budget, we're not yet using any additional paid features.

We set up custom caching rules for every page before login that can stay the same for a period of time. Sometimes this is groups of pages (like blog posts). This dials in the amount of time we think they can stay in cache without having to update. The difference between being in cache or not is around 120ms for us. That sounds like a small amount of time, but in terms of page speed it's significant. So optimizing those cache times is worthwhile for us.

We are using both Polish and Mirage for images. Polish tries to reduce image file sizes and Mirage aims to tailor image to devices.

We use literally every Cloudflare "protocol optimization". In fact we use every recommended optimization everywhere except "Rocket Loader" which caused problems for us.

Which leads us to the suggestion if you do the same. Our approach was to start with a single Cache Rule that says don't cache anything. Then add rules in and test the site with each addition. The same for the recommend optimizations. Turn one on, test the site. Only if it works should you turn the next one on. It's better to follow a slow and simple process with lots of Purging the cache and testing than it is to try to hunt down which one is breaking something later.

Final Thoughts

We're grateful to Cloudflare for providing these services to us for free under their startup program. Together with the Amazon credits we received, and (we've not yet announced) Azure one, we're gaining recognition and capability to act like the big boys. We've got more work to do to apply this to the logged in experience, but it's already bringing great benefits to the site.

It's also made us think. We've recently released a plugin to automatically do SEO headers in Wordpress. Why aren't images better catered for? Perhaps, just maybe, we'll do something for that. A lot of our small to medium website users are Wordpress and could benefit.