Boards.ie blog

We’d like to talk to other people running large vBulletin forums to discuss scaling and other growth planning.

To give you an idea of what I mean by large, Boards.ie currently pushes out 45-50 page requests per second (threads and posts, not images or static files), has more than 5,000 users online at peak every day (and growing), 120 posts every five minutes at peak and averages about 500 queries per second across the three production DB servers (discounting replication).
There are more than 1.3 million threads, nearly 19 million posts, and over 300,000 registered accounts. We serve more than 30 million pageviews and 1.6 million unique visitors per month (Google Analytics).

We also had a post last week with a (high level) overview of our current hardware.

If these numbers look familiar, we’d like to talk, and hope you’d like to also! We’d like to know what sort of changes you’ve made to improve performance, what you’ve cut out or disabled (or added) and we’ll be happy to share our insights too.

Drop me a mail ross@boards.ie if you’d like to get in touch.

Boards.ie has, more or less, expanded one server at a time. Beginning life on a single shared server, spending a stint under the desk of one of our founders, then the big move to two separate servers, and more as demand increased. Eventually, by the start of this year, we had sprawled up to 18 racked machines, some of which were non-operational, some which we couldn’t use properly due to the organic nature of how all that machinery had come together.

One major step for us has been consolidating that hardware. We’d been spread across two cabinets in Digiweb’s data-centre for several months, the hack in January set off a series of delays that kept us away from the hardware longer than we would have liked. Over the last month we’ve taken several trips out to the data centre and completed some important stages.

Previously, our hardware configuration looked something like this (this is what a decade’s worth of slow growth looks like):

Cabinet 2009, before changes. Internet notably on right.

Problems

In this configuration, we had servers which were using other servers as shelves, literally stacked on top of each other. Obviously this made any physical maintenance quite tedious. The back of the cabinet (not shown here) was a jungle of criss-crossing cables, some of which weren’t even connected at one end, but were so tangled up with others that removing them presented more risk than the reward was worth. We were also sharing the cabinet with several other customers.

The method of load-balancing we were using was also fairly basic, blindly spinning requests off to back ends using one of our oldest surviving machines (a dual core Pentium III). If one web server dropped offline for any reason, 20% of page requests would simply fail. If the load balancer dropped offline, all requests would fail. Additionally, recovery from a failure of the load balancer was not trivial. We were also having increasing trouble from poorly written web spiders and malicious spambots who would occasionally drive the site to a crawl.

While it had rarely been an issue, our static content (images, mostly) was also being served from a single machine.

Also, physical space was starting to become an issue; we were quickly going to have problems fitting more hardware in the same place as the rest, and our internal switch was down to a couple of free ports. Our growth curve is relatively predictable and we knew we’d need more capacity within about 3-4 months.

Solving

We started with the problem we knew was coming, capacity. Digiweb gave us our own cabinet and hooked it up to the old cabinet so that we could start adding more hardware. We decided to make sure we Did It Right with the new cabinet, and set about making sure everything would be accounted for in the layout of the machines, right down to trying to be consistent about cable colours.

We bought a pair of machines to serve as replacements for our ailing P3 load balancer, a couple of beefy (8 core, 32GB RAM) ones to serve as new database machines and a pair of new routers.

New servers, boxed

Over a couple of trips, we moved all of our machines from the old cabinet to the new one, retiring the older machines where possible, and getting rails for anything that was freestanding. We even managed to keep downtime relatively low, though moving the primary database server meant we had to drop offline – the vBulletin software that Boards.ie runs on doesn’t give us the option for a “read only” mode without a significant overhaul.

Old cabinet, midway through move

Our new load balancers have been set up to mirror each other and perform automatic failover in the event of one failing (using CARP and pfsync). This has also allowed us to have a much more robust firewall and a redundant internal DNS cache (djbdns). We’re using varnish to do the actual load balancing, as well as taking care of dead machines at the back end.

We’ve also implemented multiple database support in vBulletin using memcached counters for weighting, which means we can easily add more machines to our database pool where required and balance the traffic.

Here’s a rough diagram of our infrastructure as it is today:

Infrastructure diagram 2010 - moving the Internet to the left was the hardest part.

And some photos!

Web servers

DB and storage

Back of DB / Storage

Back of web servers