As a lot of you have been noticing, the site has recently been sluggish every now and again, mostly during peak hours (somewhere between 8pm and midnight server time). To give you an idea of how complex this stuff is, I'll try to explain the sequence of events and what I believe is the cause (and fix) for the problems.

First off, some background.

In the ancient beginning (roughly 10 years ago), FUMBBL ran on a single server with fairly unoptimized code and database queries. This was all well and good since the number of users was small enough to not stress the system. Over time, our user base grew and I went through a sequence of upgrading the hardware to cope with the increased traffic. When the site hardware was extensive enough, and our user base continued to increase, I ventured into the long and ardous process of optimizing the code and database queries to keep the site responding quickly enough to not be stressful to end users.

Back in 2008 and 2009 FUMBBL was responding to requests from roughly 200 coaches online at peak times. Things were looking good. In mid 2009 Cyanide's game hit the stand and GW sent us the infamous Cease and Desist letter, and a lot of people were expecting us to completely shut down. It's also important to realize that we were also stuck with the LRB 4 ruleset at the time and development of the client we used had effectively halted.

FUMBBL did take a significant hit in terms of userbase from this, but we proved to be more resilient than what many people thought. At the lowest point, we had a third of the traffic that we used to.

Since then, we've doubled up on effort with a lot of work on improving the website and integrating a new client (thanks Kalimar!). Since the lowest point, we've managed to increase traffic by 50%.

Now, despite the site handling a lot more users back in early 2009, I've added a bunch of new features that increase the number of website requests which leads me to the actual problems as of late.

Today, FUMBBL handles on the order of 2 million page loads per month (which averages out to give or take 1 request per second). Each page load involves loading the HTML page itself, plus each asset that is part of the site (images, stylesheets, scripts). With a conservative average, let's say that there are 9 assets like this and we're looking at 10 website "requests" per second. Now, this is on average over a full 30 day period and this isn't exactly hard to deal with for the servers.

The problem, however, is that the distribution of the traffic is very different from the average. At the peak times over the last couple of days, there has been on the order of 150 coaches online at the same time. This does not include people who simply access the site and (for example) read the forums without being logged in.

Before I go in to the details of what this means, let's back up a bit to the problems we've had lately. The way things progressed from my perspective is as follows:

1. Over the last couple of weeks, the website has been intermittently slow. Normally, I don't personally have any significant latency on loading pages and things are in general practically instant (since I have a practically dedicated gigabit network to the servers). Now and again, however, pages seemed to take a second or two to load. Because of the rarity of this, I didn't really think much about it.

2. A couple of days ago, I was notified that the game server went down. Ok, nothing really exceptional here. The FFB server does have a few bugs, and occasional crashes happens from time to time. I restarted the server and all was well.

3. The day after, the game server crashes again. Luckily, I'm at home since it's in the evening. People also complain that the site is very very slow. I confirm this, and pages are taking 20 seconds to load for me which is just completely unacceptable. Ok, so something is wrong, but what?

4. I start looking into the state of the servers.
- The web server is running at 5% CPU use. There is no memory swapping and no significant hard drive activity. The network is also not congested with traffic. hmm..
- So, let's check the database. low CPU use, no disk activity, no flood of connections to the database, no network issues. hmm again..

5. Some time passes, because I can't figure out what's wrong.

6. Then, it dawns on me: I had a similar problem back in late 2007/early 2008 (or so.. that was a while ago).

Now, this is where we come back to the story. Remember the 10 requests per second on average? With 150 people on the site at once, and all sorts of new features that keep making website requests, I realize that the web server simply can't process all the requests without running out of connections.

"Being online" on the web is a bit complicated since there's no persistent connection to the site as such (I'll skip the details on this since this blog is long enough as it is). For site purposes, being online means loading a page within the last 10 minutes. During these 10 minutes, an average coach loads on the order of 25 pages, which ends up as as 0.4 requests per second. Multiply this with 150 and you have 75 requests per second to deal with.

To make this almost impossible to understand, though, there are a number of optimizations done by both browsers and the web server. First off, the browser wants to be as quick as possible, so it opens up multiple connections to the server to download assets (pages, images, etc). It also caches things so it won't have to download the same thing multiple times for no reason. This effectively reduces the number of connections a browser opens to the server.

The server does things to improve speed. One of the major features is that it allows a browser to download multiple assets through a single connection, which again reduces the number of connections needed per browser and page. Ok, so far so good.. So what's the problem?

In order to make this actually work in practice, the server has to open a connection when the browser asks for it, process the first request and then wait for some specified amount of time for the next request to come down the line. This wait time has to be long enough to allow the browser to receive and process the previous request and then possibly re-request a new asset.

On the surface, this is all well and good, but this also means that the connection must remain open for a number of seconds. This time is normally defaulted to 15 seconds on the server. Remember how I said browsers open multiple connections to the server to increase speed? Firefox, for example, starts up to 6 connections like this for each page load.

Each connection takes up a fair amount of resources (on the order of 25MB of RAM on the server), so there is a configured limit to the number of these connections (more on that later).

With 150 users on the site at the same time, the server has to deal with a pretty large number of connections. Empirical tests (counting the number of processes running and comparing to the number of people online), the web server has 1 connection open for each user at any one time, plus a set of spare servers to deal with spikes in connections.

Simplified, the FUMBBL site was configured as follows before today:

- A maximum of 150 connections open
- Connection timeout at 15 seconds
- 10-20 "spare" processes waiting for connections

Now, with 150 people on the site at once, it's not that hard to realize that the 150 connections are not enough. If all of these people open up a connection to the site, the server can't process more users at all, and any further requests will have to wait. Remember that there are also non-logged-in users accessing the site and we're likely looking at 200+ people.

And since the browsers will try to open a second connection sometimes, the browsers will retain the first one for as long as possible (15 seconds), and the site will slow to a crawl for basically everyone, every time.

Ok, so realizing this, I have now configured the server to the following:

- A maximum of 512 connections open (meaning a theoretical max memory use of 12GB)
- Connection timeout at 10 seconds (this can't be too low, in order to keep things working properly)
- 40-100 spare processes.

Assuming that this is indeed the reason for the slowdown, this change should improve things immensly and the site should be responsive even during peak hours again. I will continue to monitor the situation and am hoping that this will, indeed, fix things. If it doesn't, I honestly don't know what could cause this issue. We'll just have to see.

Thanks for taking the time to read this massive wall of text, and I hope you understood at least parts of it :)

You've spent enough time reading now, go find a game to play and have fun!