Over the easter weekend, I finally managed to get some more progress on the new LFG system I posted about recently. You can find a reasonably up to date state of it at:

https://fumbbl.com/p/lfg2

Yes, there are bugs. The most notable one is that clicking on the checkbox for a team doesn't do much at all (it actually processes the click twice). Instead, you can click on the team row itself and it should toggle properly. You'll also note that the table is not properly themed yet (no rounded corners, no drop shadow).

The reason that this is a blog and not an announcement is because I wanted to let you guys know a bit what's going on under the hood of that page, because it's pretty interesting technology.

The page is built using a number of application layers:

1. PHP page generator. This is the code that generates the page itself (theme, and the table wrapper). It's effectively an empty table and the controls at the bottom. This part is fairly straight-forward and doesn't really need much description. There's not much logic here at all.

2. Javascript. Those of you who are technical minded can view the javascript through your browsers. As with a lot of the javascript added to the site lately, the code heavily depends on the jQuery framework. jQuery simplifies things immensly for me as the developer, and adds lots of neat features. If you're a developer yourself, you'll notice that it's very imperative in nature and uses globally defined functions. Not very portable design, but I know very well what context it runs in so I didn't want to bother with the added complexity of isolating it more :). As you may notice, the code includes a lot of AJAX calls, which takes us to the next layer.

3. PHP proxy / facade. This code is what the Javascript calls to fetch data. It's fairly simple in nature, because its main purpose is to forward the request from the PHP engine to a Glassfish server (Java EE - Enterprise Edition) that's also running in the server park.

4. Java. This is a back-end server I've mentioned in an earlier blog. This is where the XMLs needed by the javascript front-end are generated. This is built on a pretty extensive framework I've designed to abstract the physical data model (ie, the data in the MySQL database). The framework handles caching of objects and allows mutual exclusion which PHP simply can't provide in a stable fashion.

The way the Java framework is implemented, I can do things like (a bit simplified):

Team team = Team.load(teamId); // Fetches a cached team instance, or loads it from the database
int treasury = (Integer) team.get(Team.TREASURY); // Fetch the team's treasury
team.set(Team.TREASURY, treasury + 10000); // Add 10k gold to the team

The set() method call doesn't actually persist the new treasury immediately, but instead marks the team as "dirty" (a concept that is common with caches) and schedules it for storing. In the framework, there's an asynchronous thread running which processes dirty objects and stores the updates to the database.

The major benefit of this approach is that I can push many changes to an object and only generate a single SQL update call. This results in a higher potential performance, and improves latency significantly. The cost of this approach is a risk of losing some non-persisted changes in case of failures, and that there's a second or two of lag between a change being pushed to it appearing in the database itself (this delay is not affecting end-users in any way).

Now, all this is well and good, but it also creates a bit of a problem. Most of the site is written directly in PHP, where the PHP reads information from the database directly. The problem appears when the PHP code updates an object that exists in the Java cache. Effectively, this means that the cached object will contain old information and the data from the Java layer will be out of date. This is visible in the new LFG page in its current state if you have a team that's not ready to play, load the LFG page (forcing your teams into the cache), ready your team and reload. The Java layer will still think that your team is not ready because the information is cached. Tricky.

There are a couple of possible approaches to getting around this issue:

1. Move all modifications into the Java layer. This is a massive project, and something I've been toying with the thought of. However, in the short term it's not feasible to do this.

2. Modify the Java framework to allow for certain "volatile" columns, which are always loaded from the database. This would most likely result in all data being volatile and therefore the caching would not be useful anymore. Not a very likely path.

3. Create an API in the Java engine that allows the PHP layer to flush individual objects out of the cache. This would allow me to notify the Java layer that an object needs to be reloaded because of a change. Most likely, this is what I'll end up doing because it's fairly simple to do and has a nice feel to it from a design perspective.


Finally, I have some venting I need to do:

* Checkboxes are annoying. If you look carefully at the "Select All" checkbox on that page, it's acting as a tri-state checkbox. It functions much like you'd expect, making it select all rows if there's at least one row that is not checked, and deselicting it if it's not. This is implemented pretty crudely in the current Javascript code, and I really need to redesign this into a more reusable fashion. I have a feeling I might want to use a similar control elsewhere (PM system for example).

* Dynamic vertical size with automatic scrollbars. Yeah, I don't think this is possible in CSS at all. I ended up with a javascript method that resizes the dialog as appropriate. I'm happy with the end result (try resizing your window when looking at that page assuming you have a fair number of teams and a screen that's large enough).

* People are using way too small screens. I linked this page on the IRC chat and instantly got complaints on how the rows are too large and the fonts are too big. I may end up having to implement a control that resizes the view for those of you who feel it's too big. The system was designed to be much larger than what you're used to in order to keep it simple. There are a lot of pages on the site which are very dense on information, which makes it complex and harder to learn. My intent with the new LFG system is to make it as easy as possible for the end user.


This turned out to be much longer than I thought. Thank you for your time!