Over the last couple of weeks, I've been doing some work on my development environment. Looking back over my blog entries, the last time I talked about how I do the dev work on the site was in 2012. That's quite a long time ago, so I figured I'd write an update.

First off, the reason for my recent work: Basically, my old iMac (late 2012 model) was starting to show strong signs of deterioration. Effectively, the built-in display is suffering from pretty bad image retention, where any bright pixels showing for a few minutes temporarily "sticks" to the panel and shows up as ghostly shapes on top of anything else showing. This was slowly getting worse and I was hoping for Apple to release a suitable machine to replace it with. Eventually, I simply got tired of waiting and decided to make the switch from MacOS to Windows.

Now, this wasn't an easy decision given that I have a fair amount of commercial tools and applications as part of my kit that only run on MacOS. Anyway, with the decision made, I got started.

Back in 2012, I was using Eclipse as my primary development environment. That has not been accurate for.. well years. While Eclipse is a reasonable IDE, it's very very slow to start, and a bit sluggish during use. Also, I've been intending (and actually working on) transitioning away from the Java back-end which makes the whole Eclipse platform redundant. Also, I got tired of the complicated update processes I ended up having to go through every now and again.

Instead, I switched to Sublime Text, a light-weight but absolutely incredibly powerful editor for the majority of the site work I do (php, javascript, typescript, css, less) and occasionally run Visual Studio Code for updates to develop the replacement back-end services in Node.js (also typescript). Sublime is relatively expensive for a text editor, but I truly feel it's worth the cost (and the license allows me to run it on any machine I am the primary user of, so no problem there).

Now, some of the setup is similar to what it was back in 2012. I still have a local copy of the website code on my machine (now a Windows 10 Pro system), and a share on the web server that hosts the development version of the site. Saving a file to the dev version has a direct effect on what's published on that version (only accessable by whitelisted IPs, so not anything public). More on this save process later, because that's sort of the primary thing I want to cover for this blog :)

The code is also connected to a version control system (Mercurial), and when I do a push to the host, it automatically synchronizes the code to the live site. This means that once I have fixed a bug, or added a feature, actually doing the technical testing on the dev site and deploying on live is truly quick and easy.

Now, the save process I mentioned above. This is much more complicated, and has been sort of a pain point for me in this transition. In principle, all I need to do is copy a file from my machine to the web server to make a deploy to either test or live. In practise, it's significantly more complex (as it often is in life).

First off, I run development code in multiple languages. PHP is pretty easy, as that's truly a copy/paste process to deploy. Get the file over to the web server, reload the page and there it is. No problem. CSS and JavaScript are in theory simple, but I'll cover a complexity about these a bit more later.

Typescript and LESS are two languages that are a bit special. They are both serve a principally similar purpose, which is to improve on existing technologies. Typescript is a superset of JavaScript, which adds a number of useful features that are not there in raw JavaScript (strict typing, modern language constructs, imports, etc) and you compile Typescript to JavaScript to make it work on the web. LESS, in a similar fashion, adds things on top of CSS (variables, simple math functions, nesting) and is similarly compiled to CSS. Note the word compile for both of these languages. This means that there's an extra step in the process: Save -> Compile -> Deploy.

On my MacOS setup, I was using a commercial third-party tool called CodeKit. What CodeKit did was to sit in the background and look at a bunch of files. When I saved a .ts (Typescript) or .less file, it would trigger the compile step automatically, and I could then copy the resulting .js (JavaScript) or .css file to the dev site (and push to the mercurial repository for live deployment).

To give you a bit more insight into how this all worked (as it is relevant later), Sublime Text has a feature it calls Build Systems. This allows you to press a key binding (ctrl-b) to run a "build script". This is simply a set of instructions to the machine to do some task. This can be to run a shell script (.bat file on Windows or a shell script on Linux/Mac), or execute a Python (another programming language) script which is the native way of doing things.

What I did in my build script on the Mac for Typescript and LESS was to wait for 2 seconds to give CodeKit time to do the compile, followed by copying the resulting .js or .css to the dev site. Not 100% ideal, and got a bit wonky sometimes (because CodeKit didn't make it in time) but it worked well enough and I was happy.

CodeKit was a really major part of the development process for me. Unfortunately, they don't have a Windows version. I looked around for equivalent tools for Windows and found something called Prepros. It looked very promising, but in the end it became obvious that it wasn't even close in terms of features, and it was acting super slow on the FUMBBL code (there are thousands of files on FUMBBL) for certain tasks. Not good.

So, I kept looking for ways to replace what CodeKit did for me. To make a long story short, I ended up with what the developers among you are probably familiar with: Webpack. Webpack does *most* of what I used to do with CodeKit, but I did end up also including another relatively well-known tool: Gulp. Both Webpack and Gulp have similar purposes: They process code and run it through a number of different stages and plugins to generate some form of resulting output.

Without going into too much detail about this, this is a rough rundown of what I have ended up with for my build and deploy process:

When I press ctrl-b on a file in sublime text, it triggers a custom Python build script. This script first detects the type of file I am trying to build. Depending on the type of file, I launch one of four different processes.

The simplest (and default) one is a simple "copy this file to the dev environment". This is used for PHP files for example, which don't need any other processing.

JavaScript (.js) files are forwarded to Gulp, which has a custom handler for these files. The chain of plugins used in this Gulp script starts with something called gulp-imports (replacing a feature from CodeKit). Simplified, it allows me to concatenate multiple javascript files into a single one (for you dev-types, this is without any type of module resolution in place). After this, the next plugin is called "uglify", which makes the JavaScript smaller in size to make it load faster (and make it harder to read). Finally, it's run through a plugin called gulp-rev. This is a new feature I've introduced to my dev chain to do what's called "cache-busting". I won't get further into this here, but let me know if you're interested and I'll make a separate blog about that. Once this is done, the sublime build script continues and copies the resulting output from Gulp to the dev environment.

LESS (.less) files are treated similarly to above. They're passed to Gulp, which runs the less compiler, then an uglify equivalent called "clean-css" followed by the same cache-busting gulp-rev tool as above. The sublime build script then deploys to the dev site as per normal.

CSS (.css) files are going to be treated similarly as above, but without the less compile step. At the time of writing, this hasn't been done just yet but isn't a big task to do.

TypeScript (.ts) files are the most complex ones. They are first sent through Webpack which compiles the files to javascript, and then they're sent on to gulp for cache-busting as above. What makes this more complicated, though, is that TypeScript by its nature is designed to have multiple files. I have a number of TypeScript files which contain utility functions, which aren't necessary to build and deploy on their own and are instead designed to assist other typescript files to do their work. So I have a process that automatically detects which scripts use the other helper scripts, and if I make a change in one of these underlying scripts, all files that depend on them will be automatically processed. This process took quite a few days to get right and working, but it's been worth the trip. The current setup I have does everything CodeKit did, but better. I don't have to have a "hacky" wait for 2 seconds thing going but instead simply wait for things to be compiled and act once it's been done.

With this updated setup, I feel that I have good control over my build process without any kind of "black boxes" that do magic things for me. It's extremely stream-lined and quick to work with and that makes me happy :)

Note that Java isn't part of this description, and I still run the Java back-end for certain important things (gamefinder and blackbox scheduler primarily). These are things I intend to convert to Node.js instead; I simply don't want to spend more time working on the Java backend. It taking more time to open the Eclipse IDE than it takes me to make a fix and deploy to test and dev is simply unacceptable to me.

I'm sure there are parts of this I've forgotten (and I've actively avoided talking about some of the other commercial tools I have on the Mac which I have no equivalent for on Windows yet).. It's been a long blog though, and I hope at least some of you have found this interesting.