Use cases

Sending e-mails in the background

The Fat Controller was used to regularly run a PHP script which sends emails queued in a database table, dynamically increasing and decreasing the number of these scripts running in parallel in accordance with the backlog of unset emails. As a result, sending time was greatly reduced.

Whenever a user sends an email in our PHP web application, for example after registration the system sends an email with the user's login details, instead of connecting to an SMTP server and sending an email directly, it creates a new row in the table "emails" in the database.

At regular intervals, CRON executes a PHP script which takes the next 10 unsent emails in the database table, connects to the SMTP server and sends the emails.

This serves two purposes; two decouple the frontend from the SMTP server from the frontend application so as any communication delay won't affect the response time for the user. Secondly, it provides a chronological log of all the emails sent by the system - the marketing department likes this.

This was all well and good until it was decided that we should send our users a newsletter twice a week. We created a system that diligently sent all our users an email and thus filled the "emails" table with over 60,000 new rows. At a rate of 10 emails a minute, sending the newsletter was going to take some time.

The obvious solution was to increase the number of emails sent each time the script was run, which we did, until we got to the point where the script was taking over a minute to run. "What's wrong with that?" you might be wondering. Well, if you're not careful, you can end up with lots of scripts running at the same time, which load the database which in turn causes each script to take longer, which causes more scripts to run, and so on. There needs to be a way to manage this without causing this kind of positive feedback.

To solve this problem, we used The Fat Controller to run the script every 30s. The script communicates with The Fat Controller by exiting with different exit statuses (http://en.wikipedia.org/wiki/Exit_status). If the script returned an exit status of 0 (i.e. execution completed ok), The Fat Controller waited another 30s before running the script again. The script is able to tell The Fat Controller that there are still more unsent items in the email queue by exiting with a special exit status of 64. The Fat Controller then immediately restarts the script plus an additional instance of the script. In this way, the number of instances running in parallel is gradually increased up to a limit (we used 4, but it depends on what your system can handle). If a script realises that there are no more items left in the queue, it returns an exit status of 0 and The Fat Controller won't run the script again for another 30s, and when it does, only one instance will be run. Of course all the scripts that are still running are left to finish as normal.

Using The Fat Controller we were able to cut the time needed to send the newsletter from several days to a few hours.

Generating newsletters

The Fat Controller was used as an application to perform one-off tasks of generating newsletters. The generation was done in batches of 100 users and The Fat Controller was used to process 6 batches at any one time.

After successfully implementing a newsletter system that sent all our users the same email (see use case 1), it was then decided to sent personalised emails along with content tailored to what each user had recently done on the site. So, rather than just generating one block of HTML and inserting it into the email queue table for all our users, we had to generate a new block of HTML derived from a whole load of other statistics that was gathered from the database for each user. As such, two problems arose.

PHP, amongst other scripting languages, whilst being great for generating dynamic HTML on a request-by-request basis is terrible for anything that has to run for any length of time. For starters, PHP can throw fatal errors with little or no warning and prior to PHP 5.3 has problem with memory management (http://www.php.net/manual/en/features.gc.performance-considerations.php). Needless to say, generating all 60,000+ emails in one go was out of the question - we needed to generate emails in batches that can safely be restarted if there's a problem.

The second problem was time. Generating all 60,000+ emails one after the other was also not realistic.

The solution was to create a script that generated newsletters for the next 500 users for whom a newsletter had not already been created. The Fat Controller was used to run 6 instances of this script, restarting each script once it finishes to maintain a pool of 6. Unlike in use case 1, The Fat Controller was run as a normal application, not a background daemon, which means we start it manually from the command line and see all the output of the child script processes. Once a newsletter has been generated for all the users, the scripts exit with exit status 0 and we can stop The Fat Controller (CTRL+C).

As The Fat Controller is running in application mode (not daemon mode), once a scripts returns that there is nothing more to do, the thread is put to sleep. Once all threads are sleeping, The Fat Controller exits. As such it was possible to create a simple one-line shell script that runs The Fat Controller and execute this from CRON once a day to send all the newsletters.

Regular processing of event on a social network

The Fat Controller is also useful for repeatedly running a script at regular intervals. Unlike CRON which runs a script every x minutes, The Fat Controller runs a script again x minutes after the previous one ended so there is never a risk of more than one script running at a time. Of course you could always implement your own locking mechanism to ensure that only one instance of the script will ever run, but The Fat Controller, being a kind sort of chap, saves you having to roll your own.

Daemonize a Java application

Unlike in C, there is no native way to daemonize a Java application. The Fat Controller can daemonise anything and can therefore easily turn any Java application into a systerm daemon that can be controlled using signals. As such any Java application can be made to start when the system boots, run in the background and then shutdown when the system itself shuts down.

The Fat Controller can function as a simple version of Apache Daemon and without the need to implement a special API in Java. If you don't need the extra complexity afforded by Apache Daemon then perhaps consider The Fat Controller.

The Fat Controller © Nick Giles
Web design © 1996 - 2008, Seznam.cz, a.s.