Manual
The Fat Controller has had quite a few new features added to it recently, so I'm afraid the documentation is a bit behind, but I am doing my best to get it up-to-date and as complete as possible. If you need any immediate help then please file a support ticket and I'll reply as soon as I can.
- Configuration options
- Running
- Threads, processes and some confusion
- Thread models
- Exit statuses
- Fixed interval thread model
- Append thread ID
- Running in debug mode
- Test fire mode
- Multiple configurations
Configuration
This section describes all the options in the config file (/etc/fatcontroller). Each option corresponds to a parameter if you want to run The Fat Controller directly from the command line or a shell script, should be fairly easy to figure out what's what - if not then please let me know!
If you need some inspiration, then take a look at the examples on the Getting Started page.
APPLICATION_NAME="fatcontroller"
This, cunningly enough, is the name of the application. All syslog entries are prefixed with this, you probably won't ever need to change this unless you want to have multiple Fat Controllers running.
WORKING_DIRECTORY="/var/www/mywebapp/scripts/"
This is used if the application runs as a daemon, which it usually does. Once the Fat Controller starts up, it changes to this directory. Usually best to set it to the directory your script resides in as it will inherit this location as its working directory.
COMMAND="/usr/bin/php"
This executable with fully qualified path. The default is given for PHP but it can be anything.
ARGUMENTS="-f sleep.php"
You can specify arguments to the executable, here the -f parameter is used to tell PHP which script it should run. (Note for PHP users that if you want to specify arguments to the script, then you must prefix them with -- e.g. -f sleep.php -- foo=bar)
LOG_FILE="/var/log/${APPLICATION_NAME}.log"
This is where The Fat Controller will log the output of the scripts. Just make sure the file exists and The Fat Controller has write permissions to it.
ERR_LOG_FILE="/var/log/${APPLICATION_NAME}.err"
If this argument is not specified or is the same as LOG_FILE, then The Fat Controller will log all output (stdout and stderr) from sub-processes into the file specified by LOG_FILE. If this argument is specified then only stdout of sub-processes will be written into the file specified by LOG_FILE and stderr of sub-processes will be written into the file specified by this argument.
SLEEP=30
The behaviour of this parameter depends on the thread model you are using (see below for more information on thread models). When using dependent or independent thread models, the "sleep" parameter determines the number of seconds to wait from one instance ending with exit status 0 (i.e. ran without errors but there's nothing left to do - e.g. queue is empty) and starting the next instance. If a script ends with exit status 64 (i.e. more still to do, e.g. queue is not yet empty) then The Fat Controller will immediately start a new instance. See this section for more information on how sleep is used in the various thread models.
SLEEP_ON_ERROR=300
If a scripts ends with exit status -1 (i.e. an error occurred) then The Fat Controller will wait a longer time before running the script again. Perhaps the error was due to a temporary issue, such as too many connections to the database, or it maybe something more serious - either way, by having a large value here ensures that erroneous scripts aren't running all the time.
THREADS=1
This specifies the number of concurrent instances of the script that can run at a time. You will have to experiment to find the most effective value for your script as it will depend heavily on what it does. Think carefully about what resources the script uses and if their performance will benefit from multitasking.
THREAD_MODEL=DEPENDENT
This parameter determines the interaction between threads. Currently there are three thread models: DEPENDENT, INDEPENDENT and FIXED. See this section for more information on thread models.
PROC_RUN_TIME_WARN=3600
The Fat Controller will write a warning message into syslog if an instance of a script runs longer than the number of seconds specified in this parameter. the script will keep on running as normal, but you might like to know if something starts to look a bit fishy. A value of zero disables this feature.
PROC_RUN_TIME_MAX=0
Any instance of a script which runs longer than the specified number of seconds will be terminated by sending SIGTERM. Whether or not the script takes any notice of the SIGTERM signal is another matter and is addressed further below.
PROC_TERM_TIMEOUT=30
If an instance of a script ignores a SIGTERM signal, basically a friendly way to terminate a process, then it's time to get tough - unless of course you're happy to have unresponsive processes running amok! The Fat Controller will wait a specified number of seconds (specified by this parameter) before sending SIGKILL which tells the kernel directly to terminate the process - boom!
FIXED_INTERVAL_WAIT=-1
This is only used in the fixed-interval thread model and specifies the maximum number of seconds to wait after the point when a new instance should be started for a free thread should all available threads be busy. If after this period no thread is available then the oldest thread will be terminated and a new instance of the script can be run. A value of 0 means it will not wait and a value of -1 means it will always wait for a thread to finish normally.
APPEND_THREAD_ID=0
It might be useful for your scripts to know which thread they have been initiated by. In this case you can set this parameter to 1 and the ID of the thread will be appended to the command as a parameter. See the this section for more information.
PID_FILE="/tmp/${APPLICATION_NAME}.pid"
This file is used in daemon mode to log the process ID of The Fat Controller. On a production machine you'll probably put it somewhere like /var/run/fatcontroller.pid
APPLICATION="/usr/local/bin/fatcontroller"
This is the path to The Fat Controller - you won't have to change this unless you moved it after installation.
RUN_ONCE=0
By setting this argument to 1, it is possible to tell The Fat Controller to run a process only once and then end. Used for daemonising things.
Running
Once you have your configuration file ready, you can start and stop the Fat Controller using the following commands:
(Note that you must be logged in as root or precede these commands with sudo).
Starting
/etc/init.d/fatcontrollerd start
Stopping
/etc/init.d/fatcontrollerd stop
Is it running?
/etc/init.d/fatcontrollerd status
Threads, processes and some confusion
Throughout this documentation you will hear plenty of talk about "threads" - but is really meant is an instance of a PHP script, or a Python script or whatever - but that's a process, right? So why talk about "threads" if what we really mean are "processes"? The answer lies in the internal workings of the Fat Controller - whenever a new process (e.g. instance of a PHP script") is created, The Fat Controller creates a new thread which then creates a new process, in this case a new PHP process running the desired script. While the PHP process is running, the thread sits and waits for it to finish. By creating a new thread for each sub-process, the main thread in The Fat Controller can carry on as normal. So, as a new thread is created for each new sub-process, it's not so daft to talk about "threads".
(I perhaps should have chosen a better, less confusing term - but the project started out very much as an "internal" project at the company where I was working, so I didn't give much thought to naming things as only I really needed to understand it. Now it's a year or so down the line and changing things now would be quite a lengthy process.)
Thread models
- DEPENDENT - As the name implies, the threads are dependent on each other. The Fat Controller starts off by creating one thread which runs an instance of the script and if it exits with exit status 0, the thread then sleeps before starting a new instance. If a script exits with status 64 then The Fat Controller creates an additional thread. This can continue until the maximum number of threads has been reached (specified in the THREADS parameter). If a script exits with any other exit status then the number of threads is reduced back to one. This is very useful if you are processing a queue and need to increase processing capacity in busy times.
- INDEPENDENT - The Fat Controller starts immediately with the number of threads specified in the THREADS parameter. Each thread is independent, meanng that if a script instance in one script exits with -1 (i.e. failure), it will have no effect on the other threads. This is mainly useful in application mode when you know that at the beginng you will have a lot of work - see example use case.
- FIXED_INTERVAL - This thread model works just like CRON, creating a new thread which runs an instance of the script every x seconds (x in this case is specified by the SLEEP parameter). There are some important considerations when using this model as discussed here.
Exit statuses
Exit statuses are the way in which the scripts communicate with The Fat Controller and inform it of what's going on. Currently there are three states that scripts can report to The Fat Controller:
- 0 : OK
Indicates to The Fat Controller that the script executed without any problem. - 64 : OK, more items still to process
Indicates that when the script finished, there are still unprocessed items which need to be processed. - -1 : ERROR
Indicates that an error occurred inside the script.
The table below describes in more detail how exit statuses are handled in each of the thread models.
| Dependent | Independent | Fixed Interval | |
|---|---|---|---|
| 0 | Will then wait the specified "sleep" time before starting a new thread. It also indicates that there are no more items still to be processed and thus sets the allowed number of concurrently running threads to 1 - threads currently running are left to run as normal. | Will then wait the specified "sleep" time before restarting the thread. All other threads are unaffected. | Threads will continue to be run at the regular interval which is specified by the "sleep" parameter. |
| 64 | The thread which has just ended will be restarted. If The Fat Controller has not already reached its set limit of concurrent threads, it will increase the allowed number of concurrently running threads. | The thread which has just ended will be restarted. | No effect. |
| -1 | Will sleep for the specified "sleep on error" before starting a new thread. The allowed number of concurrently running threads to 1 - threads currently running are left to run as normal. | Will sleep for the specified "sleep on error" before restarting the thread. All other threads are unaffected. | Will sleep for the specified "sleep on error" before continuing running threads at the regular interval. |
Q: What's the point of "sleep on error"?
A: Sometimes errors are due to increased traffic so the "wait and try again" solution works. Sometimes errors are more serious and won't correct themselves, but in this situation at least The Fat Controller won't be constantly running scripts which create errors.
Fixed interval thread model
I will write this section as soon as I can. In the meantime, if you have any questions just create a support ticket
Append thread ID
I will write this section as soon as I can. In the meantime, if you have any questions just create a support ticket
Running in debug mode
Debug mode prints a ton more information into your syslog. Definitely not recommended for production, but if you are having a hard time diagnosing a problem, then the extra log entries created by debug mode may help.
Debug mode is invoked when The Fat Controller starts up:
sudo /etc/init.d/fatcontrollerd debug
Test fire mode
Thanks to a great idea in a comment on the blog this argument has been added. If this argument is specified then The Fat Controller initialises but does not actually run, i.e. it does not daemonise (if it is specified to be in daemonise mode) nor does it run any processes. It is useful combined with the --debug option when testing to check arguments have been correctly read and interpreted.
Multiple configurations
One of the most common questions I am asked is "how do I set up multiple configurations with the Fat Controller?". The solution is to run multiple instances of the Fat Controller - each with its own configuration file. Here's a recipe to get started:
Start by making a copy of the default Fat Controller config file:
sudo cp /etc/fatcontroller /etc/fatcontroller_job1 sudo cp /etc/fatcontroller /etc/fatcontroller_job2
Now you can edit these new config files to specify the correct scripts and parameters.
To start and stop them, you specify the configuration file as the second parameter to the init script, for example:
sudo /etc/init.d/fatcontrollerd start /etc/fatcontroller_job1 sudo /etc/init.d/fatcontrollerd stop /etc/fatcontroller_job1 sudo /etc/init.d/fatcontrollerd status /etc/fatcontroller_job1
If the second parameter isn't specified then The Fat Controller uses the default config file (/etc/fatcontroller) as usual.