[TL;DR: To supplement Heroku-managed app servers, we launched custom EC2 instances to host Delayed Job worker processes. See the satellite_setup github repo for rake tasks and Chef recipes that make it easy.]
Artsy engineers are big users and abusers of Heroku. It's a neat abstraction of server resources, so we were conflicted when parts of our application started to bump into Heroku's limitations. While we weren't eager to start managing additional infrastructure, we found that--with a few good tools--we could migrate some components away from Heroku without fragmenting the codebase or over-complicating our development environments.
There are a number of reasons your app might need to go beyond Heroku. It might rely on a locally installed tool (not possible on Heroku's locked-down servers), or require heavy file-system usage (limited to
log/, and not permanent or shared). In our case, the culprit was Heroku's 512 MB RAM limit--reasonable for most web processes, but quickly exceeded by the image-processing tasks of our delayed_job workers. We considered building a specialized image-processing service, but decided instead to supplement our web apps with a custom EC2 instance dedicated to processing background tasks. We call these servers "satellites."
We'll walk through the pertinent sections here, but you can find Rake tasks that correspond with these scripts, plus all of the necessary cookbooks, in the satellite_setup github repo. Now, on to the code!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Next, we'll do some basic server prep and install our preferred Ruby version. We'll again use Fog, this time to SSH into the new instance and run the following commands:
1 2 3 4 5 6 7 8 9
The first command replaces some broken links in the AMI's
/etc/apt/sources.list, so that the later package installation commands have a chance. We then install a few necessary packages, download and install Ruby 1.9.2 from source, and install RubyGems. Finally, we install Chef so we can let
chef-solo drive the remainder of the configuration (and manage it going forward).
Chef deserves a few blog posts of its own, but know that it provides a platform-independent DSL for specifying an environment's configuration. Opscode supplies a deep set of related tools, but for our purposes,
chef-solo--which applies a local set of configuration "cookbooks" to an individual server--will be plenty. These lines copy our local cookbooks folder (originally in
config/satellite) up to a temporary location on the server, then execute the
chef-solo command via SSH:
The first file referenced in that command simply declares where cookbooks will be found. In our case, they're stored alongside
solo.rb in the temporary directory.
The next file,
configure.json, specifies that Chef should apply the
configure recipe found in the
Finally, let's look at the
example_app::configure recipe. First it will make sure necessary packages and gems are installed. These, of course, might be different for your app.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Next, it will ensure that a user and group (named for
example_app) are created and configured:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
Next, it creates supporting directories.
1 2 3 4 5 6 7 8 9 10 11 12 13
We rely heavily on the New Relic plug-in, and want to enable it in our custom environment as well. Heroku usually takes care of injecting a
newrelic.yml configuration file into our app servers, so we'll have to replicate that in our custom environment. Depending on what plug-ins you've enabled (e.g., Sendgrid), you might need to replicate other configuration files or initializers.
1 2 3 4 5
Monit is a great tool for starting, managing, and monitoring long-running processes. It can be configured with all sorts of thresholds and alerting. (We've included only a simple configuration in the github repo for now.) Let's include the
monit recipe, to ensure that monit is installed and running, and then add the necessary configuration for monit to start and monitor our delayed_job worker process.
To keep disk space from becoming a problem, we'll automatically rotate all of the logs in our app's shared
1 2 3 4 5 6 7
That last line loads up the neighboring
deploy.rb recipe, which is responsible for checking out the appropriate version of the codebase to the server and [re]starting the delayed_job worker:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
deploy Chef command creates a new timestamped directory under
/app/example_app/releases and cleans up any older release directories, leaving the most recent 5 (much in the style of Capistrano). It also symlinks necessary directories and files and restarts the delayed_job worker.
Did you notice how we ensure that Heroku and the satellite use the same version of the codebase? We've formalized
production branches and update them with the commits we intend to deploy. Of course, we can't simply
git push heroku master anymore. Instead, we push to the appropriate branch, then push that to heroku and re-run the satellite's
deploy recipe. Here, we've wrapped the process up into a single
deploy:[staging|production] rake task. (The precise branches might vary depending on your development workflow.)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
Our satellite requires access to some of the same environment variables as our Heroku web apps (such as a database host or mail server credentials). To keep these synchronized, we rely on a
config/heroku.yml file. (This duplication is an obvious hazard and deserves improvement, but we've actually found it convenient to have these locally for easy access from Rake tasks, etc.)
In the satellite_setup github repo, we've simplified this set-up into a few tasks that we use on an ongoing basis:
1 2 3 4 5 6 7 8
This arrangement has worked for us so far. Satellite servers cost a little more, but can benefit from arbitrary customization and are served out of the same data-centers as our Heroku-based web apps and S3 storage. Since the same app is running transparently in 2 different environments, our developers' workflow has hardly needed modification. In fact, the portability enforced by Heroku's design (elaborated in The Twelve-Factor App) made this transition relatively straightforward.
Some worthy enhancements might be:
- Improving monitoring and notifications
- Extending the recipes to manage multiple parallel workers
- Using Chef attributes to replace uses of
example_appwith a parameter
- Cleaning up the duplication of Heroku configuration values in