Background jobs in Ruby

Worker

You are developing a Ruby application where the user can sign up, and after submitting the form, the user has to receive an email. Will you send it immediately? If so, the user will have to wait while the application connects to the email server and sends the actual email. That’s bad for the user experience.

To solve that problem, the email should be sent in the background. That way it could be enqueued to be sent later, and the user sees the confirmation page quickly. Much better!

This is accomplished by a queue system, which runs in the background waiting for new jobs to be executed. That’s the term used to refer to the tasks. In our case, the email is the job to be executed.

There are a few of popular systems. A few need a database, such as Delayed::Job, while others prefer Redis, such as Resque and Sidekiq.

Some common jobs that should be executed in the background are:

  • Sending an email or a massive quantity
  • Resizing an image
  • Importing a batch of data
  • Updating a search server

Delayed::Job

This systems needs a database, because it uses a table to manage jobs. The “delayed” part in its name comes from the way it enqueues a job: using the delaymethod. So if we have an object to be run like this:

With Delayed::Job it can be enqueued this way:

It’s only one method in the middle, very handy. However, the class can also be adapted so a method is always processed by Delayed::Job asynchronously. That’s accomplished using the handle_asynchronously helper:

Delayed::Job even can assign priorities, run at a given time, and enqueue to multiple queues. For instance, to run a job in 5 minutes with a priority of 2, enqueue it like this:

Or using the handle_asynchronously feature, we could write it as follows:

And then calling the method as before:

Resque

Resque is based on Delayed::Job, but it uses Redis instead of a database. It also provides a Sinatra application to monitor jobs so you can see which ones are running and their queues.

To make a class compatible with Resque so it can be run in the background, a new method must be implemented: perform. This is the method that will be called by Resque:

The self.perform method can do anything, it doesn’t need to be a “caller” like in this case. I could have moved the long running method inside.

To schedule a job to be run at a specified time, Resque needs the resque-scheduler gem. Once installed the jobs can be enqueued like this:

Resque doesn’t support numeric priorities as Delayed::Job, but it’s based in the order the queues are defined. So the first ones have the higher priority. This is defined when launching the process:

Resque will be checking the “high” queue and running its jobs. When that queue is empty, it will start checking the “low” queue.

Then the class must define the queue to use:

Sidekiq

Sidekiq also needs Redis to work, but its main difference is that it uses threads, so several jobs can be executed in parallel using the same Ruby process. Also, it uses the same message format than Resque, so the migration should be easy.

Knowing its use of threads, it’s clear that its main intention was to be the fastest job processing system for Ruby. It’s several times faster than Resque and Delayed::Job. Summing up that it’s easy to use, it’s the system that I like to adopt for new projects and the one that I recommend nowadays.

Like Resque, it includes a Sinatra application to monitor jobs, which ones are scheduled, which ones failed and are pending to retry, and so on.

Since it’s compatible with Resque, classes to be processed by Sidekiq must implement the perform method:

It’s also very similar to run this job in the future:

It even supports Delayed::Job’s syntax:

To assign a priority to a job, it uses the same approach than Resque: strict queue ordering. When creating them, the first ones have the higher priority. Again, this is defined when launching Sidekiq:

This is one way to define queues. They can also be defined as weighted queues, so they can be chosen randomly taking into account that one can have more weight than other, and therefore much probable to be chosen:

Now there are 2 queues that can be represented like an array and be sampled:

This way the “low” queue will have more opportunities to be chosen independently of the “high” queue being empty or not.

Like in Resque, the class must define the queue to use:

There are unofficial gems that support Delayed::Job numerical priorities, but it’s not supported out of the box by Sidekiq.

ActiveJob

Because each system has its own syntax, Rails offers ActiveJob, a standard way to deal with these systems, so the application can be agnostic and able to switch among systems like when using ActiveRecord for the database, keeping the same syntax.

The first step is to configure it with the chosen system. The instructions are generally provided by the system itself.

Classes and the way objects are enqueued look very similar to Sidekiq:

Of course, it supports scheduling:

The way to define a queue for a class is the same as Sidekiq:

But it can also be set when enqueuing the job:

Please note that these queues must be created using the queuing system of choice when starting the process.

Conclusion

Background jobs are an essential part in any big project. Choosing a queue managing system is a decision to be made sooner or later. Although there are other alternatives out there that are active as well, these 3 are the most popular ones.

I recommend Sidekiq because it’s easy to use and efficient. It’s also very fast compared to Delayed::Job and Resque. Its excellent user interface to manage jobs and queues can also give it another point.

Nonetheless it requires Redis, so if your application runs short of RAM, maybe it would be a good idea to stick with Delayed::Job and your database. It would be a good start, and by using ActiveJob you can always migrate to Sidekiq in the future without changing a single line of code.

Did you like it? Please share it:

Get my ebook for free

10 ideas that helped me become a better developer (and may help you too)

Subscribe to my mailing list and get my ebook on 10 ideas that helped me become a better developer.

About Me

David Morales

David Morales

I'm David Morales, a computer engineer from Barcelona, working on projects using Ruby on Rails and training on web technologies.

Learn More