We help our clients to roll-out MVP as earliest as possible using a unique Rapid Prototyping approach of our Canvas to Cloud methodology. For that purpose, we use tools and technologies which are easy to learn and integrate. So delayed_job with ActiveJob has been our first choice for any MVP being built at BoTree for all background processing.

Once clients see traction in the MVP, they want us to build the enterprise solution and scale it. By nature, delayed_job doesn’t scale well for large number of jobs because of following limitations:

  1. Stores jobs in a database table, it slows down your application if there is a huge backlog of pending/failed jobs.
  2. Spin off processes instead of thread for each job.

To overcome this limitation in terms of scalability, we have a few choices like Sidekiq, Resque, Sucker Punch etc. for background processing and Sidekiq has always been our choice for following reasons:

  • Stores jobs in Redis (in-memory database) which is inherently faster over other databases.
  • Spin off very lightweight workers for each job to conserve resources.
  • Provides Web UI to monitor the job processing.

Here are a few simple steps to replace delayed_job with Sidekiq:

Step 1: Update the Gemfile

Replace

gem 'delayed_job_active_record'

or

gem 'delayed_job_mongoid'

with

gem 'sidekiq'

and bundle the application again.

Step 2: Change ActiveJob queue adapter in config/application.rb

Replace

config.active_job.queue_adapter = :delayed_job

with

config.active_job.queue_adapter = :sidekiq

Step 3: Create config/sidekiq.yml and configure queues

:queues:
  - [default, 1]

You can configure as many queues with priority as required here.

Step 4: Start Sidekiq

We can start delayed_job with following command

bin/delayed_job start

and Sidekiq with following command from the application root

sidekiq

Step 5. Cleanup

delayed_job stores jobs in delayed_jobs table which now no longer required. So write a migration to drop this table

class DropDelayedJobTable < ActiveRecord::Migration
  def up
    drop_table :delayed_jobs
  end

  def down
    raise ActiveRecord::IrreversibleMigration
  end
end

and run the migration to free up some space.

Additional Changes

Devise

If you are using devise and send notification emails to users in background, then you need to make following changes for Sidekiq

Step 1: Modify config/initializers/devise_async.rb

Replace

Devise::Async.backend = :delayed_job

with

Devise::Async.backend = :sidekiq

Step 2: Add mailer queue to config/sidekiq.yml

:queues:
  - [mailer, 2]
  - [default, 1]

Steps 3: Make sure you override following method in User model
def send_devise_notification(notification, *args)
  devise_mailer.send(notification, self, *args).deliver_later
end

Capistrano

If you deploy application using capistrano then you need to do the following steps to replace delayed_job with Sidekiq

Step 1: Modify Gemfile

Replace

gem 'capistrano3-delayed-job', require: false

with

gem 'capistrano-sidekiq', require: false

and bundle the application.

Step 2: Modify Capfile

Replace

require 'capistrano/delayed-job'

with

require 'capistrano/sidekiq'

Step 3: Modify config/deploy.rb

There is a known bug that prevents sidekiq from starting when pty is true on Capistrano 3.

Set capistrano environment variable pty for that reason

set :pty, false

and then deploy the application again.

Step 4: Manage remote Sidekiq from local machine

cap production sidekiq:start            # Start sidekiq
cap production sidekiq:stop             # Stop sidekiq
cap production sidekiq:restart          # Restart sidekiq

Sidekiq Web

Let us enable this useful visual interface to monitor the jobs. Add following configuration in routes.rb of your application.