Migrate from delayed_job to Sidekiq
We help our clients to roll-out MVP as earliest as possible using a unique Rapid Prototyping approach of our Canvas to Cloud methodology. For that purpose, we use tools and technologies which are easy to learn and integrate. So delayed_job with ActiveJob has been our first choice for any MVP being built at BoTree for all background processing.
Once clients see traction in the MVP, they want us to build the enterprise solution and scale it. By nature, delayed_job doesn’t scale well for large number of jobs because of following limitations:
- Stores jobs in a database table, it slows down your application if there is a huge backlog of pending/failed jobs.
- Spin off processes instead of thread for each job.
To overcome this limitation in terms of scalability, we have a few choices like Sidekiq, Resque, Sucker Punch etc. for background processing and Sidekiq has always been our choice for following reasons:
- Stores jobs in Redis (in-memory database) which is inherently faster over other databases.
- Spin off very lightweight workers for each job to conserve resources.
- Provides Web UI to monitor the job processing.
Here are a few simple steps to replace delayed_job with Sidekiq:
Step 1: Update the Gemfile
and bundle the application again.
Step 2: Change ActiveJob queue adapter in config/application.rb
config.active_job.queue_adapter = :delayed_job
config.active_job.queue_adapter = :sidekiq
Step 3: Create config/sidekiq.yml and configure queues
:queues: - [default, 1]
You can configure as many queues with priority as required here.
Step 4: Start Sidekiq
We can start delayed_job with following command
and Sidekiq with following command from the application root
Step 5. Cleanup
delayed_job stores jobs in delayed_jobs table which now no longer required. So write a migration to drop this table
class DropDelayedJobTable < ActiveRecord::Migration def up drop_table :delayed_jobs end def down raise ActiveRecord::IrreversibleMigration end end
and run the migration to free up some space.
If you are using devise and send notification emails to users in background, then you need to make following changes for Sidekiq
Step 1: Modify config/initializers/devise_async.rb
Devise::Async.backend = :delayed_job
Devise::Async.backend = :sidekiq
Step 2: Add mailer queue to config/sidekiq.yml
:queues: - [mailer, 2] - [default, 1] Steps 3: Make sure you override following method in User model
def send_devise_notification(notification, *args) devise_mailer.send(notification, self, *args).deliver_later end
If you deploy application using capistrano then you need to do the following steps to replace delayed_job with Sidekiq
Step 1: Modify Gemfile
gem 'capistrano3-delayed-job', require: false
gem 'capistrano-sidekiq', require: false
and bundle the application.
Step 2: Modify Capfile
Step 3: Modify config/deploy.rb
There is a known bug that prevents sidekiq from starting when pty is true on Capistrano 3.
Set capistrano environment variable pty for that reason
set :pty, false
and then deploy the application again.
Step 4: Manage remote Sidekiq from local machine
cap production sidekiq:start # Start sidekiq cap production sidekiq:stop # Stop sidekiq cap production sidekiq:restart # Restart sidekiq
Let us enable this useful visual interface to monitor the jobs. Add following configuration in routes.rb of your application.
This configuration protects production application via basic authentication. All you need to do is set the username and password in environment with the mentioned environment variables.
Now restart the application and you can access Sidekiq web interface at /sidekiq path in any web browser.
At BoTree Technologies, we build enterprise apps with our web development team of 30+ engineers.
Consulting is free - let us help you grow!
Choose Your Language
- Digital Marketing
- IT Consulting
- Project Management