Rails cronjobs - Moving from Whenever to Sidekiq-cron (With ActiveJob)

1st April 2024 – 1265 words

Recently, we migrated one app’s cronjobs from using simple whenever Crontab to a more enhanced Sidekiq-Cron.

Whenever - The old way

We used to have a simple whenever setup in our Rails app. It was easy to use and worked well for a long time. You can define your cronjobs in a Ruby DSL and then generate a crontab file with whenever --update-crontab during deployment. The system crond will then execute the jobs very reliably. If you change the crontab at the end of the deployment you have also no downtime in the cronjobs.

Plus:

  • Easy to define cronjobs,
  • reliable execution with system crond on one machine
  • Zero downtime during deployment by default

Minus:

  • No monitoring by default - If your cronjobs fail to load/boot, you will not notice it unless you check the logs.
  • No retries - If a job fails, it will not be retried.
  • Reliability depends on the machine running the crond. If you have multiple hosts, you have to decide where to run the cronjobs.
  • Not easily portable and difficult to integrate into Kamal or other Container engines (Kubernetes etc.)
  • Chance of performance issues, as you always need to boot the whole app. If you have long-running jobs, your RAM might be exhausted.

So, because we are moving to a container infrastructure and found some obstacles with running cronjobs in a container, we decided to move to Sidekiq-Cron.

Alternatives

I started to look at alternatives and found a few, mostly based upon Sidekiq worker - which we already use in our app. Our requirements were:

  • Should work with ActiveJob
  • have a web interface to monitor the jobs
  • should be easy to run - Best case, no new Daemon is required

So I found a few alternatives:

  • Solid Queue - Recurring Tasks Solid Queue is the new way for ActiveJob background jobs. Because we have a couple of Jobs that rely on Sidekiq and some of our plugins (Because our app is older than ActiveJob). Also, it was just recently released, so we decided to rather go with other solutions and maybe try out Solid Queue in a different project first.
  • Discourse MiniScheduler After running a Discourse instance I already found the nice DSL they use to define Sidekiq Workers. Just inside the Job/Worker class itself, instead of a external yml/json whatever. On the other hand, it has no specific ActiveJob support.
  • Sidekiq-Scheduler - Seems to be a very popular Gem. According to this blog post the WebUI of Sidekiq-Cron is better. So we rather tried that
  • Sidekiq-Cron - This seems to be well maintained, has an CRUD-API interface for defining dynamic jobs (like customer specific jobs) and static ones, and a nice integration into the Sidekiq web interface.

Sidekiq-Cron - The new way

We decided to go with Sidekiq-Cron. But we enhanced it with a DSL - like Discourse MiniScheduler. Therefore, we created a new parent class:

class Cronjob < ApplicationJob
  queue_as :low
  class << self
    attr_accessor :cronjob_definition
  end
  def self.cronjob(cron:, namespace: nil, name: nil, dev: false, description: nil)
    r = Fugit.parse(cron)
    if r.nil? || r.class != Fugit::Cron
      raise ArgumentError, "Invalid cron expression: #{cron}, #{r.inspect}"
    end
    @cronjob_definition = {
      name: name || self.name.demodulize.underscore,
      cron:,
      class: self,
      description:,
      namespace:,
      source: 'schedule',
      status:
        if Rails.env.development? && !dev
          'disabled'
        else
          'enabled'
        end,
      active_job: true
    }.compact
  end

  def self.load!
    Rails.configuration.eager_load_namespaces.each(&:eager_load!) if Rails.env.development?

    schedule = Cronjob.subclasses.map(&:cronjob_definition).compact
    r = Sidekiq::Cron::Job.load_from_array!(schedule)
    if r.present?
      warn "Cronjob maybe not loaded: #{r.inspect}"
    end
  end
end

And now we load the cronjobs during application boot:

# config/initializers/cronjobs.rb

Sidekiq.configure_server do |config|
  config.on(:startup) do
    if Rails.env.production? || ENV['RAILS_ENABLE_CRON'].present?
      Cronjob.load!
    end
  end
end

With this infrastructure in place, we can now define our cronjobs in the Jobs themselves:

# app/jobs/cronjob/cleanup_job.rb

class Cronjob::CleanupJob < Cronjob
  cronjob cron: '0 0 * * *', description: 'Cleanup old data'

  def perform
    # Do something
  end
end

…and it will be loaded during the application boot. By default, we disable the cronjobs in development, but you can enable them one-by-one for testing purposes by setting the environment variable RAILS_ENABLE_CRON and mark the individual job with dev: true.

Migrate each Schedule statement: runner & Rake task

Now, there was a long work: We had used whenever a lot, to just run specific ruby methods, like:

every 1.day, at: '4:30 am' do
  runner "MyModel.cleanup_old_data"
end

This does not work anymore, so we have to create Job class for every one of those tasks, something like:

# app/jobs/cronjob/cleanup_old_data_job.rb
class Cronjob::CleanupOldDataJob < Cronjob
  cronjob cron: 'every day at 4:30pm', description: 'Cleanup old data'
  def perform
    MyModel.cleanup_old_data
  end
end

Also, some Gems like blazer defines a couple of rake tasks that had to be migrated:

# app/jobs/cronjob/blazer_daily_job.rb

class Cronjob::BlazerDailyJob < Cronjob
  cronjob cron: '0 0 * * *', description: 'Run Blazer Daily'
  def perform
    # you might run the task like this:
    # Rake::Task['blazer:daily'].invoke
    # Or just have a look inside the task and copy the code here
    Blazer.run_checks(schedule: '1 day')
    Blazer.send_failing_checks
  end
end

Conclusion

For many of our simpler projects, we still use Whenever, which is dead simple. But for the more complex ones, we will run Sidekiq-Cron. The migration was a little tedious, as we have tons of cronjobs. But having them all now in the Sidekiq-WebUI is a big win. In some cases, it is now easy to just run a daily/weekly job early by clicking the button in the Web-UI. Also, we can now easily see if a job ran, or failed and retry it. Using the DLS makes the definition very easy and readable. Also, if you rename, move or delete a job, you don’t need to remember to update another yml/schedule whatever file

Tags: Rails Sidekiq
Related posts