Hanami with Sidekiq | Alchemists

Hanami with Sidekiq

As your Hanami application grows, you might want to add asynchronous background processing. A good gem to handle this for you is Sidekiq. You might also want Sidekiq Scheduler for dealing with scheduled jobs that repeat on a specific schedule. This is especially important if you are in a situation where you can’t upgrade to Sidekiq Pro which includes this feature by default. Let’s look at both gems and how you can wire them up in your application.

Table of Contents

Setup
Configuration
Settings
Provider
Routes
Authentication
Jobs
RSpec
Procfile
Conclusion

Setup

To get started, add both the Sidekiq and Sidekiq Scheduler gems as follows:

bundle add sidekiq
bundle add sidekiq-scheduler

Now you can alphabetically sort — or use RubyCop to auto-correct — and commit changes.

Configuration

To configure both Sidekiq and Sidekiq Scheduler, you’ll need a config/sidekiq.yml file. Example:

:queues:
  - ["within_1_minute", 1]
  - ["within_1_hour", 1]
  - ["within_1_day", 1]
:scheduler:
  :dynamic: true
  :rufus_scheduler_options:
    :max_work_threads: <%= ENV.fetch "HANAMI_MAX_THREADS", 5 %>

The above breaks down as follows:

queues: Necessary for Sidekiq. These are your latency queues as documented in The secret to happy queues where within_1_hour should be your default queue. Always a good Sidekiq practice.
scheduler: Necessary for Sidekiq Scheduler. Enables dynamic job scheduling including a configuration for the Rufus Scheduler that Sidekiq Schedule sits atop of.

Optionally, if you need specific jobs scheduled upon application boot, you can use config/sidekiq_scheduler.yml to define these jobs for Sidekiq Scheduler. Example:

firmware_poller:
  class: Demo::Jobs::Pollers::Firmware
  cron: "0 */6 * * *"
  description: "Polls for firmware changes."
model_poller:
  class: Demo::Jobs::Pollers::Model
  cron: "0 0 */1 * *"
  description: "Polls for model changes."
screen_poller:
  class: Demo::Jobs::Pollers::Screen
  cron: "0/5 * * * *"
  description: "Polls for device screen updates."

We’ll come back to this configuration, in a moment, when discussing the Sidekiq provider. In the meantime, you’ll need to create a configuration for launching Sidekiq as a separate process. Example:

# config/sidekiq.rb

require "hanami/boot"

You’ll need the path to this file for when you start Sidekiq as a separate process like via your Procfile (more on this shortly).

Settings

You’ll need to add the keyvalue_url which maps to the KEYVALUE_URL environment variable so you can easily connect to Valkey, Dragonfly, or Redis depending on your preference. Example:

# demo/config/settings.rb

module Demo
  # The application settings.
  class Settings < Hanami::Settings
    setting :keyvalue_url, constructor: Types::Params::String.constrained(filled: true)
  end
end

Using the keyvalue prefix allows you to be agonistic of which keyvalue memory store backend you choose. Each backend supports the redis:// scheme which makes swapping backends effortless.

Provider

As detailed in Hanami Containers, the best way to use Sidekiq and Sidekiq Scheduler is as a provider so you can inject Sidekiq into any object that needs it. In this case, use a provider class instead of the default provider configuration. Example:

# demo/app/providers/sidekiq.rb

# auto_register: false

module Demo
  module Providers
    # The Sidekiq provider.
    class Sidekiq < Hanami::Provider::Source
      include Deps[:logger]

      RESOLVER = proc { Object.const_get "Sidekiq" }

      def initialize(resolver: RESOLVER, **)
        @resolver = resolver
        super(**)
      end

      def prepare
        require "sidekiq"
        require "sidekiq-scheduler"
        require "yaml"
      end

      def start
        configure_server
        configure_client
        register :sidekiq, sidekiq
      end

      private

      attr_reader :resolver

      def configure_client
        sidekiq.configure_client do |configuration|
          configuration.redis = {url: slice[:settings].keyvalue_url}
          configuration.logger = slice[:logger]
        end
      end

      def configure_server
        sidekiq.configure_server do |configuration|
          configuration.redis = {url: slice[:settings].keyvalue_url}
          configuration.logger = slice[:logger]
          configuration.on(:startup) { load_schedule }
        end
      end

      def sidekiq
        @sidekiq ||= resolver.call
      end

      def load_schedule
        jobs = YAML.load_file slice.root.join("config/sidekiq_scheduler.yml")

        jobs.each do |schedule_name, options|
          resolver.call.set_schedule schedule_name, options
          job_name = options["class"]
          Object.const_get(job_name).perform_in 0
        rescue NameError, TypeError
          logger.error { "Unable to initialize job: #{job_name}." }
        end
      end
    end
  end
end

RSpec

The following is lightweight verification of the prepare and start steps of the provider lifecycle. You can always do more if you need more robust testing.

# spec/app/providers/sidekiq_spec.rb

require "hanami_helper"

RSpec.describe Demo::Providers::Sidekiq do
  subject :provider do
    described_class.new provider_container:, target_container:, slice:, resolver: proc { sidekiq }
  end

  let(:provider_container) { Dry::Core::Container.new }
  let(:target_container) { Dry::Core::Container.new }
  let(:slice) { Hanami.app }
  let(:sidekiq) { class_spy Sidekiq }

  describe "#prepare" do
    it "answers false due to already being loaded" do
      expect(provider.prepare).to be(false)
    end
  end

  describe "#start" do
    it "configures server" do
      provider.start
      expect(sidekiq).to have_received(:configure_server)
    end

    it "configures client" do
      provider.start
      expect(sidekiq).to have_received(:configure_client)
    end
  end
end

The above provider allows you to configure Sidekiq, Sidekiq Scheduler, and load the jobs — defined earlier — via config/sidekiq_scheduler.yml. The use of a provider class (and corresponding spec) is built in the same vien as documented in Hanami Containers. Pay special attention to the private load_scheudle method because it loads the Sidekiq Scheduler configuration and loops over each job to set it’s schedule.

You might think it would be easier to load the entire YAML configuration and then immediately set the schedule via #schedule= but that would erase all of your schedules each time you restart the application which is definitely not what you want. By loading the YAML configuration and looping over each configuration to set the schedule via #set_schedule, you end up gracefully updating your job schedules without completely blowing them away each time server is restarted.

With the above in place, all you need to do is register your provider as follows:

# demo/config/providers/sidekiq.rb

require_relative "../../app/providers/logger"

Hanami.app.register_provider :sidekiq, source: Terminus::Providers::Sidekiq

This grants you the ability to manage scheduled jobs within your application. Here’s an example from the Hanami console:

sidekiq = Hanami.app[:sidekiq]

# Create (schedule) the "demo" job.
sidekiq.set_schedule "demo",
                     {
                       cron: "* * * * *",
                       class: "Demo::Jobs::Demo",
                       queue: "within_1_minute"
                     }

# Get a specific schedule.
sidekiq.get_schedule "demo"

# Get multiple schedules by name.
sidekiq.get_schedule "demo", "other"

# View all schedules.
sidekiq.get_all_schedules

# Reloads all schedules and answers the full schedules hash.
sidekiq.reload_schedule!

# Remove the "demo" job schedule.
sidekiq.remove_schedule "demo"

# View the Rufus configuration.
SidekiqScheduler::Scheduler.instance.rufus_scheduler

💡 Due to Sidekiq Scheduler being built atop the Rufus Scheduler and Fugit, you can use tools like Crontab Guru to schedule your jobs. Fugit also supports natural language but has several limitations so use the cron syntax since it’s more reliable.

Routes

To make use of Sidekiq’s Web UI, you only need to require the web functionality and then mount Sidekiq.

require "sidekiq/web"
require "sidekiq-scheduler/web"

module Demo
  class Routes < Hanami::Routes
    mount Sidekiq::Web, at: "/sidekiq"
  end
end

Authentication

When using Hanami with Rodauth, you can have Rodauth ensure you can only access the Sidekiq Web UI when logged in. To do this, you’ll need middleware that can wrap the Sidekiq::Web middleware. Example:

Middleware

# demo/app/middleware/sidekiq_auth.rb
# auto_register: false

module Demo
  module Middleware
    class SidekiqAuth
      def initialize application
        @application = application
      end

      def call environment
        rodauth = environment["rodauth"]

        halted = catch :halt do
          rodauth.require_account
          nil
        end

        halted || application.call(environment)
      end

      private

      attr_reader :application
    end
  end
end

The above can then be used via your routes as follows:

require "sidekiq/web"
require "sidekiq-scheduler/web"

module Demo
  class Routes < Hanami::Routes
    slice :authentication, at: "/" do
      use Authentication::Middleware
      mount Middleware::SidekiqAuth.new(Sidekiq::Web), at: "/sidekiq"
    end
  end
end

That’s it. Read my Hanami with Rodauth article to learn more about how the authentication slice works.

Jobs

The kinds of jobs you create is up to you but you’ll want to have a Base class you can inherit from in terms of providing common functionality to all of your jobs. Example:

# demo/app/jobs/base.rb

# auto_register: false

require "dry/monads"
require "sidekiq"

module Demo
  module Jobs
    # The base abstract class for which all jobs inherit from.
    class Base
      include Dry::Monads[:result]
      include Sidekiq::Job
    end
  end
end

💡 Ensure you add the auto_register: false pragma to all jobs because you don’t want to litter you Hanami components with jobs you’ll never need to instantiate as you can let Sidekiq manage this for you. In cases where you need to inject a job for dynamic calling, use Initable.

RSpec

For RSpec, add the following to your Hanami Helper:

# spec/hanami_helper.rb

require "sidekiq/testing"

Sidekiq::Testing.inline!

RSpec.configure do |config|
  # Truncated for brevity...

  config.before(:suite) { Hanami.app.start :sidekiq }
  config.before { Sidekiq.redis(&:flushdb) }
end

The before suite configuration ensures Sidekiq is properly loaded for your test environment. Otherwise, the before each test configuration might not flush the Sidekiq test database properly which can result intermittent test failures. You’ll also want to a database ID of 1 or higher so you don’t clobber your local development/production database (which should default to zero).

Procfile

If you have a Profile, add the following entry:

worker: bundle exec sidekiq -r ./config/sidekiq.rb

The above will ensure the Sidekiq server is launched as a separate process for processing all jobs.

Conclusion

This only scratches the surface of what you can do with both Sidekiq and Sidekiq Scheduler. You’ll most likely want to tweak this configuration further, depending on your needs, but at least this will get you up and running quickly.