Eric Guo's blog.cloud-mes.com

Hoping writing JS, Ruby & Rails and Go article, but fallback to DevOps note

Monitor Sidekiq Service With Monit & Capistrano

• Permalink

Sidekiq as a background processing facility in Rails, usually more and more important during business grow, but it's relative quite possible failed to startup due to there is some code error, so here is my solution to monitor it.

I choose the monit instead of god beause monit is written in C and very small memory footprint. The capistrano-sidekiq also having a out of box support, but when you combine with monit, capistrano, sidekiq it still trick, so I feel writing a blog is worth.

Install monit

Install monit
yum install monit # assure using CentOS

Configure monit

vi /etc/monitrc:

Configure monit
set daemon  60

 
set mailserver localhost

 
set mail-format {

     from: monit@your-domain.com

  subject: monit alert --  $EVENT $SERVICE

  message: $EVENT Service $SERVICE

 
Hi all people emails address;

 
Event happen at: $DATE

Action:          $ACTION

Host:            $HOST

Description: $DESCRIPTION

 
Your faithful employee,

 
Monit (robot)

Hint: when resolved, please replay all.

}

 
set alert receiver1@mail.com

set alert receiver2@mail.com

Configure in capistrano-sidekiq

At config/deploy/production.rb

Set Sidekiq Monit options in the deploy configuration
set :sidekiq_monit_conf_dir, '/etc/monit.d' # for CentOS folder

set :sidekiq_monit_use_sudo, false

At Capfile

Load the Capistrano Sidekiq Monit tasks
require 'capistrano/sidekiq'

require 'capistrano/sidekiq/monit' # to require monit tasks

Enable alert by customize the monit template.

Generate the Sidekiq Monit configuration template
rails g capistrano:sidekiq:monit:template

and change as below:

Add alert checks to the Sidekiq Monit template
# Monit configuration for Sidekiq :  <%= fetch(:application) %>

<% processes_pids.each_with_index do |pid_file, idx| %>

check process <%= sidekiq_service_name(idx) %> with pidfile "<%= pid_file %>"

  if changed pid then alert

  start program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiq] %> <%= sidekiq_config %> --index <%= idx %> --pidfile <%= pid_file %> --environment <%= fetch(:sidekiq_env) %> <%= sidekiq_concurrency %> <%= sidekiq_logfile %> <%= sidekiq_queues %> <%= sidekiq_options_per_process[idx] %> -d'" with timeout 30 seconds

 
  stop program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiqctl] %> stop <%= pid_file %>'" with timeout <%= fetch(:sidekiq_timeout).to_i + 10  %> seconds

  group <%= fetch(:sidekiq_monit_group, fetch(:application)) %>-sidekiq

 
<% end %>

Deploya to generate minit config

Deploya to generate minit config
cap production deploy

After that checking if /etc/monit.d having sidekiq conf file. or running cap sidekiq:monit:config to generate config.

Running monit at production

Running monit at production
monit

To test the mail, using monit reload, if still can not received the mail, checking my postfix blog.

Comments