Sidekiq as a background processing facility in Rails, usually more and more important during business grow, but it's relative quite possible failed to startup due to there is some code error, so here is my solution to monitor it.

I choose the monit instead of god beause monit is written in C and very small memory footprint. The capistrano-sidekiq also having a out of box support, but when you combine with monit, capistrano, sidekiq it still trick, so I feel writing a blog is worth.

Install monit

yum install monit # assure using CentOS

Configure monit

vi /etc/monitrc:

set daemon  60
 
set mailserver localhost
 
set mail-format {
     from: monit@your-domain.com
  subject: monit alert --  $EVENT $SERVICE
  message: $EVENT Service $SERVICE
 
Hi all people emails address;
 
Event happen at: $DATE
Action:          $ACTION
Host:            $HOST
Description: $DESCRIPTION
 
Your faithful employee,
 
Monit (robot)
Hint: when resolved, please replay all.
}
 
set alert receiver1@mail.com
set alert receiver2@mail.com

Configure in capistrano-sidekiq

At config/deploy/production.rb

set :sidekiq_monit_conf_dir, '/etc/monit.d' # for CentOS folder
set :sidekiq_monit_use_sudo, false

At Capfile

require 'capistrano/sidekiq'
require 'capistrano/sidekiq/monit' # to require monit tasks

Enable alert by customize the monit template.

rails g capistrano:sidekiq:monit:template

and change as below:

# Monit configuration for Sidekiq :  <%= fetch(:application) %>
<% processes_pids.each_with_index do |pid_file, idx| %>
check process <%= sidekiq_service_name(idx) %> with pidfile "<%= pid_file %>"
  if changed pid then alert
  start program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiq] %> <%= sidekiq_config %> --index <%= idx %> --pidfile <%= pid_file %> --environment <%= fetch(:sidekiq_env) %> <%= sidekiq_concurrency %> <%= sidekiq_logfile %> <%= sidekiq_queues %> <%= sidekiq_options_per_process[idx] %> -d'" with timeout 30 seconds
 
  stop program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiqctl] %> stop <%= pid_file %>'" with timeout <%= fetch(:sidekiq_timeout).to_i + 10  %> seconds
  group <%= fetch(:sidekiq_monit_group, fetch(:application)) %>-sidekiq
 
<% end %>

Deploya to generate minit config

cap production deploy

After that checking if /etc/monit.d having sidekiq conf file. or running cap sidekiq:monit:config to generate config.

Running monit at production

monit

To test the mail, using monit reload, if still can not received the mail, checking my postfix blog.

Some situation is better avoid Sidekiq like reporting the exception mail, so it's always the best practices to building a Postfix Satellite SMTP service instead.

Install Postfix

yum install -y postfix cyrus-sasl-plain # cyrus-sasl-plain is no need for Ubuntu
yum erase -y sendmail* # In case Sendmail is installed

Configure

Similar to gmail or SendGrid, every mail service vendor offer different combination of setting, below is only apply to AliYun

/etc/postfix/main.cf change

# comment out below
#inet_interfaces = localhost
relayhost = [smtp.mxhichina.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CApath = /etc/ssl/certs
smtp_use_tls = yes
smtp_generic_maps = hash:/etc/postfix/generic
mynetworks = 127.0.0.0/8
mydestination =

Create /etc/postfix/sasl_passwd

[smtp.mxhichina.com]:587    no-reply@domain.com:PASSWORD

Create /etc/postfix/generic

root@staging.domain.com   no-reply@domain.com

Create a database file

postmap /etc/postfix/sasl_passwd /etc/postfix/generic

Prevent non-root access:

chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db /etc/postfix/generic /etc/postfix/generic.db

Restart Postfix service

/etc/init.d/postfix restart # or service postfix restart

Debug

It's very common that postfix will failed to sent mail at first time, so add below to main.cf.

debug_peer_level = 3
debug_peer_list = smtp.mxhichina.com

And open another console to testing via:

echo "body of your email" | mail -s 'mail subject from console' -r 'no-reply@domain.com' guochunzhong@domain.com

You may also need postconf -n to review postfix configuration or postconf -d to review all setting.

YOu can also monitor the postfix activity via tail -f /var/log/syslog.

Dash 3 switch to the subscription mode, if you just want to see the Rails document, in fact, it's relative easier to build in your own box.

Building Rails API Document

cd ~/git/oss/
git clone https://github.com/rails/rails
cd rails/
bundler install
rake rdoc

Building Rails Guides

cd guides/
rake guides:generate:html

Enable Apache

sudo apachectl start

Link Rails documents

cd /Library/WebServer/Documents/
sudo ln -s /Users/guochunzhong/git/oss/rails/doc/rdoc/ rails_api
sudo ln -s /Users/guochunzhong/git/oss/rails/guides/output/ rails_guides

View in localhost

visit Rails API at http://localhost/rails_api/

visit Rails Guides at http://localhost/rails_guides/

My log to build a Ubuntu server to fit my software stack usage, R & Ruby.

Install SSH

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install open-vm-tools
sudo apt-get install openssh-server

Install JDK

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update && sudo apt-get install oracle-java7-installer

Install rvm

sudo su - # switch to root if not yet
apt-get install curl
curl -L get.rvm.io | bash -s master # install rvm
logout
sudo su -
rvm requirements # to install all depends

Install Ruby

rvm install ruby-2.2.3
gem source --remove https://rubygems.org/
gem source --add https://gems.ruby-china.com/
gem update --system
apt-get install libgmp3-dev
echo "gem: --no-document" >> /etc/gemrc
gem update
gem install bundler

Install Oracle Client

apt-get install unrar
apt-get install libaio-dev # must need if you want to use sqlplus
mkdir -p /opt/oracle && cd /opt/oracle
unrar x /home/eric/instantclient-all-linux.x64-11.2.0.4.0.rar .
chmod +x instantclient_11_2/sqlplus
ln -s /opt/oracle/instantclient_11_2/libclntsh.so.11.1 /opt/oracle/instantclient_11_2/libclntsh.so
ln -s /opt/oracle/instantclient_11_2/sqlplus /usr/local/bin/sqlplus
echo "/opt/oracle/instantclient_11_2" >> /etc/ld.so.conf.d/oracle_instantclient.conf

# append below line to /etc/enviroment file
ORACLE_BASE=/opt/oracle
ORACLE_HOME=/opt/oracle/instantclient_11_2
TNS_ADMIN=/opt/oracle/network/admin
LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2
OCI_HOME=/opt/oracle/instantclient_11_2
OCI_LIB=/opt/oracle/instantclient_11_2
OCI_LIB_DIR=/opt/oracle/instantclient_11_2
OCI_INCLUDE_DIR=/opt/oracle/instantclient_11_2/sdk/include
NLS_LANG=AMERICAN_AMERICA.AL32UTF8
rvmsudo_secure_path=1

Install Oracle gems ruby-oci8

reboot
sudo su -
ldconfig
mkdir -p network/admin
mv /home/eric/tnsnames.ora .
gem install ruby-oci8 # should be no any error here :-)

Install R

sudo vi /etc/apt/sources.list
# append below line to end of sources.list
# you can view mirror at http://cran.r-project.org/mirrors.html
deb http://cran.ism.ac.jp/bin/linux/ubuntu trusty/

apt-get update
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 51716619E084DAB9
# replace your NO_PUBKEY error above, if pause/block port, using 80 as below line
gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E084DAB9
gpg -a --export E084DAB9 | sudo apt-key add -
apt-get update # make sure no error here

apt-get install r-base

Install ROracle

wget http://cran.r-project.org/src/contrib/DBI_0.3.1.tar.gz
R CMD INSTALL DBI_0.3.1.tar.gz
wget http://cran.r-project.org/src/contrib/ROracle_1.2-1.tar.gz
R CMD INSTALL --configure-args='--with-oci-inc=/opt/oracle/instantclient_11_2/sdk/include --with-oci-lib=/opt/oracle/instantclient_11_2' ROracle_1.2-1.tar.gz

Install Passenger

Just following Passenger documentation

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7
sudo apt-get install -y apt-transport-https ca-certificates
 
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main > /etc/apt/sources.list.d/passenger.list'
sudo apt-get update
 
sudo apt-get install -y nginx-extras passenger

passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /usr/local/rvm/gems/ruby-2.2.3/wrappers/ruby;
passenger_max_pool_size 30;

Install R studio server

Install R studio server

sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-0.99.473-amd64.deb
sudo gdebi rstudio-server-0.99.473-amd64.deb

# Server Configuration File
www-address=127.0.0.1

server {
  listen       80;
  server_name  cvprstudio;
  server_name  cvprstudio.sandisk.com;
  client_max_body_size 10m;
  location / {
    proxy_pass http://127.0.0.1:8787;
    proxy_redirect http://127.0.0.1:8787/ $scheme://$host/;
  }
}

cd /etc/nginx/sites-enabled/
ln -s /etc/nginx/sites-available/cvprstudio cvprstudio
service nginx restart

Install RStudio Shiny Server

Install RStudio Shiny Server

R -e "install.packages('shiny', repos='https://cran.rstudio.com/')"
wget https://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.4.0.721-amd64.deb
gdebi shiny-server-1.4.0.721-amd64.deb

server {
  listen       80;
  server_name  shinyapp;
  server_name  shinyapp.sandisk.com;
  location / {
    proxy_pass http://127.0.0.1:3838;
    proxy_redirect http://127.0.0.1:3838/ $scheme://$host/;
  }
}

cd /etc/nginx/sites-enabled/
ln -s /etc/nginx/sites-available/shinyapp shinyapp
service nginx restart

Following patch process in previous blog to re-install the shiny

R CMD INSTALL shiny_0.12.2-server.tar.gz 

I'm stucked quite a while when refresh a Galaxy Note II for friends, so I decide write something down:

1. Wait some time after USB drive installed N7100 driver, something due to china poor network and gfw, the driver install from internet may failed.
2. Root before install any 3rd party recovery, it's hardest part of change, I reference this for the Europe model, but you may need to found another way. This is must need before refresh any 3rd party recovery, because seems if not rooted, Knox will stop you do any further action.
3. Install a 3rd recovery image, for example N7100 CWM Recovery v6.0.5.1
4. Install customed notes 4 ROM for note 2, you can using this or mirror link

When the hkp port blocked, so rvm suggested gpg --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3 always failed to running, here is how to resolve such problem.

1. Find a hkp port not block server, run export after import the key:

   gpg --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3 gpg --export --armor D39DC0E3

2. Go to hkp port blocked server, run:

   gpg --import -

and copy and paste the step 1 server public key content and press Ctrl+D

Another option you can try is using port 80, but it's can not always work.

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 561F9B9CAC40B2F7

eventmachine didn't release a new gems for a long time, so we have to manually compile if you decide using Ruby 2.1.3, it's probably the most complex gems which can still successfully install on windows as far as I known, here is the steps for your interesting:

git clone https://github.com/eventmachine/eventmachine
cd eventmachine && bundle install

Now you will meet the bluecloth can not install problem first, which you can following stackoverflow steps including patch ext/bluecloth.h and 6 steps.

Now you need using updated version of rake-compiler (v0.9.3) to continue, so change eventmachine.gemspec to that version and do the bundle install again.

Now we can running rake package and it should generate the updated 1.0.3 gem file you need, but before that you need another depend lzma package mentioned in Ruby Installer Google Group, I only tested x86 but the x64 should be same.

1. Put the download openssl-1.0.0m-x86-windows.tar.lzma to c:\temp
2. run C:\DevKit\devkitvars.bat
3. c:\Temp> bsdtar --lzma -xf openssl-1.0.0m-x86-windows.tar.lzma
4. c:\git\eventmachine\pkg> gem install ./eventmachine-1.0.3.gem --platform=ruby -- --with-opt-dir=C:/Temp

Now enjoy eventmachine on ruby 2.1.3 windows!

gem dependency parser --reverse-dependencies or -R in short only works in local, so if you want to get all gems from rubygems, can using:

ruby -ropen-uri -rpp -ryaml -e 'pp YAML.load(open("https://rubygems.org/api/v1/gems/parser/reverse_dependencies.yaml"))'

The tip is coming from Faria DevTips.

Except simply install the ruby 2.1.3 from rubyinstaller and it's DevKit, here is my log when I meet the problem during ruby 2.1.3 on a Windows 7 32bits machine.

1. Comments out the warn "DL is deprecated, please use Fiddle" at C:\Ruby21\lib\ruby\2.1.0\dl.rb

2. Install yajl-ruby via gem install yajl-ruby -v 1.1.0 --platform ruby

3. Install RedCloth via gem install RedCloth --platform ruby and move the file C:\Ruby21\lib\ruby\gems\2.1.0\gems\RedCloth-4.2.9\lib\redcloth_scan.so to new created folder C:\Ruby21\lib\ruby\gems\2.1.0\gems\RedCloth-4.2.9\lib\2.1.

4. Install sqlite3 via:

   1. run C:\DevKit\devkitvars.bat
   2. mkdir c:\temp
   3. download http://packages.openknapsack.org/sqlite/sqlite-3.7.15.2-x86-windows.tar.lzma to c:\temp
   4. c:\Temp>bsdtar --lzma -xf sqlite-3.7.15.2-x86-windows.tar.lzma
   5. c:\Temp>gem install sqlite3 --platform=ruby -- --with-opt-dir=C:/Temp

5. Install bcrypt via gem install bcrypt --platform ruby

6. Install win32console via gem install win32console --platform ruby

7. Install ffi via gem install ffi --platform ruby

8. Install pg via gem install pg --platform ruby

9. Install mysql following stackoverflow via gem install mysql2 --platform=ruby -- '--with-mysql-dir="C:\mysql-connector"'

10. Install puma via gem install puma -- --with-opt-dir=c:\temp

Setting global environment setting:

CURL_CA_BUNDLE=C:\Ruby21\share\ca-bundle.crt
SSL_CERT_FILE=C:\Ruby21\share\cacert.pem
NLS_LANG=AMERICAN_AMERICA.UTF8

Also do not using ansicon in ruby 2.1.3 any more, seems not compatible.