Eric Guo's blog.cloud-mes.com

Hoping writing JS, Ruby & Rails and Go article, but fallback to DevOps note

在Macbook Pro 2016中原生安装tensorflow 1.0

• Permalink

前几天的TensorFlow开发者峰会宣布的TensorFlow 版本达到1.0，再加上过去一年人工智能，机器学习非常非常热，虽然作为一名Ruby程序员，但形势所迫，又不得不捡起了Python，准备稍稍了解一下。

TensorFlow的安装官网写的很清晰，我选择的是virtualenv的安装，安装的python版本是brew的版本。

brew install python

sudo easy_install pip

sudo pip install --upgrade virtualenv

virtualenv --system-site-packages ~/tensorflow

source ~/tensorflow/bin/activate 

pip install --upgrade tensorflow

安装完成后每次运行需要先运行source ~/tensorflow/bin/activate，嫌麻烦可以把这句加到.bash_profile中。

echo "source ~/tensorflow/bin/activate" >> ~/.bash_profile

安装完成后可以跑一下tensorflow版的“Hello Word”验证一下安装是否正确，需要先进入python的交互环境：

python

然后运行：

import tensorflow as tf

hello = tf.constant('Hello, TensorFlow!')

sess = tf.Session()

print(sess.run(hello))

问题来了，一堆CPU指令没有优化，Macbook Pro 2016已经没有核弹卡了，只能跑在CPU版上了，如果CPU再不优化，感觉已经很难愉快的玩下去了，所以只能选择从源码安装了。

源码安装就很烦了，下载源吗就非常费网络，还必须FQ，编译也需要至少25分钟时间，不过亲爱的读者，我已经帮你们做好了，现在可以使用我的安装包直接升级到原生CPU优化版：

pip install --upgrade http://baye-deploy.oss-cn-shanghai.aliyuncs.com/tensorflow-1.1.0-cp27-cp27m-macosx_10_12_x86_64.whl

巴爷供销社，一个生产级别的微信小应用商城开源实现

• Permalink

今天是微信小应用发布的日子，很高兴巴爷科技（上海）有限公司能作为第一批微信小应用的Rails厂商，能为Rails社区提供一个开源的参考实现。（包括后端）

微信小应用的介绍文章已经很多了，快速的开发体验，iOS／Android不必分别开发俩次都是很大的优点。但同时，腾讯微信也不是上次的公众号发布时的微信了，7亿的用户，近乎100%的中国市场，所以腾讯变的非常非常小心谨慎，所以也不要指望能像原来的公众号，能从腾讯分到近乎免费的流量红利。

微信小程序限制非常之多：微信小程序二维码只能通过扫一扫进入，不能通过图片扫描；微信小程序没有应用商店，只能通过发现，小程序的的搜索功能寻找；更令人发指的是，非合作伙伴的话，还必须搜索全名才行，也就是说，只有腾讯投资的京东，才能搜索“京东”找到“京东购物”，像巴爷科技，搜索“巴爷”是无法得到任何结果的，必须搜索“巴爷供销社”。

微信小应用在开发方面，微信基本保持了和服务号近似的接口，例如微信支付的对接。但也有不同的地方，例如敏感数据的加解密等。另外由于我们开始开发等时间较早，持续到现在，很多API不断发生改变，这也确实给我们的开发造成了一些困扰。

微信小应用递交审核方面，由于我们不是内测的200个名单中的公司，而是作为第二批公测用户，递交审核的周期和苹果差不多，一次审核周期在1～2天，甚至周末也有人审核。但不好的地方在于，拒的话，只会告诉你一个问题，而且一旦发现应用有bug，有和应用说明中没写的功能，超出微信小应用所选分类的功能，都会被拒。所以如果你的应用如果想快速上线，基本很难。审核期间我们在是否需要ICP证方面被拒了3次，中国的政策很奇怪，理论上只要你有营利性的网上经营行为就必须要ICP证，否则就属于非法经营。但是如果你到电信管理局去申请这个ICP证，办事人员却会以你的公司业务量太小，拒绝你的申请（🐷要养肥了杀？），但好在腾讯在最后一刻，终于在审核上线时，不需要微信小应用递交ICP证了，所以，现在能访问到“巴爷供销社”，还是要感谢腾讯在政府关系方面做出的努力。

微信小程序的github开源地址：https://github.com/bayetech/wechat_mall_applet

后端基于active_model_serializers的参考实现地址：https://github.com/bayetech/wechat_mall_applet_backend

A Rubyist Web Developer Macbook Pro With Touch Bar Installation Notes

• Permalink

Just got the Macbook Pro 2016 with touch bar 15' edition, setting it up takes my one full day. If the China network condition is good, I believe it will take much less time.

I decide the Macbook Pro from scratch instead of using migration assist because I want to keep the system clean. I will install as much application as possible from App Store, then the Homebrew, if both software repository can not find it, install it manually.

Apple store installation is the easiest one, once you log in and you will get all the application you have purchased.

brew are also relative easier, I even using brew to install Chrome and Firefox, via brew cask install google-chrome.

I would like to use percona instead of MySQL because, in China, Aliyun RDS is primarily based on percona, so I favor using the same software in dev env.

But you can not simply install percona directly.

Install mysql, then percona to avoid pitfall

brew install mysql

brew services start mysql

brew services stop mysql

brew remove mysql

brew install percona-server

brew services start percona-server

twitter cli need to rename tt instead of t

gem install t

mv /usr/local/bin/t /usr/local/bin/tt

Nokogiri using system library

brew install libxml2

# If installing directly

gem install nokogiri -- --use-system-libraries \

  --with-xml2-include=$(brew --prefix libxml2)/include/libxml2

# If using Bundle

bundle config build.nokogiri --use-system-libraries \

  --with-xml2-include=$(brew --prefix libxml2)/include/libxml2

bundle install

proxychains using Shadowsocks

To add Shadowsocks setting for proxychains.

# add below at end

# socks5  127.0.0.1 1080

vi /usr/local/etc/proxychains.conf

It will finally take 76Gb after installing all my needs, about 33% used, so I believe in buying a 256Gb SSD edition of rMBP should be enough for my next four years usage.

Deploy Ruby-china as a New Forum Software to Aliyun

• Permalink

This wiki based on the Ubuntu 14.04.5 LTS Server.

Prerequisites

apt-get update && apt-get upgrade

apt-get dist-upgrade

apt-get install software-properties-common htop curl git

memcached

apt-get install memcached

Redis

add-apt-repository ppa:chris-lea/redis-server

apt-get update && apt-get install redis-server

PostgreSQL

echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" > /etc/apt/sources.list.d/pgdg.list

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

apt-get update && apt-get install postgresql

apt-get install libpq-dev

imagemagick

apt-get install imagemagick

node.js

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

apt-get install -y nodejs build-essential

Oracle Java

add-apt-repository ppa:webupd8team/java

apt-get update

apt-get install oracle-java8-installer

Elasticsearch

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -

echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

apt-get update && apt-get install elasticsearch

update-rc.d elasticsearch defaults

service elasticsearch start

RVM

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

\curl -sSL https://get.rvm.io | bash -s stable

Ruby 2.4.0

rvm requirements

echo "ruby_url=https://cache.ruby-china.org/pub/ruby" > ~/.rvm/user/db # if need mirror.

rvm install 2.4.0 --disable-binary

rvm use 2.4.0 --default

echo "gem: --no-document" >> /etc/gemrc

echo "gem: --no-document" >> ~/.gemrc

gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/ # if need mirror

gem install bundler

Nginx

add-apt-repository ppa:nginx/stable

apt-get update && apt-get install nginx

Create deploy user

sudo adduser \

  --system \

  --shell /bin/bash \

  --gecos 'deploy user' \

  --group \

  --disabled-password \

  --home /data/www deploy

mkdir /data/www/deploy/.ssh

chmod 711 /data/www/deploy/

cp ~/.ssh/authorized_keys /data/www/deploy/.ssh/

chown deploy:deploy -R .ssh/

chmod 700 /data/www/deploy/.ssh/

Prepare setting files while doing cap deploy

cap production deploy

cp config/secrets.yml.default  config/secrets.yml

cp config/database.yml.default config/database.yml

cp config/config.yml.default   config/config.yml

cp config/redis.yml.default    config/redis.yml

cp config/puma.example.rb      config/puma-web.rb

cp config/elasticsearch.yml.default config/elasticsearch.yml

Change as necessory

vi /data/www/ruby-china/shared/config/database.yml

vi /data/www/ruby-china/shared/config/config.yml

vi /data/www/ruby-china/shared/config/secrets.yml

notice SECRET_KEY_BASE can get from rails secret

Create DB

sudo su - postgres

psql

CREATE ROLE deploy;

ALTER ROLE deploy LOGIN;

CREATE DATABASE ftghub_prod WITH ENCODING='UTF8' OWNER='deploy';

vi /etc/postgresql/9.6/main/pg_hba.conf

# TYPE  DATABASE        USER            ADDRESS                 METHOD

local  	ftghub_prod    	deploy 					                peer
service postgresql restart

Deploy

cap production deploy

Mapping nginx to puma

upstream ruby_china_backend {

    # This is the socket we configured in unicorn.rb

    server unix:///data/www/ruby-china/shared/tmp/sockets/puma.sock fail_timeout=0;

    keepalive 3;    

}

server {

    listen 80;

    server_name ftghub.com;

 
    root /www/ruby-china/current/public;

    gzip_static on; #to serve pre-gzipped version

 
    if (-f $document_root/system/maintenance.html) {

      rewrite ^(.*)$ /system/maintenance.html break;

    }

 
    location ~ (/assets|/system|/avatar.png|/favicon.ico|/*.txt) {

      access_log        off;

      expires           14d;

      gzip_static on;

      add_header  Cache-Control public;

    }

 
    location / {

      if ($host != 'ftghub.com') {

        rewrite ^/(.*)$ http://ftghub/$1 permanent;

      }

      proxy_redirect     off;

      proxy_set_header   Host $host;

      proxy_set_header   X-Forwarded-Host $host;

      proxy_set_header   X-Forwarded-Server $host;

      proxy_set_header   X-Real-IP        $remote_addr;

      proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;

      proxy_buffering    on;

      proxy_http_version 1.1;

      proxy_set_header   Upgrade $http_upgrade;

      proxy_set_header   Connection "Upgrade";

      proxy_set_header   X-Forwarded-Proto https;

      proxy_pass         http://ruby_china_backend;

      gzip on;

    }

 
    error_page 500 502 503 504 /500.html;

        location = /500.html {

        root /www/ruby-china/current/public;

    }

}

Running DB seed

bundle exec rake db:seed RAILS_ENV=production

Reindex ElasticSearch

Must ensure those model at least having one record each.

rake environment elasticsearch:import:model CLASS=Page FORCE=y RAILS_ENV=production

rake environment elasticsearch:import:model CLASS=Topic FORCE=y RAILS_ENV=production

rake environment elasticsearch:import:model CLASS=User FORCE=y RAILS_ENV=production

Monitor Sidekiq Service With Monit & Capistrano

• Permalink

Sidekiq as a background processing facility in Rails, usually more and more important during business grow, but it's relative quite possible failed to startup due to there is some code error, so here is my solution to monitor it.

I choose the monit instead of god beause monit is written in C and very small memory footprint. The capistrano-sidekiq also having a out of box support, but when you combine with monit, capistrano, sidekiq it still trick, so I feel writing a blog is worth.

Install monit

yum install monit # assure using CentOS

Configure monit

vi /etc/monitrc:

set daemon  60

 
set mailserver localhost

 
set mail-format {

     from: monit@your-domain.com

  subject: monit alert --  $EVENT $SERVICE

  message: $EVENT Service $SERVICE

 
Hi all people emails address;

 
Event happen at: $DATE

Action:          $ACTION

Host:            $HOST

Description: $DESCRIPTION

 
Your faithful employee,

 
Monit (robot)

Hint: when resolved, please replay all.

}

 
set alert receiver1@mail.com

set alert receiver2@mail.com

Configure in capistrano-sidekiq

At config/deploy/production.rb

set :sidekiq_monit_conf_dir, '/etc/monit.d' # for CentOS folder

set :sidekiq_monit_use_sudo, false

At Capfile

require 'capistrano/sidekiq'

require 'capistrano/sidekiq/monit' # to require monit tasks

Enable alert by customize the monit template.

rails g capistrano:sidekiq:monit:template

and change as below:

# Monit configuration for Sidekiq :  <%= fetch(:application) %>

<% processes_pids.each_with_index do |pid_file, idx| %>

check process <%= sidekiq_service_name(idx) %> with pidfile "<%= pid_file %>"

  if changed pid then alert

  start program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiq] %> <%= sidekiq_config %> --index <%= idx %> --pidfile <%= pid_file %> --environment <%= fetch(:sidekiq_env) %> <%= sidekiq_concurrency %> <%= sidekiq_logfile %> <%= sidekiq_queues %> <%= sidekiq_options_per_process[idx] %> -d'" with timeout 30 seconds

 
  stop program = "/bin/su - <%= @role.user %> -c 'cd <%= current_path %> && <%= SSHKit.config.command_map[:sidekiqctl] %> stop <%= pid_file %>'" with timeout <%= fetch(:sidekiq_timeout).to_i + 10  %> seconds

  group <%= fetch(:sidekiq_monit_group, fetch(:application)) %>-sidekiq

 
<% end %>

Deploya to generate minit config

cap production deploy

After that checking if /etc/monit.d having sidekiq conf file. or running cap sidekiq:monit:config to generate config.

Running monit at production

monit

To test the mail, using monit reload, if still can not received the mail, checking my postfix blog.

Building a Postfix Satellite SMTP Service to Avoid Active Mailer Run in Aliyun

• Permalink

Some situation is better avoid Sidekiq like reporting the exception mail, so it's always the best practices to building a Postfix Satellite SMTP service instead.

Install Postfix

yum install -y postfix cyrus-sasl-plain # cyrus-sasl-plain is no need for Ubuntu

yum erase -y sendmail* # In case Sendmail is installed

Configure

Similar to gmail or SendGrid, every mail service vendor offer different combination of setting, below is only apply to AliYun

/etc/postfix/main.cf change

# comment out below

#inet_interfaces = localhost

relayhost = [smtp.mxhichina.com]:587

smtp_sasl_auth_enable = yes

smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

smtp_sasl_security_options = noanonymous

smtp_tls_CApath = /etc/ssl/certs

smtp_use_tls = yes

smtp_generic_maps = hash:/etc/postfix/generic

mynetworks = 127.0.0.0/8

mydestination =

Create /etc/postfix/sasl_passwd

[smtp.mxhichina.com]:587    no-reply@domain.com:PASSWORD

Create /etc/postfix/generic

root@staging.domain.com   no-reply@domain.com

Create a database file

postmap /etc/postfix/sasl_passwd /etc/postfix/generic

Prevent non-root access:

chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db /etc/postfix/generic /etc/postfix/generic.db

Restart Postfix service

/etc/init.d/postfix restart # or service postfix restart

Debug

It's very common that postfix will failed to sent mail at first time, so add below to main.cf.

debug_peer_level = 3

debug_peer_list = smtp.mxhichina.com

And open another console to testing via:

echo "body of your email" | mail -s 'mail subject from console' -r 'no-reply@domain.com' guochunzhong@domain.com

You may also need postconf -n to review postfix configuration or postconf -d to review all setting.

YOu can also monitor the postfix activity via tail -f /var/log/syslog.

View Rails API and Guides in Localhost

• Permalink

Dash 3 switch to the subscription mode, if you just want to see the Rails document, in fact, it's relative easier to build in your own box.

Building Rails API Document

cd ~/git/oss/

git clone https://github.com/rails/rails

cd rails/

bundler install

rake rdoc

Building Rails Guides

cd guides/

rake guides:generate:html

Enable Apache

sudo apachectl start

Link Rails documents

cd /Library/WebServer/Documents/

sudo ln -s /Users/guochunzhong/git/oss/rails/doc/rdoc/ rails_api

sudo ln -s /Users/guochunzhong/git/oss/rails/guides/output/ rails_guides

View in localhost

visit Rails API at http://localhost/rails_api/

visit Rails Guides at http://localhost/rails_guides/

Installation Log on Ubuntu 14.04 LTS on vSphere 5.5

• Permalink

My log to build a Ubuntu server to fit my software stack usage, R & Ruby.

Install SSH

Upgrade Ubuntu and enable SSH server
sudo apt-get update

sudo apt-get upgrade

sudo apt-get install open-vm-tools

sudo apt-get install openssh-server

Install JDK

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update && sudo apt-get install oracle-java7-installer

Install rvm

sudo su - # switch to root if not yet

apt-get install curl

curl -L get.rvm.io | bash -s master # install rvm

logout

sudo su -

rvm requirements # to install all depends

Install Ruby

rvm install ruby-2.2.3

gem source --remove https://rubygems.org/

gem source --add https://gems.ruby-china.com/

gem update --system

apt-get install libgmp3-dev

echo "gem: --no-document" >> /etc/gemrc

gem update

gem install bundler

Install Oracle Client

apt-get install unrar

apt-get install libaio-dev # must need if you want to use sqlplus

mkdir -p /opt/oracle && cd /opt/oracle

unrar x /home/eric/instantclient-all-linux.x64-11.2.0.4.0.rar .

chmod +x instantclient_11_2/sqlplus

ln -s /opt/oracle/instantclient_11_2/libclntsh.so.11.1 /opt/oracle/instantclient_11_2/libclntsh.so

ln -s /opt/oracle/instantclient_11_2/sqlplus /usr/local/bin/sqlplus

echo "/opt/oracle/instantclient_11_2" >> /etc/ld.so.conf.d/oracle_instantclient.conf

# append below line to /etc/enviroment file

ORACLE_BASE=/opt/oracle

ORACLE_HOME=/opt/oracle/instantclient_11_2

TNS_ADMIN=/opt/oracle/network/admin

LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2

OCI_HOME=/opt/oracle/instantclient_11_2

OCI_LIB=/opt/oracle/instantclient_11_2

OCI_LIB_DIR=/opt/oracle/instantclient_11_2

OCI_INCLUDE_DIR=/opt/oracle/instantclient_11_2/sdk/include

NLS_LANG=AMERICAN_AMERICA.AL32UTF8

rvmsudo_secure_path=1

Install Oracle gems ruby-oci8

update the library and install ruby-oci8
reboot

sudo su -

ldconfig

mkdir -p network/admin

mv /home/eric/tnsnames.ora .

gem install ruby-oci8 # should be no any error here :-)

Install R

add R source
sudo vi /etc/apt/sources.list

# append below line to end of sources.list

# you can view mirror at http://cran.r-project.org/mirrors.html

deb http://cran.ism.ac.jp/bin/linux/ubuntu trusty/
prepare to install R 
apt-get update

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 51716619E084DAB9

# replace your NO_PUBKEY error above, if pause/block port, using 80 as below line

gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E084DAB9

gpg -a --export E084DAB9 | sudo apt-key add -

apt-get update # make sure no error here
install R
apt-get install r-base

Install ROracle

manual install the ROracle
wget http://cran.r-project.org/src/contrib/DBI_0.3.1.tar.gz

R CMD INSTALL DBI_0.3.1.tar.gz

wget http://cran.r-project.org/src/contrib/ROracle_1.2-1.tar.gz

R CMD INSTALL --configure-args='--with-oci-inc=/opt/oracle/instantclient_11_2/sdk/include --with-oci-lib=/opt/oracle/instantclient_11_2' ROracle_1.2-1.tar.gz

Install Passenger

Just following Passenger documentation

install phusionpassenger
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7

sudo apt-get install -y apt-transport-https ca-certificates

 
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main > /etc/apt/sources.list.d/passenger.list'

sudo apt-get update

 
sudo apt-get install -y nginx-extras passenger
vilink
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;

passenger_ruby /usr/local/rvm/gems/ruby-2.2.3/wrappers/ruby;

passenger_max_pool_size 30;

Install R studio server

Install R studio server

sudo apt-get install gdebi-core

wget https://download2.rstudio.org/rstudio-server-0.99.473-amd64.deb

sudo gdebi rstudio-server-0.99.473-amd64.deb
vilink
# Server Configuration File

www-address=127.0.0.1
vilink
server {

  listen       80;

  server_name  cvprstudio;

  server_name  cvprstudio.sandisk.com;

  client_max_body_size 10m;

  location / {

    proxy_pass http://127.0.0.1:8787;

    proxy_redirect http://127.0.0.1:8787/ $scheme://$host/;

  }

}

cd /etc/nginx/sites-enabled/

ln -s /etc/nginx/sites-available/cvprstudio cvprstudio

service nginx restart

Install RStudio Shiny Server

Install RStudio Shiny Server

run as root
R -e "install.packages('shiny', repos='https://cran.rstudio.com/')"

wget https://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.4.0.721-amd64.deb

gdebi shiny-server-1.4.0.721-amd64.deb
vilink
server {

  listen       80;

  server_name  shinyapp;

  server_name  shinyapp.sandisk.com;

  location / {

    proxy_pass http://127.0.0.1:3838;

    proxy_redirect http://127.0.0.1:3838/ $scheme://$host/;

  }

}

cd /etc/nginx/sites-enabled/

ln -s /etc/nginx/sites-available/shinyapp shinyapp

service nginx restart

Following patch process in previous blog to re-install the shiny

R CMD INSTALL shiny_0.12.2-server.tar.gz

Notes to Convert Galaxy Note 2 N7100 to Note 4 by Just Refresh ROM

• Permalink

I'm stucked quite a while when refresh a Galaxy Note II for friends, so I decide write something down:

  1. Wait some time after USB drive installed N7100 driver, something due to china poor network and gfw, the driver install from internet may failed.
  2. Root before install any 3rd party recovery, it's hardest part of change, I reference this for the Europe model, but you may need to found another way. This is must need before refresh any 3rd party recovery, because seems if not rooted, Knox will stop you do any further action.
  3. Install a 3rd recovery image, for example N7100 CWM Recovery v6.0.5.1
  4. Install customed notes 4 ROM for note 2, you can using this or mirror link