Eric Guo's blog.cloud-mes.com

Hoping writing JS, Ruby & Rails and Go article, but fallback to DevOps note

Deploy next.js App to a Dedicateed Rocky Linux V8

Permalink

Create a new user

adduser changelog
sudo su - changelog
mkdir .ssh
chmod 700 .ssh
vi .ssh/authorized_keys # and paste your public key
chmod 600 .ssh/authorized_keys

Install Node.js 18 and Yarn

curl -sL https://rpm.nodesource.com/setup_18.x -o nodesource_setup.sh
sudo bash nodesource_setup.sh
sudo dnf install -y nodejs
sudo yum install gcc-c++ make
curl -sL https://dl.yarnpkg.com/rpm/yarn.repo | sudo tee /etc/yum.repos.d/yarn.repo
sudo yum install yarn

From DigitalOcean blog

Fix permissions for the deploy folder

sudo mkdir /var/www
cd /var/www
sudo mkdir changelog
sudo chown changelog:changelog changelog/

Install other tools

sudo yum install git
sudo dnf update
sudo dnf install epel-release
sudo dnf install htop
echo 'machine git.thape.com.cn login Eric-Guo password personal_token_here' >> ~/.netrc

Install PM2 & Nginx

sudo dnf install nginx
sudo npm install pm2@latest -g

Or alternatively install PM2 as system daemon

Copy Nginx configuration and disable firewall

cp /var/www/changelog/current/config/nginx/changelog.conf /etc/nginx/conf.d/
systemctl restart nginx
firewall-cmd --zone=public --add-service=https
firewall-cmd --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
systemctl restart firewalld

Install Rails App on Amazon Linux 2023 From Scratch

Permalink

Install pre-request

sudo dnf install nodejs
sudo dnf install nginx
sudo dnf install git

Install yarn

sudo curl -sL https://dl.yarnpkg.com/rpm/yarn.repo -o /etc/yum.repos.d/yarn.repo
sudo yum install yarn

Install rbenv and ruby-build

cd # as a ec2-user
git clone https://github.com/rbenv/rbenv .rbenv
echo 'eval "$(~/.rbenv/bin/rbenv init - bash)"' >> ~/.bash_profile
mkdir -p "$(rbenv root)"/plugins
git clone https://github.com/rbenv/ruby-build "$(rbenv root)"/plugins/ruby-build
git clone https://github.com/andorchen/rbenv-china-mirror.git "$(rbenv root)"/plugins/rbenv-china-mirror

Install ruby 3.2.2

sudo dnf install -y gcc rust patch make bzip2 openssl-devel libyaml-devel libffi-devel readline-devel zlib-devel gdbm-devel ncurses-devel
rbenv install 3.2.2
rbenv global 3.2.2
echo "gem: --no-document" > ~/.gemrc

Fix permission for deploy folder

sudo mkdir /var/www
cd /var/www
sudo mkdir oauth2id
sudo chown ec2-user:ec2-user oauth2id/

Do puma config

cap staging deploy
cap staging puma:config

Install additional tools

This maybe require by gems like unf_ext

sudo dnf install autoconf gcc-g++

Install CronTab which require by ACME.

sudo yum install cronie -y
sudo systemctl enable crond.service
sudo systemctl start crond.service
sudo systemctl status crond.service

Install ACME

git clone https://github.com/acmesh-official/acme.sh.git
cd ./acme.sh
./acme.sh --install -m your@email.com

Fix node error

Add to /etc/environment

NODE_OPTIONS="--openssl-legacy-provider"

PLease notice such NODE_OPTIONS will break VS code / Cursor if set in local.

New nginx conf

Sample nginx configure file
upstream puma_oauth2id_staging {
server unix:/var/www/oauth2id/shared/tmp/sockets/puma.sock fail_timeout=0;
}
server {
listen 80;
server_name sso-id.com;
return 301 https://$host$1$request_uri;
}
server {
listen 443;
ssl on;
ssl_certificate /home/ec2-user/.acme.sh/sso-id.com_ecc/fullchain.cer;
ssl_certificate_key /home/ec2-user/.acme.sh/sso-id.com_ecc/sso-id.com.key;
server_name sso-id.com;
root /var/www/oauth2id/current/public;
try_files $uri/index.html $uri @puma_sccsa_production;
client_max_body_size 4G;
keepalive_timeout 10;
error_page 500 502 504 /500.html;
error_page 503 @503;
location @puma_sccsa_production {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://puma_oauth2id_staging;
# limit_req zone=one;
access_log /var/www/oauth2id/shared/log/nginx.access.log;
error_log /var/www/oauth2id/shared/log/nginx.error.log;
}
location ^~ /assets/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
location ^~ /packs/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
location = /50x.html {
root html;
}
location = /404.html {
root html;
}
location @503 {
error_page 405 = /system/maintenance.html;
if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html break;
}
rewrite ^(.*)$ /503.html break;
}
if ($request_method !~ ^(GET|HEAD|PUT|PATCH|POST|DELETE|OPTIONS)$ ){
return 405;
}
if (-f $document_root/system/maintenance.html) {
return 503;
}
}

Upgrading PostgreSQL to 13 in Ubuntu 20.04 for Gitlab 16.0

Permalink

Changing the apt repository

By default, Ubuntu 20.04 installs PostgreSQL 12. If you want to upgrade to version 13, you need to use the official PostgreSQL repository.

To change the repository, run the following commands in the terminal:

# Create the file repository configuration:
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# Import the repository signing key:
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Update the package lists:
sudo apt-get update

Installing PostgreSQL 13

sudo apt-get -y install postgresql-13

Checking if PostgreSQL 13 is available

dpkg --get-selections | grep postgres
pg_lsclusters

Stopping and upgrading the database

sudo service postgresql stop
sudo pg_renamecluster 13 main main_pristine
sudo pg_upgradecluster 12 main

Starting the database

sudo service postgresql start
pg_lsclusters

Stopping the old database

sudo pg_dropcluster 12 main --stop
sudo pg_dropcluster 13 main_pristine --stop

It’s also safe to upgrade directly to version 15. For more information, you can refer to the original reference.

Running Stable Diffusion v2.1 Base on the Mac Studio and macOS 13 Ventura

Permalink

Stable Diffusion and MidJourney are the two of the most popular text-to-image models today.. MidJourney need to pay minimal $8 per month but Stable Diffusion is $9 per month but you can freely to run in local.

I have no NVidia GPU, but I have a Base model of Mac Studio, so I would like to try after Apple declared CoreML support Stable Diffusion.

Please make sure you have a good network and have upgraded macOS, aka macOS Ventura before beginning.

brew install miniconda
gh repo clone apple/coremltools
cd coremltools # just clone repo
conda init bash
./scripts/build.sh --python=3.10
xcode-select --install

Then running

python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" -i models/coreml-stable-diffusion-v1-4_original_packages -o output --compute-unit CPU_AND_GPU --seed 305
# If not running stable-diffusion-v1-4, the --model-version must be specified.
python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" --compute-unit CPU_AND_GPU -o output --seed 1106 -i models/coreml-stable-diffusion-2-1-base_original_packages --model-version stabilityai/stable-diffusion-2-1-base
# Fix the model path if an error is reported.
subl /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/python_coreml_stable_diffusion/pipeline.py
subl /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/python_coreml_stable_diffusion/coreml_model.py

Upgrade Brew Installation Python 3 Package

Permalink

I have a long history with python, recently due to I’m starting using cursor.so, I started to give a try again in python, but now in MacOS brew version.

List all installed package:

python3 -m pip list

Setuptools can be updated via pip, without having to re-brew Python:

python3 -m pip install --upgrade setuptools

Pip can be used to upgrade itself via:

python3 -m pip install --upgrade pip

List of all outdated packages:

pip list --outdated

Upgrade package one by one:

python3 -m pip install --upgrade ipython

Install via requirements.txt via proxy

python3 -m pip install -r requirements.txt --proxy socks5://127.0.0.1:6153

2023年的工作小结与计划

Permalink

之前在2020年定的计划已经过半,由于今年发生了很多重大变化,特别是AI,深度生成技术的突然出现,有必要重新订立一下新的计划。

这里订立计划还是和2020年一样的原则:一,计划不能和世界线的发展发生大的偏差和背离;二,订立的计划必须能够执行。

世界线

世界发生了深刻的变化,IT 的所有领域几乎都发生了停滞,中台的提出者阿里直接拆分了,鸿蒙失败了,唯一的亮点技术 chatGPT 的确可以提高程序员30%的生产力,但是无论从美国还是中国,这种本质上能够代替人类书写代码的技术,只会进一步降低开发成本,无法增加新的需求(除了 AI 技术本身的硬件需求以外)。

所以目前单位的工作核心任务还是降本增效,减少浪费,减少重复开发,确保现有平台得到充分的利用,能够让现有的平台,代码适应新的 AI 时代是工作的重点。

IT技术发展预测

语言

计算机语言和人类语言将发生融合现象,简洁的计算机语言,比如类型可省略的 ruby, javascript,python 会进一步流行,未来人人都是程序员。

新的计算机语言将极难出现,因为 chatGPT 技术不会对这些新的计算机语言进行训练,这就无形中大大提高了新语言的引入门槛。最终计算机语言会分裂为两个大类,高速语言和胶水语言,高速语言有C/C++, Rust,胶水语言有 ruby, javascript,python,如果两方面的特性都需要,那么选Go。

数据库

MySQL 和 PostgreSQL永远是唯二的选择。Oracel 在 M1 推出后的两年还没有支持苹果架构,其他数据库也完全无法同 MySQL 和 PostgreSQL 的人气和成熟度相比较。

跨平台

已经没有跨平台了,现在只有移动端平台和桌面端平台,移动端平台就是微信小程序或者移动端网页,桌面端管理类应用已经全部web化,专业类应用也会逐渐web化,因为Chrome在113版本发布WebGPU,长期看CAD应用也将Web化。操作系统平台已经没有人关注,如果真的有的话,也只会考虑生态与支持的软件能不能允许的更好。目前苹果的生态最生机勃勃,微软的生态存量软件最丰富。鸿蒙已经失败了。

跨平台的意义相比 Java 时代,意义已经很小了,web 平台本身就是跨平台的,而各个平台也同样可以通过一系列服务暴露平台本身的特有服务,所以未来没有操作系统之争,更多的是一种融合。(美国Windows的桌面市场占用了跌倒了57%

Web 平台

随着 chatGPT 完全使用 web 平台发布,随着Chrome 的能力越来越多(包括通知,驱动蓝牙硬件等等),Web 平台的重要性进一步提高。

但是新的 Web 开发框架不会再大规模出现了,因为新的框架 chatGPT 无法辅助编程,所以无法推广,这个情况和新语言类似。

人工智能

AI 会进入高速发展期,智能与智能的自动化应用和结合是重点,需要重点关注智能的落地和应用。

Serverless

越来越多的人发现,云计算的成本比自建机房高,现在已经开始了下云运动,Serverless更多的成为一种软件架构上的概念,而不是软件发布的概念。

DevOps / 运维自动化

自动化运维将只留下换硬盘的工人。Google 已经开始对 SRE 进行裁员了。

数据平台/中台

概念已经死了,阿里拆封了,搞中台的 N+1 了。数据平台也是个伪概念,只有数据是真的。

已经完成的工作小结

经过 4 年开发,基本各种需求都或多或少的建构了信息管理平台。当然这些系统由于建构时间有先后,建构人员有不同,质量良莠不齐,运营成本差异大,用户满意度也各不相同。我主力制作的基于 Rails 全栈框架的,在质量和运营成本上都基本做到了最优。用户满意度方面,本质上用户还是希望自己做的越少越好,系统做的越多越好,所以过去几年,满意度提升缓慢。由于现在我们代码的质量基础都很优异,所以还是留下了极大的改进空间,所以随后几年可以做的事很多。

经过 4 年基本建立了两个数据仓库,一个 Oracle 的,一个 MySQL 的。各个业务线的数据都能在单位内部,特别是 IT 部内部流通,MySQL的仓库性能也基本不用考虑太多,是今后的重点发展方向。

未来一年计划

进一步推动 API 互通

目前单位针对各个业务口,需求口的系统基本已经建立了,但是还没有相互调用,相互互通,在保证安全的前提下,通过API相互调用还需要加强。

探索引入深度生成技术

深度生成技术包括语言文字生成,图片生成等。在帮助用户减少工作量方面,大有可为。

逐步提升用户体验

随着前端开发技术的发展,用户将越来越挑剔,前端界面也将越来越复杂,单个页面所能做的事情也会越来越多,在保持低成本开发的同时,要提升用户的体验,就必须开发复杂的用户界面,需要在 Rails 做后端的基础上,逐步引入 Turbo 和 React 。

加强团队合作

谨慎评估员工的表现和贡献,提高员工工作效率和满意度,在需求清晰的情况下,尽量不打扰开发人员,帮助他们始终高效开发,同时也通过Gitlab统一代码管理平台,进行适度监控,适当鞭策开发。

Fix Yarn Packaging Signature Invalid - EXPKEYSIG 23E7166788B63E1E Yarn Packaging

Permalink

When running apt-get update at Ubuntu 20.04.5 LTS, got below errors:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Failed to fetch https://dl.yarnpkg.com/debian/dists/stable/InRelease The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
W: Some index files failed to download. They have been ignored, or old ones used instead.

To resolve just running below command.

export YARNKEY=yarn-keyring.gpg
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo gpg --dearmour -o /usr/share/keyrings/$YARNKEY
echo "deb [signed-by=/usr/share/keyrings/$YARNKEY] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

Deploy New Rails 7 and Ruby 3.2 App in Ubuntu 20.04

Permalink

Create new user

adduser pp_vendor
sudo su - pp_vendor
mkdir .ssh
chmod 700 .ssh
vi .ssh/authorized_keys # and paste your public key
chmod 600 .ssh/authorized_keys

Enable new user as sudo

sudo su -
cd /etc/sudoers.d/
echo "pp_vendor ALL=(ALL) NOPASSWD:ALL" > 85-pp_vendor-user

Install Rust

sudo apt install rustc
rustc --version # here is rustc 1.61.0

Install rbenv and Ruby 3.2.1

sudo apt-get install libyaml-dev
sudo apt install rbenv
sudo su - pp_vendor
mkdir -p "$(rbenv root)"/plugins
git clone https://git.thape.com.cn/rails/ruby-build.git "$(rbenv root)"/plugins/ruby-build
git clone https://git.thape.com.cn/rails/rbenv-china-mirror.git "$(rbenv root)"/plugins/rbenv-china-mirror
rbenv install 3.2.1
rbenv global 3.2.1
echo "gem: --no-document" > ~/.gemrc
eval "$(rbenv init -)" >> ~/.bash_profile # or past the `rbenv init -`
rbenv shell 3.2.1

Create MySQL DB user

mysql -u root -p
CREATE DATABASE pp_prod character set UTF8mb4 collate utf8mb4_bin;
CREATE USER 'pp_vendor'@'%' IDENTIFIED BY 'pp_vendor_password';
GRANT ALL ON pp_vendor.* TO 'pp_vendor'@'%';
FLUSH PRIVILEGES;

Link rbenv to make capistrano works

mkdir -p ~/.rbenv/bin
cd ~/.rbenv/bin
ln -s /usr/bin/rbenv rbenv

create deploy folder

cd /var/www
sudo mkdir pp_vendor
sudo chown pp_vendor:pp_vendor pp_vendor/
echo "machine git.thape.com.cn login Eric-Guo password token_of_personal" >> ~/.netrc

Manually Do Upload in ActiveStorage in Rails Console

Permalink

Upload file is at local:

rails console
a=Article.last
a.download_file.attach(io: File.open("/home/deployer/Manual.docx"), filename: "使用手册区.docx")

Upload file is at remote:

rails console
file = URI.open(remote_file_full_url)
a=Article.last
a.download_file.attach(io: file, filename: remote_file_name)