公司新业务, 希望支持第三方开发者开发插件, 并实现插件代码的托管. 开发者可以上传代码, 审核通过后自动做资源分配, 代码发布等.
支撑公司内部开发人员和第三方开发者也需要关注更多的内容.
- 安全风险, 对于未知的第三方开发者. 要做好资源隔离并避免污染.
- 保证业务连续性.
- 自动实现代码发布, 支持回退/指定tag发布, 减少人工成本.
- 方便查看第三方应用产生的日志.
- 支持区分测试环境/生产环境.
业务上也需要支持框架, 如权限鉴定/一套k/v存储接口等.
运维的架构, 从来都应该是面向需求, 要满足业务需求, 但是又不能过早引入复杂的设计, 一步一步进行演变.
每个应用会分配给开发者一个仓库地址. 开发人员提交代码后, 在运维平台提交发布请求(仓库地址, 生成/测试环境), 管理员审核通过/拒绝, 自动发布代码到指定环境., 采用的实现方案如下:
通过docker来实现环境隔离.
通过nginx upstream机制做故障转移.
通过gitlab ci/cd实现代码发布接口.
通过运维平台提供资源分配, 生成配置文件, 调用发布接口, 提供日志接口, 修改子域名dns.
以第三方开发者的插件是nginx/php实现, 平台提供一个标准模板.
docker资源隔离:
提供Dockerfile
[nginx Dockerfile]
FROM alpine:3.3
MAINTAINER NGINX Docker Maintainers "docker-maint@nginx.com"
ENV NGINX_VERSION 1.10.1
ENV GPG_KEYS B0F4253373F8F6F510D42178520A9993A1C052F8
ENV CONFIG "\
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nobody \
--group=nobody \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_addition_module \
--with-http_sub_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_random_index_module \
--with-http_secure_link_module \
--with-http_stub_status_module \
--with-http_auth_request_module \
--with-http_xslt_module=dynamic \
--with-http_image_filter_module=dynamic \
--with-http_geoip_module=dynamic \
--with-http_perl_module=dynamic \
--with-threads \
--with-stream \
--with-stream_ssl_module \
--with-http_slice_module \
--with-mail \
--with-mail_ssl_module \
--with-file-aio \
--with-http_v2_module \
--with-ipv6 \
"
RUN \
apk add --no-cache --virtual .build-deps \
gcc \
libc-dev \
make \
openssl-dev \
pcre-dev \
zlib-dev \
linux-headers \
curl \
gnupg \
libxslt-dev \
gd-dev \
geoip-dev \
perl-dev \
&& curl -fSL http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz -o nginx.tar.gz \
&& curl -fSL http://nginx.org/download/nginx-$NGINX_VERSION.tar.gz.asc -o nginx.tar.gz.asc \
&& export GNUPGHOME="$(mktemp -d)" \
&& gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEYS" \
&& gpg --batch --verify nginx.tar.gz.asc nginx.tar.gz \
&& rm -r "$GNUPGHOME" nginx.tar.gz.asc \
&& mkdir -p /usr/src \
&& mkdir -p /var/log/nginx \
&& mkdir -p /var/cache/nginx/client_temp \
&& mkdir -p /var/cache/nginx/proxy_temp \
&& mkdir -p /var/cache/nginx/fastcgi_temp \
&& mkdir -p /var/cache/nginx/uwsgi_temp \
&& mkdir -p /var/cache/nginx/scgi_temp \
&& tar -zxC /usr/src -f nginx.tar.gz \
&& rm nginx.tar.gz \
&& cd /usr/src/nginx-$NGINX_VERSION \
&& ./configure $CONFIG --with-debug \
&& make \
&& mv objs/nginx objs/nginx-debug \
&& mv objs/ngx_http_xslt_filter_module.so objs/ngx_http_xslt_filter_module-debug.so \
&& mv objs/ngx_http_image_filter_module.so objs/ngx_http_image_filter_module-debug.so \
&& mv objs/ngx_http_geoip_module.so objs/ngx_http_geoip_module-debug.so \
&& mv objs/ngx_http_perl_module.so objs/ngx_http_perl_module-debug.so \
&& ./configure $CONFIG \
&& make \
&& make install \
&& rm -rf /etc/nginx/html/ \
&& mkdir /etc/nginx/conf.d/ \
&& mkdir -p /usr/share/nginx/html/ \
&& install -m644 html/index.html /usr/share/nginx/html/ \
&& install -m644 html/50x.html /usr/share/nginx/html/ \
&& install -m755 objs/nginx-debug /usr/sbin/nginx-debug \
&& install -m755 objs/ngx_http_xslt_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_xslt_filter_module-debug.so \
&& install -m755 objs/ngx_http_image_filter_module-debug.so /usr/lib/nginx/modules/ngx_http_image_filter_module-debug.so \
&& install -m755 objs/ngx_http_geoip_module-debug.so /usr/lib/nginx/modules/ngx_http_geoip_module-debug.so \
&& install -m755 objs/ngx_http_perl_module-debug.so /usr/lib/nginx/modules/ngx_http_perl_module-debug.so \
&& ln -s ../../usr/lib/nginx/modules /etc/nginx/modules \
&& strip /usr/sbin/nginx* \
&& strip /usr/lib/nginx/modules/*.so \
&& runDeps="$( \
scanelf --needed --nobanner /usr/sbin/nginx /usr/lib/nginx/modules/*.so \
| awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
| sort -u \
| xargs -r apk info --installed \
| sort -u \
)" \
&& apk add --virtual .nginx-rundeps $runDeps \
&& apk del .build-deps \
&& rm -rf /usr/src/nginx-$NGINX_VERSION \
&& apk add --no-cache gettext \
\
# forward request and error logs to docker log collector
&& ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
COPY nginx.conf /etc/nginx/nginx.conf
COPY nginx.vh.default.conf /etc/nginx/conf.d/default.conf
EXPOSE 80 443
CMD ["nginx", "-g", "daemon off;"]
docker用到的nginx配置文件:
user nobody;
worker_processes 4;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
use epoll;
worker_connections 10240;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
nginx.vh.default.conf:
server {
listen 80;
server_name localhost;
#charset koi8-r;
#access_log /var/log/nginx/log/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
[php Dockerfile]
FROM php:7-fpm-alpine
MAINTAINER foobar Docker Maintainers "docker@foobar.com"
# install php-core-extensions
RUN apk add --no-cache freetype libpng libjpeg-turbo freetype-dev libpng-dev libjpeg-turbo-dev libmcrypt-dev && \
docker-php-ext-configure gd \
--with-gd \
--with-freetype-dir=/usr/include/ \
--with-png-dir=/usr/include/ \
--with-jpeg-dir=/usr/include/ && \
NPROC=$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1) && \
docker-php-ext-install -j${NPROC} gd mcrypt && \
apk del --no-cache freetype-dev libpng-dev libjpeg-turbo-dev
生成环境nginx故障转移的实现方式:
默认使用docker环境处理请求, 当docker挂掉后, 可以切换到宿主机进行处理(宿主机处理有安全风险),用这个方式实施简单,不需要引入docker集群相关的内容.
upstream app1000239 {
server 10.47.49.233:11239;
server 127.0.0.1:10239 backup;
}
server {
listen 443 ssl;
server_name app1000239.foobar.com;
server_tokens off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location / {
proxy_pass http://app1000239;
}
access_log /data/logs/nginx/${host}_access.log combined;
}
server {
listen 127.0.0.1:10239;
root /data/web/app1000239/web/;
index index.html index.htm index.php;
location ~* \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location / {
index index.php;
try_files $uri $uri/ /index.php$is_args$args;
}
access_log /data/logs/nginx/app1000239_vm_access.log combined;
}
gitlab 实现ci/cd
gitlab支持通过在代码仓库中放置.gitlab-ci.yml来实现, 但是如果开发者直接支持管理.gitlab-ci.yml会有学习成本和较大的安全风险, 因此可以通过2个仓库来达到目的.
开发者可以上传代码到仓库A. 然后在运维平台提交发布申请.
仓库A对应的管理仓库AA. AA可以存放.gitlab-ci.yml, php/nginx的配置文件等. 运维平台可以调用仓库AA的代码发布来完成发布.
AA仓库的目录结构:
.gitlab-ci.yml (存放发布相关)
codes/ (这里是用的gitlab的子模块功能, 把仓库A当做子模块)
conf/ 这里存放渲染好的nginx/php配置文件, docker-compose文件
仓库AA可以设置一些默认参数, 并开启一个webhooks监听pipline event, 向运维平台发送发布成功/失败的结果.
附录:
[docker-compose.yaml文件]
version: '2'
services:
app-nginx:
image: registry.foobar.com/images/foobar-nginx
volumes_from:
- app-php
ports:
- "1.1.1.1:11038:80"
command: nginx -g 'daemon off;'
links:
- app-php
extra_hosts:
- "api.foobar.com:2.3.4.5"
app-php:
image: registry.foobar.com/images/foobar-php-fpm-gd
extra_hosts:
- "api.foobar.com:2.3.4.5"
volumes:
- /data/web/foobar_app1000038:/usr/share/nginx/html
- ./app-php.conf:/usr/local/etc/php-fpm.d/www.conf
- ./app-nginx.conf:/etc/nginx/conf.d/default.conf
- /etc/localtime:/etc/localtime:ro
- /etc/resolv.conf:/etc/resolv.conf:ro
[app-php.conf]
[www] user = www-data group = www-data listen = 127.0.0.1:9000 pm = dynamic pm.max_children = 100 pm.start_servers = 10 pm.min_spare_servers = 5 pm.max_spare_servers = 20
[app-nginx.conf]
server {
listen 80;
server_tokens off;
root /usr/share/nginx/html/web/;
index index.html index.htm;
location ~* \.php$ {
fastcgi_pass app-php:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
location / {
index index.php;
try_files $uri $uri/ /index.php$is_args$args;
}
location = /50x.html {
root /usr/share/nginx/html;
}
error_page 500 502 503 504 /50x.html;
access_log /var/log/nginx/access.log main;
}
[.gitlab-ci.yml]
image: foobar/open_docs_php:latest
before_script:
- ping git.foobar.com -w 2
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- ssh-add <(echo "$GIT_CLONE_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- cd $CI_PROJECT_DIR && git checkout . && git submodule update --init --force
- cd $CI_PROJECT_DIR/codes && echo ${commit_ref} && git fetch --all && git checkout ${commit_ref} .
stages:
- test
- build
- deploy
deploy_stage:
stage: deploy
script:
- if [ "${deploy_to}" == "development" ]; then echo "development" && rsync -vzrtog --delete $CI_PROJECT_DIR/codes/ $SSH_USER@$SSH_HOST:$DEV_BASE_DIR/ --exclude=.git; elif [ "${deploy_to}" == "production" ]; then echo "production" && rsync -vzrtog --delete $CI_PROJECT_DIR/codes/ $SSH_USER@$SSH_HOST:$DEV_BASE_DIR/ --exclude=.git; else echo "no variable provided"; exit 1; fi
运维平台实现:
开发者创建新应用后分配一个仓库账号,地址
分配域名:
创建针对测试/正式环境用的域名.
调用域名商接口配置对应的A记录.
创建对应的管理仓库.
初始化管理仓库:
创建.gitlab-ci.yml
创建子模块并放在codes/目录
渲染出的配置文件放到conf目录
创建仓库的变量(.gitlab-ci.yml里面会用到)
创建webhooks
调用gitlab ci发布代码, 推送conf里配置文件到宿主服务器, 使用docker-compose启动docker镜像, 并启动/重启外部nginx服务.
运维平台用的flask + sqlalchemy配合一些模块如python-gitlab, GitPython等实现.
整个代码/服务托管的运维方案这样就转起来了.