rancher部署测试(版本1.4.2).docx

基于rancher的容器解决方案2017年3月目录基于rancher的容器解决方案1一、架构介绍21.1 Rancher解决的问题21.2 Rancher架构31.3 Rancher基本概念和术语31.4 主机规划5二、安装包准备5三、docker环境准备7四、操作系统环境准备10五、搭建harbor企业级数据仓库115.1 搭建harbor企业级数据仓库(https,webUI)115.2配置docker client访问harbor镜像仓库认证13六、使用docker运行git lab服务13七、使用docker运行rancher server服务15八、Rancher Server中起容器基础架构服务17九、Rancher 节点起大数据组件测试18十、Rancher 节点配置ceph client19十一、Rancher 节点配置flocker存储插件19十二、测试使用flocker存储插件实现docker的有状态迁移22概述22测试环境23硬件测试环境23软件测试环境24部署方式24测试过程24测试内容24测试场景24测试方法24测试样例25一、 架构介绍Rancher+kubernetes+flocker+openstack+ceph1.1 Rancher解决的问题Rancher是一个完善的开箱即用的容器管理平台。 当企业开始去在生产环境部署 Docker 容器时， 面临的首个巨大挑战是如何把一组数量众多的开源技术集成在一起。如下图所示，容器管理涉及到的问题领域包括：存储、网络、监控、编排和调度Rancher 开发、集成和贡献了在生产环境中运行容器所需的所有必要技术。Rancher Labs集成和发布市场领先的容器编排和调度框架，如：DockerSwarm 和 Kubernetes，同时还开发了常用的应用目录，企业用户管理，访问控制，容器网络和持久存储的技术模块。这样为开发者和运维者同时提供了完整的功能和优异的用户体验。使用 Rancher，企业不需要再为追赶和集成那些，存在于快速发展的容器生态系统中的无穷无尽技术而忧虑。与之相反，他们可以在一次部署 Rancher 之后就把精力都放到如何快速开发应用，和对业务的提高改善上。Rancher特性： 100%开源 易于使用 企业级 基础架构无关性 同时支持 Swarm 和 Kubernetes1.2 Rancher架构1.3 Rancher基本概念和术语Flocker basics:Docker Data volumes are tied to a single node where Container is created. When Containers are moved across nodes, Data volumes dont get moved. Flocker mainly addresses this issue of moving the Data volumes along with the Container.Following picture from ClusterHQillustrates the above point.Following picture from ClusterHQ describes the architecture of Flocker. Flocker agent runs in each node and it takes care of talking to the Docker daemon and Flocker control service. Flocker control service takes care of managing the volumes as well as managing Flocker cluster. Currently supported storage includes Amazon AWS EBS, Rackspace block storage, EMC ScaleIO. Local storage is available on experimental basis. Both REST api and Flocker CLI is used to manage volumes as well as Docker containers.Flocker as Docker plugin:In the current released version of Flocker, it not possible to manage Docker volumes using Flocker with Docker frontend tools. With Docker and Flocker experimental release, it is possible for Docker to manage volumes using Flocker as a Data volume plugin. Following picture from ClusterHQ illustrates the Docker with Flocker plugin architecture. Data volumes will be managed using Docker front-end tools. This means all Docker Orchestration tools like Compose, Swarm can manage Docker volumes. Docker will support multiple plugins to manage Data volumes and Flocker will be 1 of the plugins. Flocker plugin will take care of managing data volumes and this includes migrating the volume associated with the Container when Container moves across hosts. Flocker will use the Container networking technology to talk across hosts, this can be native Docker networking or Docker networking plugins.1.4 主机规划主机名称IP备注rancher12rancher节点rancher23rancher39harbor01基于Web的镜像仓库gitlab版本管理二、 安装包准备下载最新版docker(centos-extra源)wget -S -c -r -np -L /centos/7/extras/x86_64/Packages/2.1下载rancher1.4.2版本Rancher镜像：Release v1.4.2Versions:rancher/server:v1.4.2rancher/agent:v1.2.0rancher/lb-service-haproxy:v0.5.9rancher-compose-v0.12.2rancher-v0.4.1rancher/healthcheck:v0.2.3rancher/net:v0.8.7rancher/net:holderrancher/metadata:v0.7.3 rancher/dns:v0.13.3rancher/network-manager:v0.4.5rancher/scheduler:v0.6.3Supported Docker VersionsDocker 1.10.3Docker 1.12.3-1.12.6docker-compose离线安装:下载/rancher/rancher/releases/tag/v1.4.2rancher-compose-linux-amd64-v0.12.2.tar.gzrancher-compose-v0.12.2，然后重新命名添加可执行权限即可：cp rancher-compose /usr/local/bin/ rancher-compose;chmod +x /usr/local/bin/docker-composedocker官方离线地址：/docker-compose/master/rancher CLI离线安装:/rancher/cli/releases/tag/v0.4.1rancher-linux-amd64-v0.4.1.tar.gzcp /root/rancher-v0.4.1/rancher /usr/local/bin/rancher私有仓库镜像：registry-2.4.1.tar下载harbor镜像仓库安装包git clone /vmware/harborgit服务器镜像：gitlab-gitlab-ce.tarmysql测试镜像：mysql5.7.10.tar/jthornber/thin-provisioning-toolsthin-provisioning-tools-master.zip./configuremakesudo make install三、 docker环境准备1.所有节点准备yum源(科大镜像站下载)centos-extra.repocentos-os.repoepel.repoyum install docker*选择docker存储驱动/engine/userguide/storagedriver/imagesandcontainers/In RedHat EnterpriseLinux7.1, OverlayFS is supported as a Technology Preview. There are currently two restrictions:1. It is recommended to useext4as the lower file system; the use ofxfsandgfs2file systems is not supported.SELinux is not supported, and to use OverlayFS, it is required to disable enforcing mode.1.Install the LVM2 and thin-provisioning-tools packages.The LVM2 package includes the userspace toolset that provides logical volume management facilities on linux.The thin-provisioning-tools package allows you to activate and manage your pool.yum install -y lvm2systemctl stop dockerrm -rf /var/lib/docker2.Create a physical volume replacing /dev/xvdf with your block device.pvcreate /dev/vdb3.Create a docker volume groupvgcreate docker /dev/vdb4.Create a logical volume named thinpool and thinpoolmeta.In this example, the data logical is 95% of the docker volume group size. Leaving this free space allows for auto expanding of either the data or metadata if space runs low as a temporary stopgap.lvcreate -wipesignatures y -n thinpool docker -l 95%VGlvcreate -wipesignatures y -n thinpoolmeta docker -l 1%VG5.Convert the pool to a thin pool.lvconvert -y -zero n -c 512K -thinpool docker/thinpool -poolmetadata docker/thinpoolmeta6.Configure autoextension of thin pools via an lvm profile.vi /etc/lvm/profile/fileSpecify thin_pool_autoextend_threshold value.The value should be the percentage of space used before lvm attempts to autoextend the available space (100 = disabled).thin_pool_autoextend_threshold = 80Modify the thin_pool_autoextend_percent for when thin pool autoextension occurs.The values setting is the percentage of space to increase the thin pool (100 = disabled)thin_pool_autoextend_percent = 20Check your work, your file file should appear similar to the following:cat /etc/lvm/profile/fileactivation thin_pool_autoextend_threshold=80thin_pool_autoextend_percent=207.Apply your new lvm profilelvchange -metadataprofile docker-thinpool docker/thinpool8.Verify the lv is monitored.lvs -o+seg_monitor9.vi /etc/docker/daemon.json storage-driver: devicemapper, storage-opts: dm.thinpooldev=/dev/mapper/docker-thinpool, dm.use_deferred_removal=true, dm.use_deferred_deletion=true 10.systemctl daemon-reloadsystemctl start dockersystemctl status dockerdocker info11.监控使用情况lvsvgsjournalctl -fu dm-event.serviceExamine devicemapper structures on the hostLsblk四、 操作系统环境准备1. 关闭所有节点防火墙2. 在所有节点上配置ntpyum install ntpntpdate u vi /etc/ntp.confsystemctl restart ntpdsystemctl status ntpdsystemctl enable ntpd3.在所有节点关闭selinux 4.安装ifconfig工具yum install net-tools5.配ssh无密码登录6.所有节点配置/etc/hosts2 rancher13 rancher29 rancher301 harbor gitlab9.所有节点/etc/sysconfig/docker文件添加ADD_REGISTRY=-add-registry harbor重启docker 服务并配置开机启动systemctl restart dockerSystemctl enable docker检查docker0网桥是否处于up状态10.更新操作系统包，否则执行命令会有报错，但不影响使用yum upgrade -yreboot删除升级后重新生成的repo文件cd /etc/yum.repos.d/rm -rf CentOS-Vault.repo CentOS-Sources.repo CentOS-Media.repo CentOS-fasttrack.repo CentOS-Debuginfo.repo CentOS-CR.repo CentOS-Base.repoyum clean all五、 搭建harbor企业级数据仓库5.1 搭建harbor企业级数据仓库(https,webUI)下载harbor离线安装包/vmware/harbor/releases生成自签名证书修改/etc/pki/tls/fdir = /etc/pki/CA # Where everything is keptmkdir -p /etc/pki/CA/cd /etc/pki/CA/openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crtopenssl req -newkey rsa:4096 -nodes -sha256 -keyout 0.key -out 0.csrtouch /etc/pki/CA/index.txtecho 01 /etc/pki/CA/serialecho subjectAltName = IP:0 fopenssl x509 -req -days 365 -in 0.csr -CA ca.crt -CAkey ca.key -CAcreateserial -extfile f -out 0.crtupdate-ca-trustmkdir -p /etc/docker/certs.d/harbor/etc/pki/CA/安装harborcd /root/harbor/common/config/nginx/certcp /etc/pki/CA/0.* .cd /root/harbor/common/config/nginxmv nginx.conf nginx.conf.bakcp /root/harbor/common/templates/nginx/nginx.https.conf /root/harbor/common/config/nginx/nginx.conf编辑nginx.confssl_certificate /etc/pki/CA/0.crt;ssl_certificate_key /etc/pki/CA/0.key; cd /root/harbor编辑harbor.cfgrootharbor harbor# cat /root/harbor/harbor.cfg |grep -v #|awk NFhostname = 01ui_url_protocol = httpsharbor_admin_password = Harbor12345auth_mode = db_authdb_password = root123self_registration = onuse_compressed_js = onmax_job_workers = 3 token_expiration = 30verify_remote_cert = oncustomize_crt = oncrt_country = CNcrt_state = Statecrt_location = CNcrt_organization = organizationcrt_organizationalunit = organizational unitcrt_commonname = harborcrt_email = 39220643ssl_cert = /etc/pki/CA/0.crtssl_cert_key = /etc/pki/CA/0.key./prepare./install.shupdate-ca-trustdocker login -u admin -p Harbor12345 0docker tag :5000/mariadb 01/libary/mariadbdocker push harbor/library/mariadb:latestdocker push 01/library/mariadb:latestharbor启停：修改harbor.cfg后./preparedocker-compose down docker-compose up -ddocker-compose ps5.2配置docker client访问harbor镜像仓库认证在所有运行docker服务的主机上将-insecure-registry从docker配置文件中移除，添加ADD_REGISTRY=-add-registry harbor登录rancher1,rancher2,rancher3主机创建目录mkdir -p /etc/docker/certs.d/0/复制ca.crt到client /etc/docker/certs.d/0/scp -p -r root0:/etc/pki/CA/*.crt /etc/docker/certs.d/0/systemctl restart docker六、 使用docker运行git lab服务1.确认gitlab-gitlab-ce.tar镜像已经上传到私有镜像仓库中并运行容器docker run -name=gitlab-ce -d -p 10022:22 -p 80:80 -restart always -volume /gitlab/config:/etc/gitlab -volume /gitlab/logs:/var/log/gitlab -volume /gitlab/data:/var/opt/gitlab :5000/gitlab-ce-p 用来指定docker容器映射宿主机的服务端口。80是web访问用的端口，22是ssh协议用的端口；volume 用来指定挂载目录。将config配置目录、data数据目录、logs日志目录挂载到宿主机上，以后备份起来也方便。编辑宿主机文件/gitlab/config/gitlab.rb 配置http协议所使用的访问地址external_url 配置ssh协议所使用的访问地址和端口 gitlab_railsgitlab_ssh_host = 00 gitlab_railsgitlab_shell_ssh_port = 10022在容器里执行gitlab-ctl reconfigure命令，或者重启容器以让新配置生效。登录git server创建project,访问地址/users/sign_in/root/myproject.gitssh:/git:10022/root/myproject.gitgitlab默认管理账号root/5iveL!fe参考网址/felix_yujing/article/details/52139070Add an SSH Key将14.5的公钥添加到gitlab项目中#Git global setupgit config -global administratorgit config -global user.email #Create a new repositorygit clone ssh:/git:10022/root/myproject.gitcd myprojecttouch README.mdgit add README.mdgit commit -m add READMEgit push -u origin master- templates OR kubernetes-templates OR swarm-templates |- cloudflare | |- 0 | | |- docker-compose.yml | | |- rancher-compose.yml | |- 1 | | |- docker-compose.yml | | |- rancher-compose.yml | |- catalogIcon-cloudflare.svg | |- config.ymlgit config -global administratorgit config -global user.email git clone ssh:/git:10022/root/swarm.gitcd myproject/mkdir templates (针对cattle模板)scp -p -r ./wordpress/ ./myproject/templcd ates/git add ./templates/git commitgit push -u origin master等几分钟，catalog自动同步git server的信息/rancher/v1.0/zh/catalog/七、 使用docker运行rancher server服务1. 确认如下rancher镜像已经上传至私有镜像仓库，注意版本rancher/server:v1.4.2rancher/agent:v1.2.0rancher/lb-service-haproxy:v0.5.9rancher-compose-v0.12.2rancher-v0.4.13. 安装mysql数据库创建rancher所用的mysql库yum install mariadb mariadb-server y修改mysql参数：mysqlddatadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.socksymbolic-links=0default-storage-engine = innodbinnodb_file_per_table = 1collation-server = utf8_general_ciinit-connect = SET NAMES utf8character-set-server = utf8max_connections=1000binlog_cache_size = 1Mbinlog_format = ROWexpire_logs_days = 10innodb_autoinc_lock_mode = 2innodb_buffer_pool_size = 300Minnodb_max_dirty_pages_pct = 99innodb_doublewrite = 1innodb_flush_log_at_trx_commit = 2innodb_lock_wait_timeout = 60innodb_locks_unsafe_for_binlog = 1innodb_stats_on_metadata = 0key_buffer = 256Mport=3306bind-address=2mysqld_safelog-error=/var/log/mariadb/mariadb.logpid-file=/var/run/mariadb/mariadb.pidsystemctl restart mariadb 启动mariadbsystemctl enable mariadb 开机自启动mysql_secure_installation 设置 root密码等相关mysql -uroot -ppasswordCREATE DATABASE IF NOT EXISTS cattle COLLATE = utf8_general_ci CHARACTER SET = utf8;GRANT ALL ON cattle.* TO cattle% IDENTIFIED BY cattle;GRANT ALL ON cattle.* TO cattlelocalhost IDENTIFIED BY cattle;4. 从私有镜像仓库及外部mysql数据库启动rancher server镜像docker run -d -restart=unless-stopped -p 8080:8080 01/library/rancher-server:v1.4.2 -db-host 2 -db-port 3306 -db-user cattle -db-pass cattle -db-name cattle检查登录rancher web界面2:8080/login是否正常(初始登录无密码)启用rancher访问控制添加镜像仓库添加自定义应用商店启动容器报错，web无法访问：/rancher/v1.2/en/faqs/server/#why-is-rancher-server-frozen-or-why-could-my-upgrade-have-failedWHY IS RANCHER SERVER FROZEN? OR WHY COULD MY UPGRADE HAVE FAILED?Caused by: liquibase.exception.LockException: Could not acquire change log lock解决：mysql use cattle;# Check that there is a lock in the tablemysql select * from DATABASECHANGELOGLOCK;+-+-+-+-+| ID | LOCKED | LOCKGRANTED | LOCKEDBY |+-+-+-+-+| 1 | | 2017-03-14 08:39:36 | 3572e1149fe3 (fe80:0:0:0:42:acff:fe11:2%eth0) |+-+-+-+-+# Update to remove the lock by the containermysql update DATABASECHANGELOGLOCK set LOCKED=, LOCKGRANTED=null, LOCKEDBY=null where ID=1;# Check that the lock has been removedmysql select * from DATABASECHANGELOGLOCK;+-+-+-+-+| ID | LOCKED | LOCKGRANTED | LOCKEDBY |+-+-+-+-+| 1 | | NULL | NULL |+-+-+-+-+1 row in set (0.00 sec)八、 Rancher Server中起容器基础架构服务1. 下载以下镜像并拷贝至所有rancher主机并加载到docker本地，注意版本正确,rancher Infrastructure Stacks 使用这些镜像rancher/healthcheck:v0.2.3rancher/net:v0.8.7rancher/net:holderrancher/metadata:v0.7.3 rancher/dns:v0.13.3rancher/network-manager:v0.4.5rancher/scheduler:v0.6.3已经下载好的镜像如下：2.加载镜像到本地并给镜像打标签docker load -input rancher_healthcheck_491349141109_v0.2.3.tardocker tag 491349141109 docker.io/rancher/healthcheck:v0.2.3docker load -input rancher_metadata_4509d408a416_v0.7.3.tardocker tag 4509d408a416 docker.io/rancher/metadata:v0.7.3docker load -input rancher_net_bb516596ce5a_holder.tardocker tag bb516596ce5a docker.io/rancher/net:holderdocker load -input rancher_network-manager_058abc0276fd_v0.4.5.tardocker tag 058abc0276fd docker.io/rancher/network-manager:v0.4.5docker load -input rancher_scheduler_e9fbe11760fd_v0.6.3.tardocker tag e9fbe11760fd docker.io/rancher/scheduler:v0.6.3docker load -input rancher_net_v0.8.7_332e26ab5cba.tardocker tag 332e26ab5cba docker.io/rancher/net:v0.8.7docker load -input rancher_dns_v0.13.3_a5567dd898f0.tardocker tag a5567dd898f0 docker.io/rancher/dns:v0.13.3最终结果如下：注意所有主机都必须执行加载和tag的脚本3. 登录rancher web界面，点击Infrastructure Stacks等待几分钟后，Infrastructure Stacks状态会自动恢复到正常4. 添加rancher主机在所有rancher客户端主机上执行docker pull 01/library/rancher-agent:v1.2.0docker tag 01/library/rancher-agent:v1.2.0 docker.io/rancher/agent:v1.2.0每一台新添加的rancher主机，都必须重复执行1，2步骤登录web控制台，点击基础架构主机添加主机：注意rancher server节点不需要填ip，非server节点需要填ipdocker run -e CATTLE_HOST_LABELS=name=rancher1&role=master -d -privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.0 2:8080/v1/scripts/7742DD4A3D87AD452FDC:1483142400000:gAZfsSGHQX5oejB0BCXje4Oc九、 Rancher 节点起大数据组件测试ZookeeperHadoophbase十、 Rancher 节点配置ceph client 1. 所有节点安装cephyum install ceph y拷贝ceph配置文件以及key文件至/etc/ceph/2. 所有节点加载rbd模块modprob rbdlsmod|grep rbd3. 执行ceph df命令确认rancher节点可以访问ceph服务十一、 Rancher 节点配置flocker存储插件 1. 在所有节点上确认已安装flocker rpm包在2.3一节中已经下载并安装flocker相关包，这里再确认一下：rootflocker-cli rancher_k8s# yum list|