CEPH分布式存储部署.doc

CEPH分布式存储部署PS：本文的所有操作均在mon节点的主机进行，如有变动另有注释作者：网络技术部 徐志权日期：2014年2月10日VERSION 1.0更新历史：2014.2.10:首次完成ceph部署文档，块设备及对象存储的配置随后添加。一、部署前网络规划1.1 环境部署主机名公网IP（eth0）私网IP（eth1）操作系统运行服务node10101CentOS6.5mon、mdsnode20202CentOS6.5osdnode30303CentOS6.5osdu 操作系统使用CentOS6.5，因为系统已经包含xfs的支持可以直接使用不需要再次编译。u 由于CentOS6.5系统的内核为2.6.32，因此要关闭硬盘的写入缓存，若高于此版本不需要关闭。#hdparm -W 0 /dev/sdb 0u 本次部署一共有一个监控节点、一个元数据节点、两个数据节点，每个数据节点拥有两个硬盘作为数据盘。1.2 网络拓扑1.3 配置服务器、安装cephl 添加ceph的rpm库key#rpm -import /git/?p=ceph.git;a=blob_plain;f=keys/release.asc#rpm -import /git/?p=ceph.git;a=blob_plain;f=keys/autobuild.ascl 添加ceph-extras库#vi /etc/yum.repos.d/ceph-extrasceph-extrasname=Ceph Extras Packagesbaseurl=/packages/ceph-extras/rpm/centos6/$basearchenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_plain;f=keys/release.ascceph-extras-noarchname=Ceph Extras noarchbaseurl=/packages/ceph-extras/rpm/centos6/noarchenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_plain;f=keys/release.ascceph-extras-sourcename=Ceph Extras Sourcesbaseurl=/packages/ceph-extras/rpm/centos6/SRPMSenabled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_plain;f=keys/release.ascl 添加ceph库#rpm -Uvh /rpms/el6/noarch/ceph-release-1-0.el6.noarch.rpml 添加epel库#rpm -Uvh /pub/epel/6/x86_64/epel-release-6-8.noarch.rpml 安装ceph#yum update -y & yum install ceph -y二、部署ceph2.1创建一个集群创建一个集群，集群名为ceph，在mon节点，这里是主机node1，执行以下操作2.1.1获取一个uuid#uuidgen2.1.2编辑配置文件#vi /etc/ceph/ceph.confglobalauth_service_required = cephx#filestore_xattr_use_omap = true#该选项只针对ext4文件系统格式auth_client_required = cephxauth_cluster_required = cephxauth supported = nonefsid = a500528d-fc02-464e-b944-bddacbb5c3d1 #填入刚才获取到的uuidpublic network = /24cluster network = /24osd pool default size = 2osd pool default min size = 1max open files = 65535mon osd full ratio = .95mon osd nearfull ratio = .85monmon data = /data/$cluster-$idosdosd journal size = 1024osd journal = /data/$name/journalosd data = /data/$name2.2创建一个mon2.2.1编辑配置文件，添加mon参数#vi /etc/ceph/ceph.confmon.node1host = node1mon addr = 01:67892.2.2创建一个mon keyring#ceph-authtool /etc/ceph/ceph.mon.keyring -create-keyring -gen-key -n mon.2.2.3创建mon数据目录#mkdir -p /data/ceph-node1#ceph-mon -i node1 -mkfs -keyring /etc/ceph/ceph.mon.keyring2.2.4启动并查看集群状态#/etc/init.d/ceph start mon.node1#ceph s出现以下提示，启动成功：rootnode1 # ceph -s cluster a500528d-fc02-464e-b944-bddacbb5c3d1 health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds monmap e1: 1 mons at node1=01:6789/0, election epoch 2, quorum 0 node1 osdmap e1: 0 osds: 0 up, 0 in pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects 0 kB used, 0 kB / 0 kB avail 192 creating2.3创建一个osd首先为节点添加一块硬盘（这里是主机node2）PS：硬件支持热插拔的情况下可以使用以下命令，在线检测新增硬盘# echo - - - /sys/class/scsi_host/host2/scan2.3.1获取osd的ID#ceph osd create /记录得到的编号，该编号是下面创建的osd的ID2.3.2编辑配置文件#vi /etc/ceph/ceph.conf添加osd.0public addr = 02cluster addr = 022.3.3同步配置文档到节点node2#scp -r root01:/etc/ceph/ root02:/etc/2.3.4部署osd节点#ssh root02#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.0 1 20g /新加的硬盘为20g，并将所有空间划分为一个分区#mkfs -t xfs /dev/sdb1#mkdir /data/osd.0#mount /dev/sdb1 /data/osd.0#ceph-osd -i 0 -mkfs -mkkey#ceph auth add osd.0 osd allow * mon allow rwx -i /data/osd.0/keyring#ceph osd crush add osd.0 0.2 root=default host=node22.3.5启动osd进程#ceph-osd -i 02.3.6查看进程#ps -ef|grep ceph-osd以下提示启动成功rootnode2 # ps -ef|grep ceph-osdroot 1364 1 0 08:42 ? 00:00:07 ceph-osd -i 02.3.7查看osd状态#exit#ceph osd statosd添加成功rootnode1 # ceph osd stat osdmap e41: 1 osds: 1 up, 1 in2.3.8开机自动启动#vi /etc/fstab添加/dev/sdb1 /data/osd.0 xfs defaults 0 0# echo ceph-osd -i 0 /etc/rc.local2.4创建一个mds创建mds# ceph-mds -i node1 -n mds.node1 -c /etc/ceph/ceph.conf -m 01:6789检查mds状态，成功启动如下rootnode1 # ceph mds state19: 1/1/1 up 0=node1=up:active三、配置ceph客户端3.1确定操作系统内核3.1.1查看内核版本#uname -r若内核版本低于2.6.32时，建议升级到2.6.34或以上3.1.2更新内核，编译ceph模块这里以升级到3.10.29为例#wget /pub/linux/kernel/v3.x/linux-3.10.29.tar.xz#tar -Jxvf linux-3.10.29.tar.xz#cd linux-3.10.29#cp /boot/config-2.6.32-431.3.1.el6.x86_64 ./.config /将当前系统内核的配置文件复制到kernel源文件的目录内#make menuconfig依次选择下面选项，ceph必须编译为模块MFile systems-Network File Systems- Ceph distributed file system保存并退出#make#make modules_install#make install确定boot木有下存在以下文件rootnode1 linux-3.10.29# ll /boot/|grep 3.10initramfs-3.10.29.imgSystem.map - /boot/System.map-3.10.29System.map-3.10.29vmlinuz - /boot/vmlinuz-3.10.29vmlinuz-3.10.29更改grub.conf#vi /etc/grub.conf添加title CentOS (3.10.29) root (hd0,0) kernel /vmlinuz-3.10.29 ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /initramfs-3.10.29.img3.1配置文件目录共享客户端PS:确定已经运行一个ceph集群，并且集群包含至少一个或以上mds节点挂载ceph分布式存储文件目录共享以下在客户端上操作#yum install ceph-fuse -y#ceph-fuse -m 01:6789 /mnt测试写入1G文件#dd if=/dev/zero of=/mnt/test.1g bs=8192 count=125000四、ceph扩容4.1增加osd节点(以下操作在node1执行，即mon节点)根据第一章在新的节点node3服务器安装ceph，并配置好环境4.1.1获取osd的ID#ceph osd create /记录得到的编号，该编号是下面创建的osd的ID4.1.2编辑配置文件#vi /etc/ceph/ceph.conf添加osd.1public addr = 03cluster addr = 034.1.3同步配置文档到节点node3#scp -r root01:/etc/ceph/ root03:/etc/4.1.4部署osd节点#ssh root03#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.1 1 20g /新加的硬盘为20g，并将所有空间划分为一个分区#mkfs -t xfs /dev/sdb1#mkdir -p /data/osd.1#mount /dev/sdb1 /data/osd.1#ceph-osd -i 1 -mkfs -mkkey#ceph auth add osd.1 osd allow * mon allow rwx -i /data/osd.1/keyring#ceph osd crush add osd.1 0.2 root=default host=node24.1.5启动osd进程#ceph-osd -i 14.1.6查看进程#ps -ef|grep ceph-osd以下提示启动成功rootnode3 # ps -ef|grep ceph-osdroot 1364 1 0 08:42 ? 00:00:07 ceph-osd -i 04.1.7查看osd状态#exit#ceph osd statosd添加成功rootnode1 # ceph osd stat osdmap e41: 2 osds: 2 up, 2 in4.2增加mon(以下操作在主机node2执行)PS：每台主机只能为ceph集群启动一个mon进程PS2：集群中只有两个mon节点时，其中一个断开会导致集群不能访问，建议mon总数3个或以上。本次mon将部署在主机node24.2.3创建新的monitor数据目录#mkdir /data/ceph-node24.2.4部署新的mon#ceph auth get mon. -o /data/ceph-node2/ceph.mon.keyring#ceph mon getmap -o /data/ceph-node2/ceph.mon.map#ceph-mon -i node2 -mkfs -monmap /data/ceph-node2/ceph.mon.map -keyring/data/ceph-node2/ceph.mon.keyring#ceph mon add node2 02:67894.2.1启动新的mon#ceph-mon -i node2 -public-addr 02:6789五、检查ceph集群状态#ceph health/ceph健康状态#ceph status/ceph当前全部状态#ceph -w/实时监控ceph状态及变化#ceph osd dump /所有osd详细状态#ceph osd tree/osd所在位置，及状态#ceph quorum_status/mon优先级状态#ceph mon dump/mon节点状态#ceph mds dump/mds详细状态六、Openstack与Ceph整合6.1整合openstack与ceph6.1.1创建池#ceph osd pool create volumes 128#ceph osd pool create images 128#ceph osd pool create backups 1286.1.2获取ceph配置文件PS:应用于openstack所有节点#scp -r 3:/etc/ceph /etc6.1.3所有openstack节点安装ceph库文件#yum install -y python-ceph ceph6.1.4创建ceph用户#ceph auth get-or-create client.cinder mon allow r osd allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images#ceph auth get-or-create client.glance mon allow r osd allow class-read object_prefix rbd_children, allow rwx pool=images#ceph auth get-or-create client.cinder-backup mon allow r osd allow class-read object_prefix rbd_children, allow rwx pool=backups6.1.5获取ceph用户的key加入到ceph配置文件中PS:应用于openstack所有节点#ceph auth get-or-create client.glance |tee /etc/ceph/ceph.client.glance.keyring#chown glance:glance /etc/ceph/ceph.client.glance.keyring#ceph auth get-or-create client.cinder |tee /etc/ceph/ceph.client.cinder.keyring#chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring#ceph auth get-or-create client.cinder-backup |tee /etc/ceph/ceph.client.cinder-backup.keyring#chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring6.1.6配置计算节点的nova-compute使用ceph PS：应用于所有计算节点#ceph auth get-key client.cinder | tee client.cinder.key#uuidgen3a37c634-3b97-698c-c0b0-cb6f01fa4533#cat secret.xml EOF 3a37c634-3b97-698c-c0b0-cb6f01fa4533 client.cinder secret EOF#virsh secret-define -file secret.xmlSecret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 created将上述所得uuid值保存至/etc/nova/nova.confrbd_secret_uuid=3a37c634-3b97-698c-c0b0-cb6f01fa4533#virsh secret-set-value -secret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 -base64 $(cat client.cinder.key)6.1.7glance、cinder、nova的ceph配置Glance：default_store=rbdrbd_store_user=glancerbd_store_pool=imagesshow_image