CEPH分布式存储部署.

1、CEPH分布式存储部署PS:本文的所有操作均在 mon节点的主机进行，如有变动另有注释 作者：网络技术部徐志权日期：2014年2月10日更新历史：2014210:VERSION 1.0首次完成ceph部署文档，块设备及对象存储的配置随后添加。部署前网络规划1.1环境部署主机名公网 IP（ eth0）私网 IP（eth1）操作系统运行服务n ode10101Cen tOS6.5mor、mdsn ode20202Cen tOS6.5osdno de303172.16.100

2、.103Cen tOS6.5osd操作系统使用CentOS6.5，因为系统已经包含 xfs的支持可以直接使用不需要再次 编译。由于CentOS6.5系统的内核为2.6.32，因此要关闭硬盘的写入缓存，若高于此版本不需要关闭。#hdparm -W 0 /dev/sdb 0本次部署一共有一个监控节点、一个元数据节点、两个数据节点，每个数据节点拥 有两个硬盘作为数据盘。1.2网络拓扑1.3配置服务器、安装 ceph添加ceph的rpm库key#rpm -import/git/?p=ceph.git;a=blob_pla in; f=keys/release.asc#rp

3、m -import/git/?p=ceph.git;a=blob_pla in; f=keys/autobuild.asc添加 ceph-extras 库#vi /etc/yum.repos.d/ceph-extrasceph-extrasn ame=Ceph Extras Packagesbaseurl=/packages/ceph-extras/rpm/ce ntos6/$basearchen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ce

4、ph.git;a=blob_pla in; f=keys/release.ascceph-extras-no archn ame=Ceph Extras n oarchbaseurl=/packages/ceph-extras/rpm/ce ntos6/noarchen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_pla in; f=keys/release.ascceph-extras-sourcen ame=Ceph Extras Sour

5、cesbaseurl=/packages/ceph-extras/rpm/ce ntos6/SRPMSen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_pla in; f=keys/release.asc 添加ceph库#rpm -Uvh /rpms/el6/noarch/ceph-release-1-0.el6 .n oarch.rpm 添加epel库#rpm -Uvhhttp:/dl.fedoraproject.

6、Org/pub/epel/6/x86_64/epel-release-6-8. no arch.rpm 安装ceph#yum update -y & yum in stall ceph -y二、部署ceph2.1创建一个集群创建一个集群，集群名为ceph，在mon节点，这里是主机nodel，执仃以下操作2.1.1 获取一个 uuid#uuidge n2.1.2编辑配置文件#vi /etc/ceph/ceph.c onfglobalauth_service_required = cephx#filestore_xattr_use_omap = true#该选项只针对 ext4文件系统格式auth

7、_clie nt_required = cephx auth_cluster_required = cephx auth supported = nonefsid = a500528d-fc02-464e-b944-bddacbb5c3d1#填入刚才获取到的uuidpublic network = /24cluster network = /24osd pool default size = 2osd pool default min size = 1max ope n files = 65535mon osd full ratio = .95

8、mon osd n earfull ratio = .85mo nmon data = /data/$cluster-$idosdosd jour nal size = 1024osd journal = /data/$ name/jo urnalosd data = /data/$ name2.2创建一个mon2.2.1编辑配置文件，添加mo n参数#vi /etc/ceph/ceph.c onfmon.no de1host = no de1mon addr = 01:67892.2.2 创建一个 mon keyring#ceph-authtool /etc/cep

9、h/ceph.mon.keyring -create-keyring -gen-key -n mon.2.2.3创建mon数据目录#mkdir -p /data/ceph-no de1#ceph-m on -i no de1 -mkfs -keyri ng /etc/ceph/ceph.m on .keyri ng2.2.4启动并查看集群状态#/etc/i nit.d/ceph start mon.no de1#ceph - s出现以下提示，启动成功：root no de1 # ceph -scluster a500528d-fc02-464e-b944-bddacbb5c3d1health

10、HEALTH_ERR 192 pgs stuck in active; 192 pgs stuck un clea n; no osds monmap e1: 1 mo ns at n ode1=01:6789/0, electio n epoch 2, quorum 0 no de1osdmap e1: 0 osds: 0 up, 0 inpgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects0 kB used, 0 kB / 0 kB avail192 creat ing2.3创建一个osd首先为节点添加一块硬盘（这

11、里是主机n ode2）PS:硬件支持热插拔的情况下可以使用以下命令，在线检测新增硬盘# echo /sys/class/scsi_host/host2/sca n2.3.1 获取 osd 的 ID#ceph osd create/记录得到的编号，该编号是下面创建的osd的ID2.3.2编辑配置文件#vi /etc/ceph/ceph.c onf添加osd.0public addr =02cluster addr=02233同步配置文档到节点node2#scp -r root01:/etc/ceph/ root192.1

12、68.100.102:/etc/2.3.4部署osd节点#ssh root02#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.0 1 20g/新加的硬盘为 20g,并将所有空间划分为一个分区#mkfs -t xfs /dev/sdb1#mkdir /data/osd.0#mou nt /dev/sdb1 /data/osd.0#ceph-osd -i 0 -mkfs -mkkey#ceph auth add osd.0 osd allow * mon allow rwx -i /data/osd.0/ke

13、yring#ceph osd crush add osd.0 0.2 root=default host =no de22.3.5启动osd进程#ceph-osd -i 02.3.6查看进程#ps -ef|grep ceph-osd以下提示启动成功root no de2 # ps -ef|grep ceph-osdroot136410 08:42 ?00:00:07 ceph-osd -i 02.3.7查看osd状态#exit#ceph osd statosd添加成功root no de1 # ceph osd statosdmap e41: 1 osds: 1 up, 1 in2.3.8开机

14、自动启动#vi /etc/fstab添加/dev/sdb1/data/osd.0xfsdefaults0 0# echo ceph-osd -i 0 /etc/rc.local2.4创建一个mds仓U建mds# ceph-mds -i nodel -n mds. nodel -c /etc/ceph/ceph.co nf -m 01:6789 检查mds状态，成功启动如下root no de1 # ceph mds state19: 1/1/1 up 0=node仁up:active三、配置ceph客户端3.1确定操作系统内核3.1.1查看内核版本#un ame -r

15、若内核版本低于 2.6.32时，建议升级到2.6.34或以上3.1.2更新内核，编译ceph模块这里以升级到3.10.29为例#wget /pub/li nu x/kernel/v3.x/li nux-3.10.29.tar.xz#tar -Jxvf lin ux-3.10.29.tar.xz #cd lin ux-3.10.29#cp /boot/config-2.6.32-431.3.1.el6.x86_64 ./.config将当前系统内核的配置文件复制到kernel源文件的目录内#make menuconfig依次选择下面选项，ceph必须编译

16、为模块MFile systems-Network File Systems-Ceph distributed file systemr二ll-i str iuted file s1RPC: Enable dprintk debugging_jgging I FSsupportCaavarTcean e t wo r k保存并退出#make#make modules_ in stall#make in stall确定boot木有下存在以下文件root no de1 lin ux-3.10.29# ll /boot/|grep 3.10ini tramfs-3.10.29.imgSystem.ma

17、p - /boot/System.map-3.10.29System.map-3.10.29vml inuz - /boot/vmli nuz-3.10.29vmli nu z-3.10.29更改 grub.conf#vi /etc/grub.c onf添加title Cen tOS (3.10.29)root (hd0,0)kernel /vmlinuz-3.10.29 ro root=/dev/mapper/V olGroup-lv_root rd_NO_LUKSLANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=V olGroup/lv_swapSYSFONT=lat

18、arcyrheb-su n16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quietini trd /in itramfs-3.10.29.img3.1配置文件目录共享客户端mds节点PS:确定已经运行一个 ceph集群，并且集群包含至少一个或以上 挂载ceph分布式存储文件目录共享以下在客户端上操作#yum in stall ceph-fuse -y#ceph-fuse -m 01:6789 /mnt测试写入1G文件#dd if=/dev

19、/zero of=/mn t/test.1g bs=8192 cou nt=125000四、ceph扩容4.1增加osd节点 （以下操作在nodel执行，即mon节点）根据第一章在新的节点node3服务器安装ceph，并配置好环境4.1.1 获取 osd 的 ID#ceph osd create/记录得到的编号，该编号是下面创建的osd的ID4.1.2编辑配置文件#vi /etc/ceph/ceph.c onf添加osd.1public addr =03cluster addr=034.1.3同步配置文档到节点node3#scp -r root

20、01:/etc/ceph/ root03:/etc/4.1.4部署osd节点#ssh root03#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.1 1 20g/新加的硬盘为 20g，并将所有空间划分为一个分区#mkfs -t xfs /dev/sdb1#mkdir -p /data/osd.1#mou nt /dev/sdb1 /data/osd.1#ceph-osd -i 1 -mkfs -mkkey#ceph auth add osd.1 osd

21、allow * mon allow rwx -i /data/osd.1/keyring#ceph osd crush add osd.1 0.2 root=default host =no de24.1.5启动osd进程#ceph-osd -i 14.1.6查看进程#ps -ef|grep ceph-osd以下提示启动成功root no de3 # ps -ef|grep ceph-osdroot136410 08:42 ?00:00:07 ceph-osd -i 04.1.7查看osd状态#exit#ceph osd statosd添加成功root no de1 # ceph osd st

22、at osdmap e41: 2 osds: 2 up, 2 in4.2 增力口 mon （以下操作在主机node2执行）PS:每台主机只能为 ceph集群启动一个 mon进程PS2：集群中只有两个mon节点时，其中一个断开会导致集群不能访问，建议mon总数3个或以上。本次mon将部署在主机node24.2.3创建新的monitor数据目录#mkdir /data/ceph-no de24.2.4部署新的mon#ceph auth get mon. -o /data/ceph-no de2/ceph.m on .keyri ng#ceph mon getmap -o /data/ceph-no

23、 de2/ceph.m on. map#ceph-m on -i no de2 -mkfs -mon map /data/ceph-no de2/ceph.m on. map -keyri ng/data/ceph-no de2/ceph.m on .keyri ng#ceph mon add node2 02:67894.2.1启动新的mon#ceph-mon -i node2 -public-addr 02:6789五、检查ceph集群状态#ceph health/ceph健康状态#ceph status/ceph当前全部状态#ceph

24、-w/实时监控ceph状态及变化#ceph osd dump/所有osd详细状态#ceph osd tree/osd所在位置，及状态#ceph quorum_status/mon优先级状态#ceph mon dump/mon节点状态#ceph mds dump/mds详细状态六、Openstack 与 Ceph 整合6.1 整合 openstack 与 ceph6.1.1创建池#ceph osd pool create volumes 128#ceph osd pool create images 128#ceph osd pool create backups 1286.1.2获取ceph配

25、置文件PS:应用于openstack所有节点#scp -r 3:/etc/ceph /etc6.1.3所有openstack节点安装ceph库文件#yum in stall -y pyth on-ceph ceph6.1.4创建ceph用户#ceph auth get-or-create clie nt.ci nder mon allow r osd allow class-read object_prefixrbd_childre n, allow rwx pool=volumes, allow rx pool=images#ceph auth get-or-creat

26、e clie nt.gla nee mon allow r osd allow class-read object_prefix rbd_childre n, allow rwx pool=images#ceph auth get-or-create clie nt.ci nder-backup mon allow r osd allow class-read object_prefix rbd_childre n, allow rwx pool=backups6.1.5获取ceph用户的key加入到ceph配置文件中PS:应用于openstack所有节点#ceph auth get-or-c

27、reate clie nt.gla nee |tee /etc/ceph/ceph.clie nt.gla nce.keyri ng#chown glance:glance /etc/ceph/ceph.client.glance.keyring#ceph auth get-or-create clie nt.ci nder |tee /etc/ceph/ceph.clie nt.ci nder.keyri ng#chow n cin der:c in der /etc/ceph/ceph.clie nt.ci nder.keyri ng#ceph auth get-or-create cli

28、e nt.ci nder-backup |tee /etc/ceph/ceph.clie nt.c in der-backup.keyri ng#chow n cin der:c in der /etc/ceph/ceph.clie nt.ci nder-backup.keyri ng6.1.6配置计算节点的 nova-compute使用ceph PS：应用于所有计算节点#ceph auth get-key clie nt.ci nder | tee clie nt.ci nder.key#uuidge n3a37c634-3b97-698c-c0b0-cb6f01fa4533#cat sec

29、ret.xml EOF3a37c634-3b97-698c-c0b0-cb6f01fa4533clie nt.ci nder secret/nameEOF#virsh secret-define -file secret.xmlSecret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 created将上述所得 uuid值保存至/etc/nova/nova.conf rbd_secret_uuid=3a37c634-3b97-698c-c0b0-cb6f01fa4533#virsh secret-set-value -secret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 -base64 $(cat clie nt.ci nder.key)6.1.7glance、cinder、nova 的 ceph 酉己置Glance:default_store=rbd rbd_store_user=gla nee rbd_store_pool=images show image direct url=True Cin der:volume_