GPFS通用并行文件系统之CentOS上部署GPFS集群

GPFS通用并行文件系统之CentOS上部署GPFS集群GPFS文件系统能够横跨在所有主机上，分布在所有磁盘上，条带化读写，高性能。信令管理机制，并发性好。可配置failgroup组，可用性高。下面是GPFS集群的部署过程.1.环境准备：yum install -y compat-libstdc+-33 rpm-build kernel-headers kernel-devel imake gcc-c+ libstdc+ RedHat-lsb2.GPFS安装：多台服务器都要安装rpm -ivh gpfs.base-3.4.0-0.x86_64.rpmrpm -ivh gpfs.docs-3.4.0-0.noarch.rpmrpm -ivh gpfs.gpl-3.4.0-0.noarch.rpmrpm -ivh gpfs.msg.en_us-3.4.0-0.noarch.rpmrootWeb02_a base# rpm -qa|grep gpfsgpfs.msg.en_US-3.4.0-0gpfs.gpl-3.4.0-0gpfs.base-3.4.0-0gpfs.docs-3.4.0-03.GPFS升级多台服务器都要安装rpm -Uvhgpfs.base-3.4.0-21.x86_64.update.rpmrpm -Uvh gpfs.docs-3.4.0-21.noarch.rpmrpm -Uvh gpfs.gpl-3.4.0-21.noarch.rpmrpm -Uvh gpfs.msg.en_US-3.4.0-21.noarch.rpmrootWeb02_a update# rpm -qa|grep gpfsgpfs.gpl-3.4.0-21gpfs.msg.en_US-3.4.0-21gpfs.base-3.4.0-21gpfs.docs-3.4.0-214.编译GPFS源码多台服务器都要安装rootWeb02_a update# cd/usr/lpp/mmfs/src/rootWeb02_a src# makeLINUX_DISTRIBUTION=REDHAT_AS_LINUX AutoconfigrootWeb02_a src# make WorldrootWeb02_a src# make InstallImagesrootWeb02_a src# make rpm #生成rpm包，生成路径会有提示rootWeb02_a src# rpm -ivh /usr/src/redhat/RPMS/x86_64/gpfs.gplbin-2.6.18-308.el5-3.4.0-21.x86_64.rpmrootWeb02_a src# rpm -qa|grep gpfsgpfs.gpl-3.4.0-21gpfs.msg.en_US-3.4.0-21gpfs.gplbin-2.6.18-308.el5-3.4.0-21gpfs.base-3.4.0-21gpfs.docs-3.4.0-215.配置主机的时间同步如果服务器之间时间不同步，部署GPFS集群时会失败rootWeb02_a src# crontab -l#time sync by yangrong at 2014-1-24*/10 * * * * /usr/sbin/ntpdate /dev/null 2&1rootNagios update# crontab -l#time sync by yangrong at 2014-1-24*/10 * * * * /usr/sbin/ntpdate /dev/null 2&16.配置ssh免密钥登陆注：也可以配置rsh免密钥，且gpfs默认是使用rsh密钥登陆远端主机rootWeb02_a src# cd /root/.ssh/rootWeb02_a .ssh# ssh-keygen -t rsarootWeb02_a .ssh# cp id_rsa.pubauthorized_keysrootWeb02_a .ssh# ssh Web02_a #登陆自己测试rootWeb02_a .ssh# cat /etc/hosts43 Nagios36 Web02_arootWeb02_a .ssh# scp -r /root/.sshrootNagios:/root #把密钥拷贝到其它主机rootWeb02_a .ssh# ssh NagiosLast login: Fri Jan 24 13:59:19 2014 from3rootNagios # exitrootNagios src# ssh Web02_aWarning: Permanently added the RSA host keyfor IP address 36 to the list of known hosts.Last login: Fri Jan 24 15:03:44 2014 fromlocalhost.localdomainrootWeb02_a # exit7.配置GPFS环境变量rootWeb02_a .ssh# echo exportPATH=$PATH:/usr/lpp/mmfs/bin /etc/profilerootWeb02_a .ssh# source /etc/profilerootWeb02_a .ssh# mmfsmmfsadm mmfsd mmfsfuncs.Linux mmfsck mmfsenv mmfsmnthelp mmfsctl mmfsfuncs mmfsmount#接下来的操作只需要在一台服务器上操作即可。此时已配置好互信，所有的配置文件信息自动同步到其它的服务器上。8.创建集群rootWeb02_a .ssh# cat /tmp/gpfsfile Web02_a:quorum-managerNagios:quorum-managerrootWeb02_a .ssh# mmcrcluster -N/tmp/gpfsfile -p Web02_a -s Nagios -r /usr/bin/ssh -R /usr/bin/scp#默认GPFS使用rcp拷贝，使用rsh远程。此处修改远程方式和复制方式#查询命令：mmlscluster9.许可配置rootWeb02_a # mmchlicense server-accept -N Web02_a,Nagios注：服务节点或quorum节点用server,其它节点用client，client节点只有挂载文件系统的权限，不能更改配置。命令如：mmchlicense client -accept -N host_a,host_b10.配置nsd盘当前采用多个分区部署GPFS集群当前分区：rootWeb02_a # fdisk -l /dev/sdbDisk /dev/sdb: 1073 MB, 1073741824 bytes255 heads, 63 sectors/track, 130 cylindersUnits = cylinders of 16065 * 512 = 8225280bytesDevice Boot Start End Blocks Id System/dev/sdb1 1 13 104391 83 Linux/dev/sdb2 14 26 104422+ 83 Linux/dev/sdb3 27 39 104422+ 83 Linux/dev/sdb4 40 130 730957+ 5 Extended/dev/sdb5 40 52 104391 83 Linux/dev/sdb6 53 65 104391 83 Linux/dev/sdb7 66 78 104391 83 LinuxrootNagios # fdisk -l /dev/sdbDisk /dev/sdb: 1073 MB, 1073741824 bytes255 heads, 63 sectors/track, 130 cylindersUnits = cylinders of 16065 * 512 = 8225280bytesDevice Boot Start End Blocks Id System/dev/sdb1 1 13 104391 83 Linux/dev/sdb2 14 26 104422+ 83 Linux/dev/sdb3 27 39 104422+ 83 Linux/dev/sdb4 40 130 730957+ 5 Extended/dev/sdb5 40 52 104391 83 Linux/dev/sdb6 53 65 104391 83 Linux/dev/sdb7 66 78 104391 83 Linux编辑nsd配置rootWeb02_a # cat /tmp/nsdfile /dev/sdb1:Web02_a:dataAndMetadata:01:/dev/sdb2:Web02_a:dataAndMetadata:01:/dev/sdb3:Web02_a:dataAndMetadata:01:/dev/sdb5:Web02_a:dataAndMetadata:01:/dev/sdb1:Nagios:dataAndMetadata:02:/dev/sdb2:Nagios:dataAndMetadata:02:/dev/sdb3:Nagios:dataAndMetadata:02:#注：此时failgroup组1与组2磁盘数并不相同，但是磁盘不等是没关系的。两个组相当于raid1。rootWeb02_a # mmcrnsd -F /tmp/nsdfile-v no#生成NSD文件rootWeb02_a # cat /tmp/nsdfile# /dev/sdb1:Web02_a:dataAndMetadata:01:gpfs1nsd:dataAndMetadata:01:system# /dev/sdb2:Web02_a:dataAndMetadata:01:gpfs2nsd:dataAndMetadata:01:system# /dev/sdb3:Web02_a:dataAndMetadata:01:gpfs3nsd:dataAndMetadata:01:system# /dev/sdb5:Web02_a:dataAndMetadata:01:gpfs4nsd:dataAndMetadata:01:system# /dev/sdb1:Nagios:dataAndMetadata:02:gpfs5nsd:dataAndMetadata:02:system# /dev/sdb2:Nagios:dataAndMetadata:02:gpfs6nsd:dataAndMetadata:02:system# /dev/sdb3:Nagios:dataAndMetadata:02:gpfs7nsd:dataAndMetadata:02:system# /dev/sdb5:Nagios:dataAndMetadata:02:gpfs8nsd:dataAndMetadata:02:system11.配置仲裁盘#仲裁盘作用，当定义的仲裁盘有一半的磁盘不可用时，该集群不可用。另：有效磁盘数小于等于整个磁盘数一半时，整个文件系统不可用。rootWeb02_a # mmchconfig tiebreakerDisks=gpfs1nsd;gpfs2nsd;gpfs3nsdVerifying GPFS is stopped on all nodes .mmchconfig: Command successfully completedmmchconfig: Propagating the clusterconfiguration data to allaffected nodes. This is anasynchronous process.rootWeb02_a tmp# mmgetstate -arootWeb02_a tmp# mmgetstate -aNodenumber Node name GPFS state - 1 Web02_a active 2 Nagios active如果mmgetstate -a状态为down,请确保：防火墙关闭，两台服务器时间同步（注意时区也要一致），/etc/hosts中没有对应字段。gpfs错误日志路径：/var/adm/ras/mmfs.log.latest#修改节点IP mmchnode -daemon-interface=36 -NWeb02_a12.创建GPFS文件系统rootWeb02_a tmp# mmcrfs vol_data -F/tmp/nsdfile -B 256K -m 2 -r 2 -j cluster -T /vol_data -v noThe following disks of vol_data will beformatted on node Web02_a: gpfs1nsd: size 104391 KB gpfs2nsd: size 104422 KB gpfs3nsd: size 104422 KB gpfs4nsd: size 104391 KB gpfs9nsd: size 104391 KB gpfs10nsd: size 104422 KB gpfs11nsd: size 104422 KB gpfs12nsd: size 104391 KBFormatting file system .Disks up to size 6.4 GB can be added tostorage pool system.Creating Inode FileCreating Allocation MapsCreating Log FilesClearing Inode Allocation MapClearing Block Allocation MapFormatting Allocation Map for storage poolsystemCompleted creation of file system/dev/vol_data.mmcrfs: Propagating the clusterconfiguration data to allaffected nodes. This is anasynchronous process.13.挂载文件系统：rootWeb02_a ras# mmmount /vol_data -aFri Jan 24 20:04:25 CST 2014: mmmount:Mounting file systems .rootWeb02_a ras# df -hTFilesystem Type Size Used Avail Use% Mounted on/dev/sda3 ext3 19G 11G 7.0G 60% /dev/sda1 ext3 190M 12M 169M 7% /boottmpfs tmpfs 123M 0 123M 0% /dev/shm/dev/vol_data gpfs 814M 333M 481M 41% /vol_datarootNagios ras# df -hTFilesystem Type Size Used Avail Use% Mounted on/dev/sda3 ext3 6.6G 3.5G 2.8G 56% /dev/sda1 ext3 190M 12M 169M 7% /boottmpfs tmpfs 249M 0 249M 0% /dev/shm/dev/vol_data gpfs 814M 333M 481M 41% /vol_data安装完成。14.开机自启动mmchconfig autoload=yes或在/etc/rc.local中添加：/usr/lpp/mmfs/bin/mmstartup -a15.可靠性测试down掉Nagiogs服务器测试。看数据是否读取正常rootWeb02_a ras# cd /vol_data/rootWeb02_a vol_data# cp /etc/hosts .rootWeb02_a vol_data# lltotal 0-rw-r-r- 1 root root 375 Jan 26 09:25hostsrootWeb02_a vol_data# cat hosts # Do not remove the following line, orvarious programs# that require network functionality willfail. localhost.localdomain localhostbogon:1 localhost6.localdomain6 localhost636 Web02_a43 NagiosrootWeb02_a vol_data# ssh NagiosLast login: Sun Jan 26 09:08:28 2014 fromweb02_arootNagios # /etc/init.d/networkstop #down掉Nagios服务器网卡Shutting down interface eth0:rootWeb02_a vol_data# mmgetstate -a #查看状态已经有一个节点down掉Nodenumber Node name GPFS state - 1 Web02_a active 2 Nagios unknownrootWeb02_a vol_data# cat/vol_data/hosts #还能正常读取，保证集群的高可用。# Do not remove the following line, orvarious programs# that require network functionality willfail. localhost.localdomain localhostbogon:1 localhost6.localdomain6 localhost636 Web02_a43 NagiosGPFS通用并行文件系统之Python自动部署GPFS集群GPFS文件系统能够横跨在所有主机上，分布在所有磁盘上，条带化读写，高性能。信令管理机制，并发性好。可配置fail组，可用性高。下面是GPFS集群的自动安装部署python代码.注:该脚本只能自动识别到系统盘之外的物理磁盘，不会识别到分区。1、gpfs文件包目录结构脚本中都按照这个目录结构进行安装2、脚本执行方法输入的是拥有存储磁盘的主机名。按回车完成rootvuestor01 scripts# python auto.py Input nodes info. Usage:hosta/01. Press Enter is complete.Please input Node info: stor01/7check ip address success!Input nodes info. Usage:hosta/01. Press Enter is complete.Please input Node info: stor02/5执行成功后，自动挂载到/vol_data目录下。rootvuestor01 scripts# df -hTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 197G 1.6G 185G 1% /tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm/dev/sda1 ext4 504M 46M 433M 10% /boot/dev/vol_data gpfs 1.2T 918M 1.2T 1% /vol_datarootstor02 # df -hTFilesystem Type Size Used Avail Use% Mounted on/dev/sda2 ext4 197G 1.6G 185G 1% /tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm/dev/sda1 ext4 504M 46M 433M 10% /boot/dev/vol_data gpfs 1.2T 918M 1.2T 1% /vol_data3、gpfs自动安装python代码分析#!/usr/bin/python import datetime import os,sys,time import commands def output_log(): #定义输出log函数 time=datetime.datetime.now() with open(loginfo.txt,wb) as f: f.write(%s n%time) def check_ip(ipaddr): #校验IP有效性函数 import sys addr=ipaddr.strip().split(.) #print addr if len(addr) != 4: print check ip address failed! sys.exit() for i in range(4): try: addri=int(addri) except: sys.exit() if addri=0: pass else: print check ip address failed! sys.exit() i+=1 else: print check ip address success!def install_rpm():#定义安装gpfs相关软件包函数 print Install rpm packages. os.system(rpm -ivh ./dependency/nss-softokn-freebl*.rpm /loginfo.txt; rpm -ivh ./dependency/rsh*.rpm /root/loginfo.txt; rpm -ivh ./dependency/glibc*.rpm /root/loginfo.txt; rpm -ivh ./dependency/libgcc*.rpm /root/loginfo.txt; rpm -ivh ./dependency/ksh*.rpm /root/loginfo.txt; rpm -ivh ./dependency/compat-libstdc+*.rpm /root/loginfo.txt;) os.system(rpm -ivh ./base/gpfs.base*.rpm /root/loginfo.txt) os.system(rpm -ivh ./base/*noarch.rpm /root/loginfo.txt) os.system(rpm -Uvh ./update/*.rpm /root/loginfo.txt) #判断内核版本，安装相应版本的源码编译包 a,b=commands.getstatusoutput(uname -r|grep 220) c,d=commands.getstatusoutput(uname -r|grep 358) if (a = 0): commands.getstatusoutput(rpm -ivh ./rpm/gpfs.gplbin-2.6.32-220.el6.x86_64-3.4.0-21.x86_64.rpm) elif (c=0): commands.getstatusoutput(rpm -ivh ./rpm/gpfs.gplbin-2.6.32-358.el6.x86_64-3.4.0-21.x86_64.rpm) else: print cant support this kernel! sys.exit() #判断是否安装成功 num_status,num_result=commands.getstatusoutput(rpm -qa |grep gpfs|wc -l) if (num_result/tmp/part_%s;scp /tmp/part_%s %s:/root/ %(host,host,host,host_all0) os.system(ssh %s df -h /tmp/osdisk_%s;scp /tmp/osdisk_%s %s:/root/ %(host,host,host,host_all0) disk_status,disk_result=commands.getstatusoutput(cat /root/part_%s |awk print $4|grep -v name |grep -v 0-9|grep -v $%host) disk_a=disk_result.split(n) #从所有磁盘列表中移除系统盘 os_status,os_result=commands.getstatusoutput(cat /root/osdisk_%s |grep /$|awk print $1 |awk -F / print $3|awk -F 0-9 print $1%host) #print 111,disk_a,1111 disk_a.remove(os_result) avail_diskhost=disk_a #输出除系统盘外的所有空闲磁盘 print avail_disk.items() for a,b in avail_disk.items(): print On the host %ss disk:%a for i in b: print /dev/%s%i #配置nsd文件 def config_file(): print Configuring diskef. with open(/root/diskef,w) as f: num= #如果只有一台主机，则把这台主机的所有磁盘分成两个fail组，通过磁盘数取余来配置 if len(avail_disk.keys()=1: for n,m in avail_disk.items(): for i in range(len(m): num.append(i) if len(num)%2 =1: f.write(/dev/%s:%s:01:n%(mi,n) elif len(num)%2 =0: f.write(/dev/%s:%s:02:n%(mi,n) time.sleep(1) #如果主机数大于等于2台，则按主机分组两个fail组，通过主机数取余来配置。 elif len(avail_disk.keys()=2: n_num= for n in range(0,len(host_all): n_num.append(n) for m in avail_diskhost_alln: if len(n_num)%2 =1: f.write(/dev/%s:%s:01:n%(m,host_alln) elif len(n_num)%2=0: f.write(/dev/%s:%s:02:n%(m,host_alln) #for n,m in avail_disk.items(): # n_num.append(n) # for i in range(len(m): # #num.append(i) # if len(n_num)%2 =1: # f.write(/dev/%s:%s:01:n%(mi,n) # elif len(n_num)%2=0: # f.write(/dev/%s:%s:02:n%(mi,n) # time.sleep(1) print Done.#def sel_tiebreakerD(): #配置gpfs集群 def config_gpfs(): print Configuring gpfs:n #avail_disk.keys(): print Check crcluster. #print host_all,host_all #校验主机数，一台时无从服务器。 if len(host_all)=1: print mmcrcluster -N %s:quorum -p %s -r /usr/bin/ssh -R /usr/b in/scp %(host_all0,host_all0) crcluster_status,crcluster_result=commands.getstatusoutput(mmcrcluster -N %s:quorum -p %s -r /usr/bin/ssh -R /usr/bin/scp %(host_all0,host_all0) #主机数有多台时指定一台从服务器。 elif len(host_all)=2: print mmcrcluster -N %s:quorum,%s -p %s -s %s -r /usr/bin/ssh -R /usr/bin/scp %(host_all0,.join(host_all1:),host_all0,host_all1) crcluster_status,crcluster_result=commands.getstatusoutput(mmcrcluster -N %s:quorum,%s -p %s -s %s -r /usr/bin/ssh -R /usr/bin/scp %(host_all0,.join(host_all1:),host_all0,host_all1) #如果主机名为空，则退出 else: print Host list is empty. exit. sys.exit() #if crcluster_status!=0: # print Fail. # sys.exit() #else: # print Done. #校验集群配置 print Check cluster configuration. cluster_status,cluster_NAME=commands.getstatusoutput(mmlscluster|grep GPFS cluster name|awk print $4 ) cluster_nam