GPFS安装手册.doc

目录GPFS 基本安装过程1GPFS performance and Infiniband:4adding extra subnets（可选）:6与LUSTRE文件系统共存环境搭建：7Gpfs 基本卸载过程8GPFS 基本安装过程同步所有服务器时间，系统时间1. 准备并安装rpm包，升级包2. 准备依赖包，cp -rf /root/GPFS/6 /var/cache/yum/x86_64/3. yum -y install rsh ; yum -y install libc.so.6 ; yum -y install ksh ; yum -y install libstdc+.so.5; yum -y install compat-libstdc+-33-3.2.3-69.el6.x86_644. cd /root/GPFS;rpm -ivh *.rpm ; rpm -Uvh update/*.rpm5. Linux客户端ssh无密码登录ssh-keygen -t rsa 会创建/root/.ssh/id-ras.pub id-ras公钥：/root/.ssh/id-ras.pub私钥：/root/.ssh/id-ras把公钥复制到authorized_keys ,并复制/root/.ssh到所有机器上的/root/目录.If your GPFS cluster is configured to use SSH/SCP, it is suggested that you increase the value ofMaxStartups insshd_configto at least 1024.6. 设置环境变量vi /root/.bash_profilePATH=$PATH:/usr/lpp/mmfs/bin启用环境变量source /root/.bash_profile7. 编译安装cd /usr/lpp/mmfs/srcmake Autoconfig make Worldmake InstallImagescd /usr/lpp/mmfs/src; make LINUX_DISTRIBUTION=REDHAT_AS_LINUX Autoconfig; make World;make InstallImagesWhen using GPFS 3.2.1 and earlier there is not an rpm option to the GPFS GPL build process. In this case to distribute the kernel module binaries you can copy the files manually. You can see alist of these files when the make InstallImages process completes. These files are mmfslinux, mmfs26, lxtrace, dumpconv, tracedevcd, and they need to be copied to the /usr/lpp/mmfs/bin directory on the other nodes.命令如下：mkdir /tmp/gpfs_cmds/cp -f /usr/lpp/mmfs/src/bin/mmfslinux /tmp/gpfs_cmds/mmfslinux cp -f /usr/lpp/mmfs/src/bin/mmfs26 /tmp/gpfs_cmds/mmfs26 cp -f /usr/lpp/mmfs/src/bin/tracedev /tmp/gpfs_cmds/tracedev cp -f /usr/lpp/mmfs/src/bin/lxtrace.sial /tmp/gpfs_cmds/lxtrace.sial cp -f /usr/lpp/mmfs/src/bin/mmdumpfilocks.sial /tmp/gpfs_cmds/mmdumpfilocks.sial cp -f /usr/lpp/mmfs/src/bin/lxtrace /tmp/gpfs_cmds/lxtrace cp -f /usr/lpp/mmfs/src/bin/dumpconv /tmp/gpfs_cmds/dumpconv cd /tmptar -cf gpfs_cmds.tar ./gpfs_cmdson other nodes copy cmds to /usr/lpp/mmfs/bin8. 创建clustermmcrcluster -N hosta:quorum,hostb:quorum -p hosta -s hostb -r /usr/bin/ssh -R /usr/bin/scp9. mmchlicense server -accept -N hosta,hostb10. (可选)修改GPFS文件，支持device-mapper设备cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices修改文件使之能够输出这种格式mapper/mpath1 dmm ，mpath1为多路径磁盘for i in multipath -ll | grep CERESDAT | awk print $1doecho mapper/$i dmmdone11. NSD配置文件(dataAndMetadata、dataOnly、metadataOnly、descOnly)DiskUsage If a value is not specified, the disk usage remains unchanged: dataAndMetadata Indicates that the disk contains both data and metadata. This is the default. dataOnly Indicates that the disk contains data and does not contain metadata. metadataOnly Indicates that the disk contains metadata and does not contain data. descOnly Indicates that the disk contains no data and no file metadata. Such a disk is used solely to keep a copy of the file system descriptor, and can be used as a third failure group in certain disaster recovery configurations. For more information, see General Parallel File System: Advanced Administration and search on Synchronous mirroring utilizing GPFS replication. 格式：DiskName:ServerList:DiskUsage:FailureGroup:DesiredNameServerList中可指定主备nsd server,分散磁盘顺序以实现负载均衡rootnode01 # cat diskdef /dev/mapper/meta01:io01:io02:metadataOnly:failgroup01:/dev/mapper/meta02:io03:io04:metadataOnly:failgroup02:12. roothosta # vi diskdef /dev/sdc:hosta:hostb:01:/dev/sdd:hostb:hosta:02:/dev/sde:hosta:hostb:01:/dev/sdf:hostb:hosta:02:/dev/sdg:hosta:hostb:01:/dev/sdh:hostb:hosta:02:/dev/sdi:hosta:hostb:01:/dev/sdj:hostb:hosta:02:13. 创建NSDmmcrnsd -F diskdef -v no（可选）创建仲裁盘:mmchconfig tiebreakerDisks=gpfs1nsd;gpfs2nsd;gpfs3nsd14. 启动GPFS,并查看启动状态mmstartup -a;mmgetstate -a15. 创建文件系统 mmcrfs vol_data -F diskdef -B 64K -T /vol_data -v no16. 挂载文件系统mmmount vol_data -a17. (可选)变相替换磁盘过程（首先数据得有冗余，加粗字体为命令）mmchdisk vol_data stop -d gpfs12nsdmmlsdisk vol_datadisk driver sector failure holds holds storagename type size group metadata data status availability pool- - - - - - - - -gpfs6nsd nsd 512 1 Yes Yes ready up system gpfs7nsd nsd 512 2 Yes Yes ready up system gpfs8nsd nsd 512 1 Yes Yes ready up system gpfs9nsd nsd 512 2 Yes Yes ready up system gpfs10nsd nsd 512 1 Yes Yes ready up system gpfs11nsd nsd 512 2 Yes Yes ready up system gpfs12nsd nsd 512 1 Yes Yes ready down system gpfs13nsd nsd 512 2 Yes Yes ready up system vi add_diskdef /dev/sdj:hosta:hostb:01:mmcrnsd -F add_diskdefmmlsnsdFile system Disk name NSD servers - vol_data gpfs6nsd hosta,hostb vol_data gpfs7nsd hostb,hosta vol_data gpfs8nsd hosta,hostb vol_data gpfs9nsd hostb,hosta vol_data gpfs10nsd hosta,hostb vol_data gpfs11nsd hostb,hosta vol_data gpfs12nsd hosta,hostb vol_data gpfs13nsd hostb,hosta (free disk) gpfs14nsd hosta,hostbmmadddisk vol_data -F add_diskdef -r The following disks of vol_data will be formatted on node hosta: gpfs14nsd: size 586061784 KBExtending Allocation MapChecking Allocation Map for storage pool system 12 % complete on Fri Jan 4 07:11:45 2013100 % complete on Fri Jan 4 07:12:23 2013Completed adding disks to file system vol_data.mmadddisk: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.Restriping vol_data .Scanning file system metadata, phase 1 . 1 % complete on Fri Jan 4 07:12:37 2013100 % complete on Fri Jan 4 07:12:50 2013Scan completed successfully.Scanning file system metadata, phase 2 . 4 % complete on Fri Jan 4 07:12:53 2013100 % complete on Fri Jan 4 07:13:54 2013Scan completed successfully.Scanning file system metadata, phase 3 . Scan completed successfully.Scanning file system metadata, phase 4 . Scan completed successfully.Scanning user file metadata . 23.60 % complete on Fri Jan 4 07:14:16 2013 ( 487171 inodes 1702 MB)100.00 % complete on Fri Jan 4 07:14:46 2013Scan completed successfully.Done18. 增加GPFS客户端节点在安装好GPFS rpm情况下（安装方式同上面所提），运行命令：mmaddnode -N client01GPFS performance and Infiniband:linux磁盘预读访问优化：for i in ls /dev/mapper/ | grep -v control | grep -v Vol;do blockdev -setra 1024 /dev/mapper/$i;donefor i in ls /dev/mapper/ | grep -v control | grep -v Vol;do blockdev -getra /dev/mapper/$i;donemmchconfig prefetchThreads=96mmchconfig worker1Threads=144maxMBpS=3200pagepool=4096MverbsPorts verbsRdma enable|disablerootnode009 # ibstatusInfiniband device mlx4_0 port 1 status: default gid: fe80:0000:0000:0000:0030:4800:4413:0001 base lid: 0x4b sm lid: 0x35 state: 4: ACTIVE phys state: 5: LinkUp rate: 20 Gb/sec (4X DDR)Infiniband device mlx4_0 port 2 status: default gid: fe80:0000:0000:0000:0030:4800:4413:0002 base lid: 0x0 sm lid: 0x0 state: 1: DOWN phys state: 2: Polling rate: 10 Gb/sec (4X)mmchconfig verbsPorts=mthca0/1 mthca0/2mmchconfig versRdma=enableOnce these two parameters are set, if a node has these ports available they are used once the GPFS daemon has started. GPFS decides to use the IB RDMA connection instead of TCP/IP before looking at the subnets parameter. If an IB RDMA connection is available on a node it is used. If there are no IB ports available on a node then the subnets processing rules take effect.For enhanced performance when multiple verbPorts are defined on a node GPFS multiplexes the traffic over all of the available ports.rootnode009 # mmchconfig verbsPorts=mlx4_0mmchconfig: Command successfully completedmmchconfig: 6027-1371 Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.rootnode009 # mmchconfig verbsRdma=enablemmchconfig: Command successfully completedmmchconfig: 6027-1371 Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.rootnode009 # mmfsadm test verbs configVERBS RDMA Configuration: Status : started Start time : Thu May 5 11:47:45 2011 mmfs verbsRDMA : enable mmfs verbsPorts : mlx4_0 mmfs verbsRdmasPerNode : 0 mmfs verbsRdmasPerConnection : 8 mmfs verbsRdmaMinBytes : 8192 mmfs verbsRdmaMaxSendBytes : 16777216 mmfs verbsRdmaTimeout : 14 mmfs verbsLibName : libibverbs.so ibv_fork_support : true Max connections : 65536 Max RDMA size : 16777216 Max RDMAs per node max : 32 Max RDMAs per node curr : 32 Number of Devices opened : 1 Device : mlx4_0 vendor_id : 713 Device vendor_part_id : 25418 Device mem register chunk : 8589934592 (0x200000000) Device max_sge : 28 Device max_qp_wr : 16351 Device max_qp_rd_atom : 16 Max RDMAs per conn max : 32 Max RDMAs per conn curr : 8 Open Connect Ports : 1 Connect port : mlx4_0/1 lid : 75 state : IBV_PORT_ACTIVE path_mtu : IBV_MTU_2048rootnode009 # mmfsadm test verbs connNSD Client Connections: cl nidx destination status curr RW peak RW file RDs file WRs file RD KB file WR KB idx cookie - - - - -