GPFS系统安装配置和测试.docx

GPFS文件系统安装配置和测试目录GPFS文件系统安装配置和测试1一、目的2二、测试架构和环境2三、系统安装配置31.安装前的准备：32.系统和相关支持包的安装33.配置SSH 无密码登录41)编辑hosts和配置密钥42)关闭防火墙及selinux安全服务43)测试ssh无密码登录配置54.安装gpfs3.4安装包55.配置GPFS61)配置集群62)配置共享盘63)编辑共享盘配置文件74)使用mmcrnsd创建NSD75)授权节点license86)启动gpfs 文件系统87)创建GPFS 文件系统8四、优化GPFS 集群配置91.查看当前集群配置92.修改集群配置参数9五、变更节点配置91.增加节点92.增加NSD93.增加硬盘及NSD控制服务器104.增加硬盘到文件系统中10六、测试步骤和结果11测试环境：11具体测试过程：11从小文件读写测试结果总结：11使用大文件单个10G，进行测试：12大文件读写测试总结:12七、总结12一、 目的 测试和验证gpfs并行文件系统功能和性能，为将来部署准备二、 测试架构和环境服务器主机名ip地址安装系统安装软件角色备注DELL 2950GPFS710.8.8.136redhat 6.3GPFS 3.4GPFS 管理服务器HBA FC 4GDELL R420GPFS610.8.8.137redhat 6.3GPFS 3.4GPFS 管理服务器HBA FC 4GIBM 3850 虚拟机GPFS210.8.8.134redhat 6.5GPFS 3.4GPFS服务器千兆网卡IBM3850 虚拟机GPFS110.8.8.133redhat 6.5GPFS 3.4GPFS 服务器千兆网卡 架构图三、 系统安装配置1. 安装前的准备：在IBM 3512 划分 3个lun,空间容量分别为：380G,88 G,30 G，映射到两台服务器确保服务器和存储连接正常确保网络连接正常2. 系统和相关支持包的安装安装redhat enterprise server Linux 6.3 或6.5安装其它系统需要支持包：yum install libstdc+ compat-libstdc+-296 compat-libstdc+-33 libXp imake gcc-c+ kernel kernel-headers kernel-devel kernel-smp kernel-smp-devel xorg-x11-xauth 3. .配置SSH 无密码登录1) 编辑hosts和配置密钥编辑hostS文件如下所示10.8.8.133 gpfs110.8.8.134 gpfs210.8.8.136 gpfs710.8.8.137 gpfs6登录到各个系统gpfs2,gpfs6,gpfs7上分别执行以下命令#ssh-keygen -t dsa在gpfs2上执行以下命令#cat /.ssh/id_dsa.pub /.ssh/authorized_keys#ssh gpfs6 cat /.ssh/id_dsa.pub /.ssh/authorized_keys#ssh gpfs7 cat /.ssh/id_dsa.pub /.ssh/authorized_keys#scp /.ssh/authorized_keys gpfs6:/.ssh/authorized_keys#scp /.ssh/authorized_keys gpfs7:/.ssh/authorized_keys完成以后 测试gpfs2,gpfs6,gpfs7连通性 ssh gpfsx date注意因为版本不一致 gpfs2(6.5)版本需要执行 restorecon R v /root/.ssh2) 关闭防火墙及selinux安全服务 service iptables stop vi /etc/sysconfig/selinux # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted3) 测试ssh无密码登录配置登录gpfs1执行命令ssh gpfs6 datessh gpfs7 datessh gpfs3 date4. 安装gpfs3.4安装包执行./gpfs_install-3.4.0-0_x86_64 text-only进行安装Cd /usr/lpp/mmfsRpm ivh *.rpmtar -zxcvf解压gpfs-3.4.0-6.x86_64.tar.gz生成升级包gpfs.base gpfs.msg.en_US gpfs.docs gpfs.gp 安装所有的rpm软件包，并升级补丁 Rpm U *.rom 检查GPFS软件包安装情况rpm -qa|grep gpfs 增加.bash_profile PATH$PATH=$PATH:$HOME/bin:/usr/lpp/mmfs/bin 编译软件cd /usr/lpp/mmfs/src make Autoconfig make World make InstallImages5. 配置GPFS1) 配置集群在gpfs7上创建集群建立文件/tmp/gpfs.nodes内容为： gpfs7:quorum-manager gpfs6:quorum-manager 执行命令： /usr/lpp/mmfs/bin/mmcrcluster -n /tmp/gpfs.nodes -p gpfs7 -s gpfs6 -r /usr/bin/ssh -R /usr/bin/scp 显示集群信息mmlscluster GPFS cluster information= GPFS cluster name: dlokcluster.gpfs7 GPFS cluster id: 722837121801769718 GPFS UID domain: dlokcluster.gpfs7 Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scpGPFS cluster configuration servers:- Primary server: gpfs7 Secondary server: gpfs6 Node Daemon node name IP address Admin node name Designation - 1 gpfs7 10.8.8.136 gpfs7 quorum-manager 2 gpfs6 10.8.8.137 gpfs6 quorum-manager2) 配置共享盘首先检查硬盘连接正常 rootgpfs7 gpfs# fdisk -lDisk /dev/sda: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0006222c Device Boot Start End Blocks Id System/dev/sda1 * 1 64 512000 83 LinuxPartition 1 does not end on cylinder boundary./dev/sda2 64 17770 142224384 8e Linux LVMDisk /dev/sdb: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0003851b Device Boot Start End Blocks Id SystemDisk /dev/sdc: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0004d5b33) 编辑共享盘配置文件rootgpfs7 gpfs# vi disk_file/dev/sdh:gpfs7:gpfs6:dataAndMetadata:g1/dev/sdi:gpfs7:gpfs6:dataAndMetadata:g2/dev/sdq:gpfs7:gpfs6:dataAndMetadata:g34) 使用mmcrnsd创建NSD mmcrnsd -F /tmp/gpfs_disk -v yes 显示NSDMmlsnsd mrootgpfs6 home# mmlsnsd -m Disk name NSD volume ID Device Node name Remarks - g1 0A0808885379C4F2 /dev/sdc gpfs6 server node g1 0A0808885379C4F2 /dev/sdh gpfs7 server node g2 0A0808885379C4F3 /dev/sdd gpfs6 server node g2 0A0808885379C4F3 /dev/sdi gpfs7 server node g3 0A0808885379C4F4 /dev/sdl gpfs6 server node g3 0A0808885379C4F4 /dev/sdq gpfs7 server node5) 授权节点license mmchlicense server -accept N hpc1,hpc2,hpc36) 启动gpfs 文件系统 Mmstartup a 查看文件系统状态 Mmgeststate a rootgpfs6 home# mmgetstate -a Node number Node name GPFS state - 1 gpfs7 active 2 gpfs6 active 3 gpfs2 active7) 创建GPFS 文件系统# mmcrfs gpfsdir1 -F /home/GPFS/nsd -T /gpfsdir1 -A yes -K always -B 2m -E no -m 2 -M 2 -n 32 -Q no -r 2 -R 2 -S yes -v no注： 其mount点为/oradata, blocksize为2m;注意此参数已经创建，不可修改，blocksize一般要参照存储端LUN设置，以获得最优性能； -m 2 -M 2表示写两份metadata数据，-r 2 -R 2表示写两份data数据； 我们已经将数据盘的failgroup分别设为2、3，GPFS会自动将数据均衡写入不同failgroup中；四、 优化GPFS 集群配置1. 查看当前集群配置# mmlsconfigrootgpfs6 home# mmlsconfigConfiguration data for cluster dlokcluster.gpfs7:-myNodeConfigNumber 2clusterName dlokcluster.gpfs7clusterId 722837121801769718autoload nominReleaseLevel 3.4.0.7dmapiFileHandleSize 32pagepool 2048MadminMode centralFile systems in cluster dlokcluster.gpfs7:-/dev/g22. 修改集群配置参数 Mmchconfig pagepool=2048m五、 变更节点配置1. 增加节点 Mmaddnode -N gpfs12. 增加NSD 首先编辑磁盘配置文件Vi /root/gpfs/disk_file2/dev/sdb:gpfs1:dataAndMetadata:nsd43. m增加硬盘及NSD控制服务器 mmcrnsd -F /root/gpfs/disk_file2 -v no4. 增加硬盘到文件系统中mmadddisk g2 -F /root/gpfs/disk_file2显示文件系统 mmlsfs g2flag value description- - - -f 32768 Minimum fragment size in bytes -i 512 Inode size in bytes -I 32768 Indirect block size in bytes -m 1 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j cluster Block allocation type -D nfs4 File locking semantics in effect -k all ACL semantics in effect -n 32 Estimated number of nodes that will mount file system -B 1048576 Block size -Q none Quotas enforced none Default quotas enabled -filesetdf No Fileset df enabled? -V 12.10 (3.4.0.7) File system version -create-time Wed May 21 15:21:30 2014 File system creation time -u Yes Support for large LUNs? -z No Is DMAPI enabled? -L 4194304 Logfile size -E Yes Exact mtime mount option -S No Suppress atime mount option -K whenpossible Strict replica allocation option -fastea Yes Fast external attributes enabled? -inode-limit 520192 Maximum number of inodes -P system Disk storage pools in file system -d g1;g2;g3;nsd4 Disks in file system -A yes Automatic mount option -o none Additional mount options -T /gpfsdir1 Default mount point -mount-priority 0 Mount prio