江西省农信社GPFS安装手册

江西省农信社GPFS安装手册

V1.0

目录

I.江西省农信社GPFS规划2

io1江西省农信社GPFS集群规划2

1.2集群Quorum方式3

lo3NSD磁盘组成3

1。4文件系统基本信息4

1.5GPFS系统环境示意图5

2.GPFS安装前准备5

2。I安装介质及补丁准备5

2.20penssl及Openssh6

2.3Linux节点安装ksh6

2.4Linux节点修改网络参数6

3.GPFS安装配置7

3.1配置ssh信任关系7

3。2AIX安装GPFS软件10

3.3Linux安装GPFS软件11

3。4GPFS集群配置12

4。GPFS日常维护19

4o1GPFS启动19

4。2GPFS停止19

4。3查看GPFS使用状况20

1.江西省农信社GPFS规划

1.1江西省农信社GPFS集群规划

IP地址节点名操作系统集群角色

10oO.I47o98gpfs98.AIXQuorum—manager

7100-02-021316

10。0.147.97gpls97.AIXQuorum-manager

7100-02-02-1316

10.0o147o49gpfs49.abs»jxnxsocomRedhat6»4forPowerclient

0gpfs50,ab$oRedhat6.4forPowerclient

10。0o147.51Redhat6.4forPowerclient

10o0.147.52gpfs52.abs.jxnxsccomRedhat6O4forPowerclient

3gpfs53.absoRedhat6.4forPowerclient

10.0。147.54gpfs54.Redhat6.4forPowerclient

10.0o147.55gpfs55,Redhat6。4forPowerclient

10.0-147。56gpfs56.abs.jxnxsacomRedhat6O4forPowerclient

10.0。147.57gpfs57.absoRedhat6.4forPowerclient

10.0,147o58gpfs58.abs<>Redhat6.4forPowerclient

10o0o147o59gpfs59.abs.jxnxsocomRedhat6。4forPowerclient

10c0.147。60gpfs60,abs.jxnxs®comRedhat6。4forPowerclient

1gpfs61.abs.jxnxsncomRedhat6»4forPowerclient

2gpfs62.abs«jxnxsocomRedhat6o4forPowerclient

10。0o147.63gpfs63.abs«Redhat6.4forPowerclient

10.0o147.64gpfs64,abs0Redhat6.4forPowerclient

5gpfs65,abSoRedhat6O4forPowerclient

10.0.147。66gpfs66,abSoRedhat6。4forPowerclient

7Redhat6<.4forPowerclient

10.00147o68gpfs68.abs.jxnxsocomRedhat6.4forPowerclient

10o0.147.69gpfs69.abs.jxnxsocomRedhat6»4forPowerclient

10.0,147。70gpfs70.abs.jxnxsocomRedhat604forPowerclient

10。0«147.71gpfs71.abs<>jxnxs.：comRedhat6.4forPowerclient

10。0o147.72gpfs72.absojxnxsccomRedhat6.4forPowerclient

3gpfs73.abs.jxnxsocomRedhat6.4forPowerclient

10.0.147。74gpfs74.abs<>Redhat6。4forPowerclient

10.0o147.75gpfs75,abs.jxnxsocomRedhat6。4forPowerclient

10.00147o76gpfs76.absojxnxsoRedhat6.4forPowerclient

com

10o0o147.77gpfs77,Redhat6.4forPowerclient

10.0。147o78gpfs78.abs.jxnxsvcomRedhat6.4forPowerclient

10.0。147.79gpfs79.abs.jxnxs=comRedhat6.4forPowerclient

0Redhat6O4forPowerclient

10。0o147。81gpfs81.abs0jxnxsocomRedhat6.4forPowerclient

10。Oo147o82gpfs82.abs<.Redhat6.4forPowerclient

10.0-147.83gpfs83,abs.jxnxsocomRedhat6.4forPowerclient

10。0.147o84gpfs84.abs.jxnxsncomRedhat6.4forPowerclient

10.0,147o85gpfs85.abs<>Redhat6.4forPowerclient

10.0o147.86gpfs86.Redhat6®4forPowerclient

10o0o147.87gpfs87.abs<>Redhat6。4forPowerclient

lUoU.14/»88gpfs88>Kedhat6.4forHowerclient

10.0。147o89gpfs89，abs.>jxnxsoRedhat604forPowerclient

com

10o0o147o90gpfs90)Redhat6。4forPowerclient

10.0.147.91gpfs91.abs.jxnx$ocomRedhat604forPowerclient

10.0o147.92gpfs924absojxnxs：.Redhat6.4forPowerclient

com

10.0n147o93gpfs93.abs»jxnxs»comRedhat6.4forPowerclient

10。0o147o94gpfs94.Redhat6.4forPowerclient

5gpfs95,Redhat6.4forPowerclient

6gpfs96>Redhat6。4forPowerclient

1.2集群Quorum方式

集群Quorum方式采用NodeQuorum+Tiebreakerdisks,两个AIX节点加上一块专用的

SAN磁盘，目前只有•个AIX节点,第二个AIX节点和SAN磁盘需要后续添加。

1.3NSD磁盘组成

PV_NAMESIZENSD_NAMEFS_NAMENSDSERVERSLIST

hdisk2500Gbjxrccb_nsdlgpfsgpfs98.abs.jxnxsocom

hdisk3500Gbjxrccb_nsd2gpfsgpfs98.abso

#Ispv

hdiskoOOf8a6a659b530bcrootvgactive

hdisklOOf8a6a46e48f429rootvgactive

hdisk2nonejxrccb_nsdl

hdisk3nonejxrccb_nsd2

#mmlsnsd

FilesystemDisknameNSDservers

gpfsjxrccb_nsdlgpfs98.abs.^

gpfsjxrccb_nsd2

1.4文件系统基本信息

#mmlsfsall

Filesystemattributesfor/dev/gpfs:

flagvaluedescription

16384Minimumfragmentsizeinbytes

512Inodesizeinbytes

16384indirectblocksizeinbytes

1D_e_fa__ult_numbe_rofmetadatareplicas

Maximumnumberofmetadatareplicas

Defaultnumberofdatareplicas

Maximum,n_umbe_rofdatareplicas

clusterBlockallocationtype

nfs4Filelockingsemanticsineffect

allACLsemanticsineffect

-n128Estimatednumberofnodesthatwillmountfilesysterr

-B524288Blocksize

noneQuotasenforced

noneDefaultquotasenabled

--filesetdfnoFilesetdfenabled7

-v13.23<、3-5.0.7；，F1lesystemversion

--create-timeTueNov1920:45:562013Filesystemcreationtime

-uyessupportforlargeLUNS?

-znoisDMAPIenabled?

-L4194304Logfilesize

yesExactmtimemountoption

-SnoSuppressatimenwuntoption

-Kwhenpossiblestrictreplicaallocationoption

--fasteayesFastexternalattributesenabled?

--inode-limit13568000Maximumnumberofinodes

-PsystemDiskstoragepoolsinfilesystem

jxrccb.nsdl;jxrccb_nsd2Disksinfilesystem

--perfileset-quotanoPer-filesetquotaenforcement

-AyesAutomaticmountoption

-ononeAdditionalmountoptions

-T/gpfsDefaultmountpoint

--niount-priority0Mountpriority

1.5GPFS系统环境示意图

IP网络

GPFSServer域

2台Powcr720服务器使用AIX全分区，组成2个GPFSServer节点

y

存储资源池

ip网络

SAN网络

GPFSServer:2个AIX分区连接V7000

SANStorage：SVC

GPFSClient:48个PowerLinux分区

GPFS管理网络:千兆交换机

2.GPFS安装前准备

2.1安装介质及补丁准备

安装包：GPFSv3.5o0<.0

补丁：GPFS—3.5.0。II

2.2Openssl及Openssh

确认AIX上openssl及openssh是否已安装，Linux默认已经安装;Islpp—1|grepopen

#IsIpp-1|grepopen60

.

openssh.base.client600.6100COMMITTEDopensecureShellCommands

.

openssh.base.server600.6100COMMITTEDopensecureShel1server

openssh.man.en_us6.00.6100COMMITTEDOpensecureShell

.

openssh.msg.en_uso90.6100COMMITTEDopensecureShellMessages-

.

openssl.baseo98.2400COMMITTEDopensecuresocketLayer

openssl.licenseo.98.2400COMMITTEDOpensecuresocketLicense

.

openssl.man.en_us608.2400COMMITTEDopensecuresocketLayer

openssh.base.client6.00.6100COMMITTEDOpensecureShel1Commands

openssh.base.servero.90.6100COMMITTEDOpensecureshellserver

openssl.base.8.2400COMMITTEDopensecuresocketLayer

2.3Linux节点安装ksh

#rpm-ivhksh-20100621—19.el6-ppc64.rpm

[root@serverletc]#rpm-qalgrepksh

ksh-20100621-19.el6.ppc64

2.4Linux节点修改网络参数

修改/etc/sysctl.conf文件，需修改的网络参数如下:

#increaseLinuxTCPbufferlimits

net.core.rmem_max=8388608

nArp.wmpm_mAX=

#increasedefaultandmaximumLinuxTCPbuffersizes

net.ipv4.tcp_rmem=40962621448388608

net.ipv4.tcp_wmem=40962621448388608

#increasemaxbacklogtoavoiddroppedpackets

dev_max_backlog=2500

[roottaserverletc]#catsysctl.conf

#KernelsysctlconfigurationfileforRedHatLinux

#

#Forbinaryvalues,0isdisabled,1isenabled,seesysctl(8)and

#sysctl.conf(5)formoredetails.

#ControlsIPpacketforwarding

net.ipv4.ip_forward-0

#Controlssourcerouteverification

net.ipv4.conf.default.rp_filter-1

#Donotacceptsourcerouting

net.ipv4.conf.default.accept_source_route=0

#ControlstheSystemRequestdebuggingfunctionalityofthekernel

kernel.sysrq-0

#controlswhethercoredumpswillappendthePIDtothecorefilename.

#usefulfordebuggingnulti-threadedapplications.

kernel.core_uses_pid=1

#controlstheuseofTCPsyncookies

net.ipv4.tcp_syncookies»1

#DisablenetfiIteronbridges.

net.bridge.bridge-nf-call-ip6tables-0

net.bridge.bridge-nf-call-iptables-0

net.bridge.bridge-nf-call-arptables=0

#controlsthedefaultnaxmimumsizeofamesagequeue

kernel.msgmnb=65536

#Controlsthemaximumsizeofamessage,inbytes

kernel.msgmax=65536

#controlsthemaximumsharedsegmentsize,inbytes

kernel.shmmax=137438953472

#controlsthemaximumnumberofsharedmemorysegments,inpages

kernel.shmall=536870912

#increaseLinuxTCPbufferlimits

net.core.rmem_max-8388608

net.core.wmemmax-8388608

I#increasedefaultandnaximumLinuxTCPbuffersizes

net.ipv4.tcp_rmem=40962621448388608

net.ipv4.tcp_wmem=40962621448388608

卜increasemaxbacklogtoavoiddroppedpackets

het.devjnaxbackloa=2500

修改完毕后执行sysctl—p/etc/sysctl.conf让配置生效

3.GPFS安装配置

3.1配置ssh信任关系

■在节点上生成并配置ssh密钥对

#ssh-keygen-trsa

#ssh-keygen-trsa

Generatingpublic/privatersakeypair.

Enterfileinwhichtosavethekey(//.ssh/id_rsa):

Createddirectory'〃.ssh'.

Enterpassphrase(emptyfornopassphrase):

Entersamepassphraseagain:

Youridentificationhasbeensavedin//.ssh/id_rsa.

Yourpublickeyhasbeensavedin//.ssh/id_rsa.pub.

Thekeyfingerprintis:

ca:01:e8:3d:fb:8f:d7:7e:da:ba:28:c7:bd:ef:58:17root®tstgpfsi

Thekey'srandomartimageis:

4—[RSA2048]——+

.o.SE

+o

・0.0••

•••+•o+..

0=0.B0+

#cdossh

#ls

id_rsaid_rsaopub

#catid_rsaopub>authorizcd_kcys

#chmod600authorized_keys

#echo"StrictHostKeyCheckingno"〉config

#chmod600config

■修改各节点的/etc/hosts文件，加入对集群中各节点的别名和ip地址

9

0

1

2

3

4

5gpfs5S.

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0gpfs7C.

1

2

3

4

5gpfs7S.

6

7

8

9

0

1

2

3

4

100147RSgpfeSah，.jxnxsrcm

6

7

8

9

0

1

2

3

4

5

6

7

8

■验证本机无需密码的ssh登录

#、shhostname'

■将。ssh目录复制到其他各节点

#scp-r/osshgpfs49o:/root/

#scp-r/osshgpfs50oabs.jxnxsocom:/root/#scp—r/.sshgpfs51oabsojxnxsoconi:/root/

#scp—r/.ssh:/root/

验证各节点间无密码的ssh登录

3.2AIX安装GPFS软件

1)gpfs3.5.0安装包放在/home/gpfsaix/iso目录下

#pwd

/home/^pfsaix/iso

total582960

1rootsystem2548NOV1823:49README

1rootsystem149237760NOV1823:49GPFS_AIX_3.5.tar

1rootsystem868352NOV1823:49gpfs.docs.data

1rootsystem61939712NOV1823:49gpfs.base

1rootsystem175104NOV1823:49gpfs.msg.en_US

1rootsystem86242304NOV1823:49gpfs.gnr

2)切换/home/gpfsaix/iso目录下，通过smittyinstallp安装GPFS软件

InstallandUpdatefronALLAvailableSoftware

Typeorselectvaluesinentryfields.

PressEnterAFTER>akinealldesiredchanges.

[EntryFields]

«INPUTdevice/directoryforsoftvaie

iSOFTyAREtoinstalllu

PREVIEWonly?(installoperationvillNOToccur)no

COJOTITsoftwareupdates?yes

SAVEreplacedfiles?no

AUTOMATICALLYinstallrequisitesoflvare?

EXTENDfilesystemsifffpar

OVERWRITEsaneorneverveISOFTVAREtoinstall

VERIFYinstallandcheckfI

DETAILEDoutput?IMovecursortodesireditenandpressEsc*7.Usearrowkeystoscroll.

ProcessmultipleVOIUJICJ?IONEORMOREite&scanbeselected.

JWXEPTnevlicenseagreeneIPressEnterAFTERmakingallselections.

PrevievnewLICENSEagree、I

VPARNana^eaentI>gpfs.bzseALL

PerformOperationinGI+3.5.0.0GPFSFileManager

PerformOperationonDI

DetachedWPARNanoI>gpfs.docsALL

RenountInstallitionDI♦3.500GPFSServerManpagey®ndDoninentation

AlternateWPARInstallI

I>epfg.，nrALL

I+3.6.0.0GPFSNativeRAID

QB

♦3.5.0.0GPFS

IFl=HelpF2=RefreshF3=Cmcel

Fl=HclpIEsc*7=SelectEsc+8=lTjagcEsc+AExit

Esc*5=ResetIEnter=Do/=Fmdn=FmdNextI»a«e

3)安装补丁

gpfs3.5版本.需要升级到350.11以上，补丁下载地址：

htlD：〃www—933.ibm。com/suDDon/「ixcenlral/swg/seleclFixes?xarenl=Cluste「+soflwai~e&

product二ibm/Dower/IBM+Gcneral+Parallel+File+SYstcm&rclcase=3。5.0&Dlatform;AIX&

funciion=all

下载后上传到服务器后解压：

gzip—d-cfilename,tar.gzItar—xvf—使用smittyupdate_all安装补J

4）安装后检查

#lSIpp

#1sI-1Ig

gpfppbase

f3.5.0.11COMMITTEDGPFSFileManager

gps.gn

pfm3.5.0.7COMMITTEDGPFSNativeRAID

gs.sg•

gpfbase3.5.0.10COMMITTEDGPFSserverMessages-u

Qpfs.do3.5.0.11COMMITTEDGPFSFileManager

▲os.3.5.0.4COMMITTEDGPFSserverManpagesand

s.

如果之前已经创建了cluster,在所有node都down的状态下，加载GPFS版本到最新版本:

mmchconfigrelease=LATEST：如果是新装的GPFS,则可以忽略此步。

3.3Linux安装GPFS软件

1）上传安装介质

安装文件位于/home/gpfsrpm/,补丁文件位于/home/gpfsrpm/updte；

[root@serverlgpfsrpm]#pwd

/home/gpfsrpm"

[root@serverlgpfsrpm]#Is-Irt

total13232

drwxr-xr-x2rootroot4096Oct900:33update

-rw-r--r--1rootroot12062128Nov1908:02gpfs.base-3.5.0-0.ppc64.rp«

-rw-r--r・・1rootroot221487Nov1908:02gpfs.docs-3.5.0-0.noarch.rpn

-rw-r--r--1rootroot96585Nov1908:02gpfs.Msg.en_us-3.5.0-0.noarch,rpni

-rw-r--r--1rootroot499693NOV1908:02gpfs.gpl-3.5.0-0.noarch.rpa

-r——r--r--1rootroot651532NOV1908:02ksh-20100621-19.el6.ppc64.rpm

-rwxr-xr-x1rootroot206Nov1909:56install.sh

2）安装GPFS软件

在安装介质所在目录卜执行:

#rpm-ivhgpfs*.rpm

3）升级GPFS软件版本

在补丁文件所在目录下执行:

#rpni—Uvhgpfs*.rpm

4)检查安装结果

#rpm-qaIgrepgpfs

[root@serverlgpfsrpm]#rpm-qalgrepgpfs

gpfs.base-3.5.0-11.ppc64

gpfs.docs-3.5.0-11.noarch

gpfs.msg.en_us-3.5.0-11.noarch

gpfs.gplbin-2.6.32-358.el6.ppc64-3.5.0-11.ppc64

gpfs.gpl-3.5.0-11.noarch

5)BuildingtheGPFSportabilitylayer

#cd/usr/lpp/mmfs/src

#makeAutoconfig

#makeWorld

#makcInstalllmagcs

3.4GPFS集群配置

■修改系统环境变量

AIX平台下修改/etc/environmcnt在PATH后增加/usr/lpp/mmfs/bin

#vi/e(c/environmenl

ffcat/ere/environment

WIBM.PROLOC-BEGIN.TAG

*Thisisanautomaticallygeneratedprolog.

*bo$61Bsrc/bos/ecc/environf»enr/environment1.26

IP

*LicensedMaterials-PropertyofIBM

■

伊COPYRIGHTinternationalBusinessMachinescorp.1989,19^4

#AllRightsReserved

体

*USGoverrwentusersRestrictedRights-use,duplicationor

ffdisclosurerestrictedbyG5AADPschedulecontractwithIBMcorp.

停

ff1.26src/bos/etc/environment/envirorwent,cmdshvbos61B,b2OO7.S4AO7/23/0709:46:21

RIBM_PROLO<i_END_TAG

*

*COMPONENT.NAWE:(CMDSH)shellrelatedcommands

*

"ORIGINS：27

ff(C)COPYRIGHTinternationalBusinessMachinesCorp.1989.1994

*AliRightsReserved

(fLicensedMaterials-PropertyofIBM

伊

*USGoverrwentusersRestrictedRights-use,duplicationor

#disclosurerestrictedbyGSAADPschedulecontractwithIBMcorp.

*

*Systemwideenvironmentfile.Thisfileshouldonlycontain

.1.commentlineswhichhavea*inthefirstcolumn,

r2.blankdnd

ff3.Linesintheformname=value.

*WARNING：Thisfileisonlyforestablishingenvironmentvariables.

・Executionofcommandsfromthisfileoranylinesother

*chanspecifiedabovenaycausefailureofrheInitializatior

*process.

ffSearchingthecurrentdirectorylastisusuallyaBIGtimesaver.

*if/usr/uebisatthebeginningofthePATHtheBSDversionofcomardswill

.befound.

pATH«/usr/bin:/eTc:/usr/sbin:/usr/ucb:/usr/bin/xil:/sbin:/usr/javaS/jr€/b1n:/usr/java5/bin:/usr/lpp/Mfs/bin—|

TZ-Asia/ShangRai

LANG-en-US

LOCPATH=/usr/lib/nls/loc

NLSPATH-/usr/l1b/nls/msg/%L/XN:/usr/l1b/nls/«sg/%L/%N・car:/usr/l1b/nls/msg/%1.%C/%H:/usr/lib/nls/msg/%1.%C/%N.car

LC_FAS7FSG=true

Linux平台卜修改/root/。bashprofilc,在PATH后增加l/usr/lpp/mmfs/bin

[root@serverl工]#cat/root/.bash_prof11e

#.bash_profile

#Getthealiasesandfunctions

if[-f-/.bashre];then

,-/.bashre

#userspecificenvironmentandstartupprograms

PATH=SPATH:$HOME/bin:/usr/lpp/mmfs/bin

exportPATH

■创建集群

按照规划，本次集群中包含I个AIX节点和48个Linuxclient；首先在1个AIX节点平台创

建GPFS集群

创建集群定义文件

#cd/home

#mkdirgpfs

#cdgpfs

#vijxrcc_cluster

10o0.147.98：quorum-manager

a)创建GPFS集群

#mmcrclusterNjxrccb_clustcrpl()；,()o147。98r/usr/bin/ssh-RZusr/bin/scpC

jxrccb_cluster-A

#mmcrcluster-Njxrccb_cluster-p8-r/usr/bin/ssh-R/usr/bin/scp-Cjxrccb_cluster-A

Warning:Permanentlyadded''(RSA)tothelistofknownhosts.

TueNov1920:01:38CST2013:6027-1664mmcrcluster:Processingnode

mmcrcluster:Commandsuccessfullycompleted

mmcrcluster:6027-1254Warning:NotallnodeshaveproperGPFSlicensedesignations.

Usethemmchlicpnspcommandtodpsienatplicense，asnppdpd.

b)配置许可

#mmchlicenseserver——accept-X10.0：147.98

#mmchlicenseserver-accept-N8

ThefollowingnodeswillbedesignatedaspossessingGPFSserverlicenses:

mmchlicense:Commandsuccessfullycompleted

c)添加Linux节点

通过如卜命令依次添加48个Linux节点：

#inmaddnode-Ngpfs!49.abs.jxnxsocom

#mmchlicenseserver—accept-Ngpfsl49.abs0jxnxsocom

#mmaddnode-Ngpfs；96o

#mmchlicenseserveraccept-Ngpfsl96o

查看集群状态

#mmlscluster

#mmlsclusrer

GPFSclusterinformation

GPFSclustername:jxrccb__

GPFSclusterid:9411937827136739327

GPFSUIDdomain:jxrccb_

Remoteshellcommand:/usr/bin/ssh

Remotefilecopycommand:/usr/bin/scp

GPFSclusterconfigurationservers:

primaryserver:

Secondaryserver:{none)

NodeDaemonnodenameIPaddressAdminnodenameDesignation

1

28gpfs98.quorum-manager

349gpfs149.abs.

450gpfS150.abs.

5gpfS151.51gpf5151.abs.

652gpfsi52.abs.

7gpfS153.abs.jxnxs.com53gpf5153.abs.

8gpf5154.abs.jxnxs.com54gpfsi54.abs.

9gpfsl56.abs.ixnxs.com56gpfsi56.abs.

l057gpfsl57.abs.

1

u58gpfsi58.abs.

259gpfsl59.abs.

l3gpfslCO.abs.jxnxs.cow60gpfslCO.abs.

l4gpfS161.abs.jxnxs.con61gpf5161.abs.

5

1gpfS162.62gpf$162.abs.

6

1gpfS163.63gpfsl63.abs.

7

164gpfsl64.abs.

8

1gprsl65.abs.jxnxs.con65gpfS165.abs.

9

1gpfsl66.abs.]66gpf5166.abs.

0

267gpf5167.abs.

1

268gpf5168.abs.

2

269gpf$169.abs.

3

2gpf5170.abs.]70gpf5170.abs.

4

271gpf5171.abs.■

5

2672gpf5172.abs.;

2773gpf5173.abs.

28gpf5174.abs.ixnxs.con74gpf5174.abs.

2975gpfsl75.abs.

20gpfsl76.abs.jxnxs.con76gpfsl76.abs."

31gpfsl77.abs.jxnxs.com77gpfsl77.abs."

3278gpfsl78.abs.'

3379gpf“79.abs.；

34gpfs180.abs.ixnxs.con80gpfS180.abs.;

3581gpfs181.abs.xnxs.com

3682gpf5182.abs.

3783gpf5183.abs.

3-gpfS184.abs.]84gpfsl84.abs.

3885gpf5185.abs.

3986gpf5186.abs.