StorNext FS安装指南.doc

StorNext FS安装指南Tags: 存储安装配置, 共享软件, HBA, StorNext, 文件系统StorNext FS 配置 （Red Hat Enterprise Linux/x86 AS 4 +win2K）一、环境描述：硬件描述：1. Brocade 39502. S2A 8000（本例中3个LUN，分别为MetaData（30GB）、Journal（100MB）、数据）3. DELL 2650 (内存 2G，建议4G)软件描述：1. Red Hat Enterprise Linux/x86 AS 4 update 1 （2.6.9-11）2. StorNext FS Server/Client 软件二、StorNext FS系统配置1准备工作1.1配置网络 # vi /etc/hosts127.0.0.1 localhost.localdomain localhost10.0.5.25 mds1 /这里为两台MDS10.0.5.26 mds21.2 安装Emulex LP8000/982驱动和应用程序安装驱动程序 tar xzf lpfc-.tar.gz /解压安装包如：#tar zxvf lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz#cd lpfc_2.6_driver_kit-8.0.16.17-2#./lpfc-install /执行安装脚本#reboot /重起服务器注意：Red Hat Enterprise Linux/x86 4 updata 1 内核中已经加载Emulex LP8000/982驱动，可以不做这一步。如果在安装驱动程序中，报IO和IRQ参数错，检查Firmware版本，升级为最新的版本。安装emulex hbanyware应用程序#tar xvf elxlinuxapps-2.1a25-8.0.16.17-1-1.tar#./install /安装emulex hnanyware#/usr/sbin/hbanyware/hbanyware /运行emulex hnanyware，查看WWN，硬件信息。注意：需要使用emulex hnanyware时需要安装lpfc_2.6_driver_kit-8.0.16.17-2.tar.gz程序，而linux 系统自带的LPFC的驱动不能被emulex hnanyware所以识别。1.3 效验HBA卡加载。通过cat /proc/scsi/scsi 命令查看HBA卡检测到的磁盘信息如：# cat /proc/scsi/scsiHost: scsi0 Channel: 00 Id: 06 Lun: 00Vendor: PE/PV Model: 15 SCSI BP Rev: 1.1Type: Processor ANSI SCSI revision: 02Host: scsi2 Channel: 00 Id: 00 Lun: 03Vendor: DDN Model: S2A 8000 Rev: 5.02Type: Direct-Access ANSI SCSI revision: 03Host: scsi2 Channel: 00 Id: 00 Lun: 08Vendor: DDN Model: S2A 8000 Rev: 5.02Type: Direct-Access ANSI SCSI revision: 031.4 效验SCSI、RAID、HBA卡加载顺序 此步骤在redhat AS 4以后的版本中，不需要做。下面的例子是redhat as 3 update4如果SCSI、RAID卡的加载在HBA卡后面。不然在本地SCSI系统盘的SWAP分区，不可用。如下#cat /etc/module.confalias scsi_hostadapter lpfc / Emulex HBAalias scsi_hostadapter1 aic7xxx /scsi卡alias usb-controller usb-ohcioptions scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1需要将加载顺序改变，如alias scsi_hostadapter1 aic7xxx /scsi卡alias scsi_hostadapter2 lpfc / Emulex HBAalias usb-controller usb-ohcioptions scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1重建内核# cd /boot# mv initrd-2.4.21-27.ELsmp.img initrd-2.4.21-27.ELsmp.img.ori# mkinitrd initrd-2.4.21-27.ELsmp.img 2.4.21-27.ELsmp# sync# reboot2安装StorNext FS2.1 以root用户登录Linux，插入StorNext FS CD#cd /mnt/cdrom如果CDROM没有被mounted，那么执行#mount /dev/cdrom /mnt# cd /mnt/cdrom/StorNextFS/linux /此路径根据实际情况而定#uname -a 查看当前系统内核的版本。如：uname -aLinux mds01 2.6.9-11.ELsmp #1 SMP Fri May 20 18:26:27 EDT 2005 i686 i686 i386 GNU/Linux# rpm -ivh snfs-server_RH_AS_4_269-2.6.3-32.i386.rpm# rpm ivh snfs-client_RH_AS_4_269-2.6.3-32.i386.rpm注意，在linux上服务器上必须安装Server和client两个软件。2.2 修改环境变量#vi /etc/profile# Path manipulationif id -u = 0 ; thenpathmunge /usr/cvfs/binfiulimit -S -c 0 /dev/null 2 &1ulimit -c unlimited /dev/null 2 &1注销root用户，重新以root用户登陆系统，才能使环境变量起作用2.3 磁盘Label2.3.1 查看盘阵信息，执行fdisk -l如： # fdisk lDisk /dev/sdc (Sun disk label): 0 heads, 0 sectors, 52221 cylindersUnits = cylinders of 1 * 512 bytesDevice Flag Start End Blocks Id System/dev/sdc3 u 0 -872571904 -436285952 5 Whole diskNote: sector size is 2048 (not 512)Disk /dev/sdg (Sun disk label): 0 heads, 0 sectors, 959 cylindersUnits = cylinders of 1 * 512 bytesDevice Flag Start End Blocks Id System/dev/sdg3 u 0 62902272 31451136 5 Whole diskDisk /dev/sdl (Sun disk label): 0 heads, 0 sectors, 2 cylindersUnits = cylinders of 1 * 512 bytesDevice Flag Start End Blocks Id System/dev/sdl3 u 0 192512 96256 5 Whole disk2.1查看disk Label#/usr/cvfs/bin/cvlabel l/dev/sdc DDN S2A 8000 5.02 CVFS ” unknown ” Sectors: 3422394368.SectorSize: 1024. Maximum sectors: 3422394368./dev/sdg DDN S2A 8000 5.02 CVFS ” unknown ” Sectors: 62900224.SectorSize: 512. Maximum sectors: 62900224./dev/sdl DDN S2A 8000 5.02 CVFS ” unknown ” Sectors: 190464. SectorSize: 512. Maximum sectors: 190464.2.2 编辑disk Label配置文件# /usr/cvfs/bin/cvlabel -c /usr/cvfs/config/cvlabels/将cvlabel输出到cvlabels配置文件中# vi /usr/cvfs/config/cvlabels将unkown更改为H_Upload，MD_H_Upload， JR_H_Upload。等，如本例中：H_Upload /dev/sdc # host 0 lun 0 sectors 3422394368 sector_size 1024 inquiry DDN S2A 8000 5.02 serial 60001FF020021101030197FECD180000MD_H_Upload /dev/sdg # host 0 lun 4 sectors 62900224 sector_size 512 inquiry DDN S2A 8000 5.02 serial 60001FF0200211010301932703C00400JR_H_Upload /dev/sdl # host 0 lun 9 sectors 190464 sector_size 512 inquiry DDN S2A 8000 5.02 serial 60001FF020021101030194A700030900 2.3 disk Label# /usr/cvfs/bin/cvlabel /usr/cvfs/config/cvlabels*WARNING* This program will over-write volume labels on thedevices specified in the file “/usr/cvfs/config/cvlabels”.After execution, the devices will only be usable by theCentraVision File System. You will have to re-partition thedevices to use them on a different file system.Do you want to proceed? (Y / N ) - y/dev/sdc DN S2A 8000 5.02 CVFS ” unknown ” Sectors: 3422394368.Do you want to label it SNFS - Name: H_Upload Sectors: 3422394368? (Y / N ) -yNew Volume Label -Device: /dev/sda SNFS Label: H_Upload Sectors: 3422394368./dev/sdg DN S2A 8000 5.02 CVFS ” unknown ” Sectors: 62900224.Do you want to label it SNFS - Name: MD_H_Upload Sectors: 62900224? (Y / N ) -yNew Volume Label -Device: /dev/sdb SNFS Label: MD_H_Upload Sectors: 62900224./dev/sdl DN S2A 8000 5.02 CVFS ” unknown ” Sectors: 190464.Do you want to label it SNFS - Name: JR_H_Upload Sectors: 190464? (Y / N ) -yNew Volume Label -Device: /dev/sdb SNFS Label: JR_H_Upload Sectors: 190464.Done. 3 source lines. 3 labels. 3. 配置StorNext FS3.1创建StorNext FS文件系统的配置文件,若查看参数信息，键入#man -M /usr/cvfs/man cvfs_config# cd /usr/cvfs/config#cp . /examples/example.cfg ./H_Upload.cfg /例程中拷贝出文件系统的配置文件# vi /usr/cvfs/config/H_Upload.cfg GlobalSuperUser Yes # Must be set to Yes for SNMS Managed File Systems #WindowsSecurity NoQuotas NoFileLocks NoDataMigration No # SNMS Managed File Systems Only #InodeExpandMin 32KInodeExpandInc 128KInodeExpandMax 8MFsBlockSize 4KJournalSize 4MAllocationStrategy RoundMaxConnections 78 /该数为一般设为客户数加+10ForceStripeAlignment YesDebug 00MaxLogSize 4MMaxLogs 4# Globals DefaultedThreadPoolSize 32 # default 16, 512 KB memory per threadInodeCacheSize 32K # 800-1000 bytes each, default 8K# BufferCacheSize 64M # default 32MB# StripeAlignSize 2M # auto alignment, default MAX(StripeBreadth)# MaxMBPerClientReserve 50 # in MBs, default 100 MB reserved per client# OpHangLimitSecs 300 # default 180 secs# DataMigrationThreadPoolSize 128 # Managed only, default 8#*# A disktype section for defining disk hardware parameters.#*DiskType MD_H_Upload #1+1 Raid 1 Mirrored Pair#Sectors 62900224 # Sectors Per Disk From Command “cvlabel -l”SectorSize 512DiskType JR_H_Upload #1+1 Raid 1 Mirrored Pair#Sectors 190464SectorSize 512DiskType H_Upload #4+1 Raid 5 Lun for Regular Data#Sectors 3422394368SectorSize 1024 /这里的1024与盘塔上该LUN的Block Size 相一致。#*# A disk section for defining disks in the hardware configuration.#*Disk MD_H_UploadStatus UPType MD_H_UploadDisk JR_H_UploadStatus UPType JR_H_UploadDisk H_UploadStatus UPType H_Upload#*# A stripe section for defining stripe groups.*StripeGroup MetaFilesStatus UPMetaData YesJournal NoExclusive YesRead Enabled Write EnabledStripeBreadth 256KMultiPathMethod RotateNode MD_H_Upload 0StripeGroup JournFilesStatus UPJournal YesMetaData NoExclusive YesRead EnabledWrite EnabledStripeBreadth 256KMultiPathMethod RotateNode JR_H_Upload 0StripeGroup H_Upload_FilesStatus UPExclusive No #Non-Exclusive StripeGroup for all Files#Read EnabledWrite EnabledStripeBreadth 1M /该参数根据应用可以调整MultiPathMethod RotateNode H_Upload 0若有多个文件系统依次创建文件系统配置文件#vi /usr/cvfs/config/ file_system_name.cfg# /etc/init.d/cvfs restart /重起cvfs服务。Unmounting SNFS filesystemsStopping SNFS DaemonsStopping SNFS PortMapperWaiting for FSMs to finishSNFS Stop 确定 Initializing StorNext Filesystem (SNFS)No disk subsystem config for SNFS, disks not startedLoading SNFS modulesStarting /usr/cvfs/bin/fsmpm.Starting /usr/cvfs/bin/cvfsdMounting SNFS filesystemsSNFS Initialized 确定 # /usr/cvfs/bin/cvmkfs H_Upload /初始化文件系统StorNext File System File System Initializer.* WARNING * This will destroy all contents and data of the H_Upload file system. Re-initialization cannot be un-done!Do you want to proceed? (Y/N) - yRe-initializing file system H_Upload .Shared Meta Data File System.Meta Data Root is on “ MetaFiles “.Resetting Arbitration Block.Writing Configuration Information Block.Writing Root Directory Block.Writing Disk Information Block.Writing Stripe Group Index “ JournFiles “.Resetting File System Journal.Creating Free List.Writing Initial 1024 Inodes.Writing Initialization Control Block.Resetting Super Block.Writing Reserved Allocations.File system H_Upload successfully created.同样方法依次初始化文件系统 # /usr/cvfs/bin/cvmkfs file_system_name3.2 创建文件系统配置文件列表的文件# cd /usr/cvfs/config# cp ./examples/fsmlist.example ./fsmlist/例程中拷贝出文件系统配置文件列表的配置文件#vi /usr/cvfs/config/fsmlist/将当前的StorNext FS文件系统的配置文件H_Upload.cfg的文件名写入fsmlistfile_system_name . 0如： H_Upload . 0/ 其中的 . 表示当前机器的ip地址,0是优先,可为一个07. 但当前默认的为0是activate3.3 创建文件系统配置服务器列表的文件# cd /usr/cvfs/config #cp ./examples/fsnameservers.example ./fsnameservers/将当前的StorNext FS文件系统的服务器的ip写入fsnameservers如: 10.5.0.2510.5.0.26在做群集时,可以将每台服务器ip加入进来.并保持每一台的fsnameservers文件一致3.2 手动启动文件系统配置# /usr/cvfs/bin/cvadminStorNext File System AdministratorEnter command(s)For command help, enter “help” or “?”.List FSSFile System Services (* indicates service is in control of FS):1* H_Upload 0 located on mds1:32790 (pid 2949)/我们可以看到系统已经自动将H_Upload启动并激活了.* 表示激活如果没有显示文件系统名称, 一般需要重新启动cvfs服务或检查fsmlist配置或者手动启动,如下Select FSM “none”snadmin start H_UploadStart FSS ” H_Upload ”FSS H_Upload start initiated.FSS H_Upload started.snadmin activate H_Upload Activate FSM ” H_Upload ”Created : Fri Aug 19 10:02:50 2005Active Connections: 24Fs Block Size : 4KMsg Buffer Size : 4KDisk Devices : 3Stripe Groups : 3Mirror Groups : 0Fs Blocks : 855598592 (3.19 TB)Fs Blocks Free : 838919232 (3.13 TB) (98%)snadmin (H_Upload) show longShow stripe groups (File System “H_Upload”)Stripe Group 0 MetaFiles Status:Up,MetaData,ExclusiveTotal Blocks:7862528 (29.99 GB) Free:7827253 (29.86 GB) (99%)MultiPath Method:RotateStripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)Realtime limit IO/sec:0 (0 mb/sec) Non-Realtime reserve IO/sec:0Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0Disk stripes:Primary Stripe 0 MetaFiles Read:Enabled Write:EnabledNode 0 MD_H_UploadStripe Group 1 JournFiles Status:Up,Journal,ExclusiveTotal Blocks:23808 (93.00 MB) Free:22784 (89.00 MB) (95%)MultiPath Method:RotateStripe Depth:1 Stripe Breadth:64 blocks (256.00 KB)Realtime limit IO/sec:0 (0 mb/sec) Non-Realtime reserve IO/sec:0Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0Disk stripes:Primary Stripe 0 JournFiles Read:Enabled Write:EnabledNode 0 JR_H_UploadStripe Group 2 H_Upload_Files Status:UpTotal Blocks:855598592 (3.19 TB) Free:838919232 (3.13 TB) (98%)MultiPath Method:RotateStripe Depth:1 Stripe Breadth:256 blocks (1024.00 KB)Realtime limit IO/sec:0 (0 mb/sec) Non-Realtime reserve IO/sec:0Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0Disk stripes:Primary Stripe 0 H_Upload_Files Read:Enabled Write:EnabledNode 0 H_Uploadsnadmin (H_Upload) who /查看系统用户信息Who (File System “H_Upload”)# SNFS I.D. Type Location Up Time License Expires - - 0 FSM 1d 3h 55m N/A1 F1F6B61DB CLI 10.5.0.12 0d 3h 6m Permanent37 ADM mds1 0d 0h 0m N/Asnadmin (H_Upload) exit4. Mount StorNext FS /当linux作为client时，需要mount cvfs文件系统# mkdir /H_Upload /创建mount点的目录# chmod 777 / H_Upload /设置/usr/snfs目录的权限，当前权限为可读可写可执行# vi /etc/fstab /当系统启动时，自动装载StorNext FS文件系统H_UploadH_Upload /H_Upload cvfs verbose=yes 0 0/手动装载StorNext FS文件系统，# mount -t cvfs H_Upload /H_Uploadmount.cvfs: Filesystem H_Upload mounted on / H_Upload三、StorNext FS系统Client端配置1Linux client安装# rpm ivh snfs-client_RH_AS_4_269-2.6.3-32.i386.rpm编辑fsmlist和fsnameservers，于MDS的文件内容一致。如：# vi fsmlistH_Upload# vi fsnameservers10.5.0.2510.5.0.26编辑自动mount文件系统信息# mkdir /H_Upload /创建mount点的目录# chmod 777 / H_Upload /设置/usr/snfs目录的权限，当前权限为可读可写可执行# vi /etc/fstab /当系统启动时，自动装载StorNext FS文件系统H_UploadH_Upload /H_Upload cvfs verbose=yes 0 02Windows client安装注意：不要用Windows 中的磁盘管理器对StorNext FS将要管理的磁盘做导入和签名运行StorNext FS client of windows的安装程序setup.exe，按默认步骤操作即可。注意下图中不要选择server files选择label my stornext file system disk devices输入ip地址服务器的ip 的地址。如本列中的ip地址 10.5.0.25,10.5.0.26查看DISK Lables,如果没有CVFS的卷标.手动点击label.对相应的磁盘做label,注意不要随意更改label名称，或将本地盘做label.选择一个dirve盘符号如X:，选择一个文件系统如H_Upload，点击Map ,同样操作将H_Upload Map 为x:选择一个drive,如X: 点击mount opint选项页，选择File name case insensitive，在new file name case中选择preseve case,此为设置大小写敏感.重新启动电脑。正常启动后，在我的电脑中多一个本地盘符x:。测试读写是否正常。注意在win2k下面将回收站的属性改为独立配置驱动器，对于网卡要将校验和相关的检测的都关掉。不然影响读写速度,如下图所示,WINDOWS 中常用的命令为C:SNFSbincvntlabel.exe /标记labelC:SNFSbinrmsvc.exe stoponly /停止snfs 服务C:SNFSbinrmsvc.exe