玉柴私有云实施方案参考h3c2014onestor运维v3

1、H3CONEStor运维大H3CTechnologies.1概2Arp-.与ss命令对H3CONEStor运维大H3CTechnologies.1概2Arp-.与ss命令对控制功能3.磁盘扩集群扩网络变handyip变clusterip变 4日志介日志类日志收告警查查看告邮件告告警查查看告邮件告5安装InfiniBand 驱6获取及安使用方参数介使用限应用举使用方应用举使用方安装选择硬扫描分区写入数7附12Arp-所有 12Arp-所有 erface=eth0 erface:eth0,datalinktype:EN10MB HYPERLINK http:/w/ 124694567826pack

2、etsreceivedbyfilter,0packetsdroppedbyEndingarp-scan1.8.1:256hostsscannedin26packetsreceivedbyfilter,0packetsdroppedbyEndingarp-scan1.8.1:256hostsscannedin1.256seconds(203.82hosts/sec).26查看测试系统内某个IP rootubuntu3:#sudoarp-scan-Ieth0erface:eth0,datalinktype:EN10MB HYPERLINK http:/w/ 1packetsreceivedbyfi

3、lter,0packetsdroppedbyEndingarp-scan1.8.1:1hostsscannedin0.415seconds(2.41hosts/sec).0Nets ss Linux 系统都会默认包含 命令，但并非所有系统都会默认包含 ss 命令。是 net-tools 工具集中的一员，而 ss 命令是 iproute 工具集中的一员，如果你无法使用 ss 命令那么可能是缺少了 iproute，需要安装一下：ss命令可以用来获取socket统计信息，它可以显示和nets 类似的内容。但ss的优势在于它能够显更详细的有关TCP和连接状态的信息，而且比nets 更快速更高效ss快的

4、秘诀在于，它利用到了TCP协议ss快的秘诀在于，它利用到了TCP协议栈中tcp_diag。tcp_diag是一个用于分析统计的模块， 可以获得Linux 内核中第一手的信息，这就确保了ss的快捷高效。当然，如果你的系统中没有 tcp_diag，ss也可以正常运行，只是效率会变得稍慢。当前服务器的网络连接统计#ss 如果只想查看 TCP sockets，那么使用-ta 选项；如果只想查看 UDP sockets，那么使用-ua 选项；如果只想查看 RAW sockets，那么使用-wa 选项；如果只想查看 UNIX sockets，那么使用-xa 选项。命令对照表控制功 控制功 格式：iptab

5、les-tDchainCRETIRIA-j-ttable ：3个filternat ptablesAINPUTspTCPjDROPptables-AINPUT-s/24-pTCPptablesAINPUTspTCPjDROPptables-AINPUT-s/24-pTCP-j # ptablesAINPUTme -ptablesAINPUTptcpieth0dport4848jACCEPT#开发TCPU的4848pre-upiptables-restore设这个协议，进来的为 8（），出去的为 0(响应).要 8 出去,允许 0 进来。在出去的端口上：iptablesAOUTPUTpicmpi

6、cmp-type8j在进来的端口上：iptablesAINPUTpicmpicmp-type0j33息physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,息physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,rootceph4:#hpssaclictrl slot=3createtype=ldWarning: Creation of this logical drive has caused array letters e arrayG(SATA,UnusedSpace:0 logicaldrive7(931.5G

7、B,RAID0,physicaldrive2I:2:2(port2I:box2:bay2,SATA,1TB,（9） 此时继续执行4.2.2（9） 此时继续执行.1 3.2.1 3.2.2 3.2.3 RBD （1） 无用户使用3.2.2 3.2.3 RBD （1） 无用户使用（2） 若有用户在使用（2） 若有用户在使用 3.3.1 handy ip 而 a.初始handy节点availableonyour#Thisfiledescribesthe#andhowtoactivatethem.Formoreinformation,#Theackautoifaceloinet#Theprimarya

8、utoifaceeth0inetbroadcast gateway #Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask availableonyour#Thisfiledescribestheandhowtoactivatethem.Formoreinfo

9、rmation,Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #ifaceeth0inetbroadcast gateway #dns-*optionsareimplementedbytheresolvconfpackage,ifinstalled dns-nameservers autoifaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask #Thisfiledescribestheavailableonyour#andhow#Thisfilede

10、scribestheavailableonyour#andhowtoactivatethem.Formoreinformation,Theack#autoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddressauto eth2ifaceeth2inetaddressnetmask availableonyour#Thisfiledescribesthe

11、andhowtoactivatethem.Formoreinformation,availableonyour#Thisfiledescribestheandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Theprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ififaceeth1inetaddress auto eth2ifaceeth2ineta

12、ddressnetmask handy handyiphandyip3.3.2 cluster ip rootceph1:/home/ceph-cluster# vi #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theack rootceph1:/home/ceph-cluster# vi #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Th

13、eprimaryautoifaceeth0inetbroadcast gateway #dns-*optionsareimplemented dns-nameservers bytheresolvconfpackage,ifautoifaceeth1inetaddress auto eth2ifaceeth2inetaddress netmask 0(0)56(84)bytesof-sistics-2packetstransmitted,0received,100%packetloss,time通60 (60) 56(84) bytes of time=0.386 time=0.257 -si

14、stics-3packetstransmitted,3received,0%packetloss,time 60 (60) 56(84) bytes of time=0.386 time=0.257 -sistics-3packetstransmitted,3received,0%packetloss,time #Thisfiledescribestheavailableonyourandhowtoactivatethem.Formoreinformation,Theackautoifaceloinet#Theprimaryautoifacevswitch0inetbroadcast gate

15、way #dns-*optionsareimplemented dns-nameservers autobytheresolvconfpackage,ifaceeth0inetaddressnetmask autoifaceeth1inetaddress netmask autoifaceeth2inetaddress auto eth3ifaceeth3inets address netmask address auto eth3ifaceeth3inets address netmask fsid = 1575eefa-d8c4-48e9-8396-1038beb2dba5 mon_ini

16、tial_members = cvknode1, cvknode2, cvknode3 mon_host = 5,6,7auth_cluster_required = auth_service_required = nt_required=filestore_xattr_use_omap=osd_journal_size = 10240 public_network = /24 mon_config_key_max_entry_size = filestore_merge_threshold = filestore_split_multiple = filestore_fd_cache_siz

17、e = filestore_fd_cache_shards = osd_enable_op_tracker = rbd_cache=host=keyring=log_file=rgw_frontends=civetwebrgw_dns_name=osd_crush_update_on_start = cluster_network/24 修改为cluster_networkosd_crush_update_on_start = cluster_network/24 修改为cluster_network fsid = 7f3d1904-ee11-47e6-968e-ff265e47510a mo

18、n_initial_members = ceph1, ceph2, ceph3mon_host = auth_cluster_required = cephx auth_service_required = nt_required=filestore_xattr_use_omap = osd_journal_size=public_network=cluster_network=mon_config_key_max_entry_size = filestore_merge_threshold = filestore_split_multiple = filestore_fd_cache_siz

19、e = filestore_fd_cache_shards = osd_enable_op_tracker = rbdcache= = 5(5)56(84)bytesof-istics-2packetstransmitted,0received,100%packetloss,time5(5)56(84)bytesof-istics-2packetstransmitted,0received,100%packetloss,time通35 (35) 56(84) bytes of time=0.119 更换故障的mon 3.4.1 ONEStorUbuntu14.04.3: 1、 在backupN

20、ode1上增加mon进程2、 删除掉故障节点primaryNode1的mon进程 / cp/etc/ceph/ceph.conf/home/ceph-rootprimaryNode1:/etc/ceph#ceph-deploy-overwrite-confconfigprimaryNode2primaryNode3 cat ceph-deploy-overwrite-confmon cat ceph-deploy-overwrite-confmonaddceph -sStep6：删除掉primaryNode1ceph-deploymondestroyStep7:在所有节点更新moncp/etc

21、/ceph/ceph.conf/home/ceph-rootprimaryNode1:/etc/ceph#ceph-deploy-overwrite-confconfigprimaryNode2primaryNode3Step8：更新clusterconfig，保证HandyrootprimaryNode1:/etc/ceph#cephconfig-keyget/24,ntp_close:1,mon_ip:83,82,mon_fqdns:backupNode1,primaryNode2,primaryNode3, public_network: /24 3.4.2 ONEStor 部署在CAS

22、 16节点的ONEStor集群cvknode1、cvknode2、cvknode3cvknode15、1、 在cvknode4上增加mon2、除掉故1、 在cvknode4上增加mon2、除掉故障节点cvknode1的mon进程。Step1:在Handy上节点，先将cvknode4 Step2:更新HandyBceph-deploy-overwrite-confconfigpushcvknode2cvknode3cvknode4cvknode5 cvknode6cvknode7cvknode8cvknode9cvknode10cvknode11cvknode12cvknode14cvknode

23、15 Step3:cat Step4:在cvknode4上增加mon节点rootcvknode1:/etc/ceph#ceph-deploy-overwrite-confmon add cephStep6:删除掉cvknode1上的monrootcvknode1:/etc/ceph#ceph-deploymondestroyStep7:在所有节点更新mon文件rootcvknode1:/etc/ceph#cpceph.conf/home/ceph-C、 rootcvknode1:/etc/ceph#cephconfig-keygetrootcvknode1:/etc/ceph#cephconf

24、ig-keycluster_network:/24,mon_ip:6,7,mon_fqdns:cvknode4,cvknode2, cvknode3, public_network: /24Step9:rootcvknode1:/etc/ceph#cephconfig-keygetrootcvknode1:/etc/ceph#cephconfig-keycluster_network:/24,mon_ip:6,7,mon_fqdns:cvknode4,cvknode2, cvknode3, public_network: /24Step9:Handy界面是否能够正常显示新增的MON内内日志4.

25、1.2 4.2.1 4.2.2 从handy在在在20GB当当提示）提示）4.4.1 NTP4.4.1 NTP restrictIPmasknetmask_IP restrictIPmasknetmask_IP noquery：不提供C ntnotrust：该C nt除非通过认证，否则该C nt2server NTPserver在 客户端与NTPServer1ntpdate ntpdateNTPServer时间一致。如下图所示，offset 是指与NTP Server 的时间差。 ntpdate不会考虑其他程序是否会阵痛，直接调整时间。一个是校准时间，一个是ONEStor集群开启NTP 服务后

26、，4.4.3 NTP 4.4.3 NTP 4.4.4 ONEStoreNTP客户端向NTPServer进行时间同步的过程，这 节点为NTP +MON2节点MON3节点MON1节点 节点只做NTP 4.4.5 ONEStor与CAS先部署 CAS 集群，设置NTPServerIP地址，再部署ONEStorONEStorCAS集群，设置NTPServerIP（2）ONEStorCAS集群，配置NTPServerIP 4.5.1 1. onestor-clidestroyhosts 集群ip(用逗号隔开onestor-clidestroy-hosts2. （1） onestor-cliprobeha

27、ndy节点（2） onestor-clionestor-cliprobeonestor-cli2. （1） onestor-cliprobehandy节点（2） onestor-clionestor-cliprobeonestor-cli 55.1.1检查InfiniBand ： 件InfiniBand5.1.3安装InfiniBand ： 件InfiniBand5.1.3安装InfiniBand ./5.1.4InfiniBand（1） 开启openibd （2） 启动opensm理 （3） 查看网口状态 （2） 启动opensm理 （3） 查看网口状态 （4） 配置vi（5） 测试网口速率

28、服务器端ifdownib0&ifupib0 ifconfig ib05.1.5m4tk8.4dpatchflex swiggfortrandkmslibgfortran3utomakepkg-5.1.5m4tk8.4dpatchflex swiggfortrandkmslibgfortran3utomakepkg-libxml2tclgraphviztcl8.4tkapt-getinstallym4tk8.4dpatchflex LinuxIOLinuxIO-d-k1为-d -m 为 -d-x-k1 -c110用增加。await的大小一般取决于服务时间(svctm)I/OI/OI/O HP6.

29、1.1hpssacliI/O HP6.1.1hpssacli6.1.2 6.1.3# hpssaclictrlallshowSmartArrayP440arinSlot0ControllerS us:CacheS us: Battery/CapacitorS # hpssaclictrlallshowSmartArrayP440arinSlot0ControllerS us:CacheS us: Battery/CapacitorS us:SmartArrayP840inSlotControllerS us:CacheS us: Battery/CapacitorS us:#hpssaclic

30、trlslot=0showconfig#hpssaclictrlslot=0showconfigdetail|grepDiskName:DiskName:DiskName:DiskName:DiskName:DiskName:DiskName:DiskName:#hpssaclictrlslot=0ldallSmartArrayP440arinSlot0arraylogicaldrive1(1.1TB,RAID0,arraylogicaldrive2(1.1TB,RAID0,arraylogicaldrive3(1.1TB,RAID0,arraylogicaldrive4(1.1TB,RAID

31、0,arraylogicaldrive5(1.1logicaldrive3(1.1TB,RAID0,arraylogicaldrive4(1.1TB,RAID0,arraylogicaldrive5(1.1TB,RAID0,arraylogicaldrive6(1.1TB,RAID0,arraylogicaldrive7(1.1TB,RAID0,arraylogicaldrive8(1.1TB,RAID0,查看某个#hpssaclictrlslot=0arrayAldall/查看slot0 阵列A SmartArrayP440arinSlot0arraylogicaldrive1(1.1TB,

32、RAID0,hpssaclictrlslot=0arrayApdallshow/查看slot0 阵列A SmartArrayP440arinSlot0arrayphysicaldrive1I:3:1(port1I:box3:bay1,SAS,1200.2GB,#hpssaclictrlslot=0logicaldrive6#hpssaclictrlslot=0pdallshowphysicaldrive1I:3:1(port1I:box3:bay1,1200.2GB):physicaldrive1I:3:2(port1I:box3:bay2,1200.2GB):physicaldrive1I:

33、3:3(port1I:box3:bay3,1200.2GB):physicaldrive1I:3:4(port1I:box3:bay4,1200.2GB):physicaldrive2I:3:6(port2I:box3:bay6,1200.2GB):physicaldrive2I:3:5(port2I:box3:bay5,1200.2GB):physicaldrive2I:3:8(port2I:box3:bay8,1200.2GB):physicaldrive2I:3:7(port2I:box3:bay7,1200.2GB):#hpssaclictrlslot=0pdall#hpssaclictrlallshowconfigdetail|grep-i#hpssaclictrlslot=0pdall#hpssaclictrlallshowconfi