大数据平台搭建与运维课件6大数据搭建与运维(项目六)

任务一Hadoop系统日志结构及分析Hadoop系统日志•查看Hadoop系统日志。默认存储位置:Hadoop系统目录下的logs子目录下。•.log文件:存储运行进程中的所有消息,到达一定长度后会另生成一个,的文件,数字越小内容越新。.out文件：存储进程的启动消息,存储机制与.log文件相同。:~$cd/usr/local/hadoop/logs::~$cd/usr/local/hadoop/logs:/usr/local/hadoop八ogs$Ishadoohadochadoop-hadoop-namenode-Master.loghadoop-hadoop-namenode-Master.outhadoop-hadoop-namenode-Master.out.1hadoop-hadoop-namenode-Master.out.2hadoop-hadoop-namenode-Master.out.3hadoop-hadoop-namenode-Master.out.4hadoop-hadoop-secondarynamenode-Master•loghadoop-hadoop-secondarynamenode-Master.outhadoop-hadoop-secondarynamenode-Master.out.1hadoop-hadoop-secondarynamenode-Master.out.2hadoop-hadoop-secondarynamenode-Master.out.3hadoop-hadoop-secondarynamenode-Master•out.4hadoop^Haster:/usr/local./hadoop/logs$Hadoop文件内容。•关注包含"ERROR"关键字的行。:/usr/local/hadoop/logs$20XX-08-O717:59:43,823.node.FSEdttLog:Startinglogsegmentat9820XX-08-0717:59:43,972INFOnode.TransferFsImage:Transfertook0.02sat58.82KB/node.TransferFsImage:Downloadedfilefsimage.ckptO000000000000000097size1832bytes...:Coingtoretain2imageswithtxtd>=95.:PurgingoldtmageFSImageFtle(ftle=/usr/local/hadoop/tnp/dfs/name/current/fstmage0OOO0OOO00000000093>.hadoop.:.hadoop..:SHUTDOWNMSG:SHUTDOWNMSC:ShuttingdownNameNodeatMaster/0************************************************************yhadoop@Maste「:/us「/local/hadoop/logs$■更改HadoopEJ志默认存储路径。•将Hadoop的日志存储路径更改到/var/logo•在/var/log下创立hadoop子目录,并修改权限。:$cd/var/loghp syslogwtmp:/v三 5iscupsaptdtst-upgradeinstallervboxadd•setup•logjournalfalllogvboxadd・gdm3lastlogtallylogvboxadd-btmpgpu-manager.:y:厂/I$1sudomkdirhadoop<i-cupsLhadoopspeech-dispatcherunattended-upgradesaptdist-upgrade下syslogwtmpinstallerfaillogjournal\vboxadd-setup・kern.log\syslog・vboxadd-・2gdm3kern.log.vboxadd-btmpgpu-manager.'loglastlog、allylogvboxadd-:;r/1 |:/var/log$sudochown-Rhadoop/var/log/hadoop[sudo]hadoop的密码:d :/var/log$

修改Hadoop的配置文件:~$cd/usr/local/hadoop/etc/hadoopster:/us「八ocal/hadoop/etc/hadoop$Iscapacity-scheduler・xmlconfiguration.xslcontainer-executor.cfgcore-site.xmlhadooD-envdhadoop-env.shr:/usr/local/hadoop/etc/hadoop$kms-Ipertieskms-site.xmlpertiesmapred-envdmapred-env.shmapped-queues・xml・templatemapred-stte.xnlslavesssl-cltent.xml.examplessl・exportHADOOP-SECURE-DN.USER=${HADOOPSECURE-DNUSER}#exDortHADOOPLOGDIRMIHADOOPLOGDI#Wherelogfilesarestored.$HADOOPHOME/1ogsbydefault.一}/SUSER#exDortHADOOPLOGDIRMIHADOOPLOGDIexportHADOOPLOGDIR=/var/log/hadoop#Wherelogfilesarestoredinthesecuredataenvironment,exportHADOOP-SECUREDN-LOGDIR=${HADOOPLOGDIR}/S{HADOOPHDFSUSER}###HDFSMoverspecificparameters###SpecifytheJVMoptionstobeusedwhenstartingtheHDFSMover.TheseoptionswillbeappendedtotheoptionsspecifiedasHADOOPOPTS--插入--任务二NameNode弟节点展障的风险顼防噩渣。任务二硬件志和HadooP日志,hadoop@bigdata2:~$df-h文件系统容量已用可用udev0tmpfs395M393M/dev/sdal63G52Gtmpfs26Mtmpfstmpfs0/dev/loop00/dev/loopl141M141M0/dev/loop243M43M0/dev/loop3141M141M0/dev/loop40/dev/loop535M35M0/dev/loop689M89M0/dev/loop7150M150M0/dev/loop80/dev/loop955M55M0/dev/loopl015M15M0/dev/loopll91M91M0/dev/loopl215M15M0/dev/loopl30/dev/loopl413M13M0/dev/loopl50tmpfs395M28K395Mtmpfs395M36K395M/dev/sr074M74M0hadoopQbigdata2:~$

已用％挂裁点0%/dev1%/run15%/2%/dev/shm1%/run/lock0%/sys/fs/cgroup100%/snap/gnome-system-monitor/57100%/snap/gnome-3-26-1604/90100%/snap/gtk-common-themes/1313100%/snap/gnome-3-26-1604/74100%/snap/gnome-system-monitor/100100%/snap/gtk-common-themes/818100%/snap/core/7270100%/snap/gnome-3-28-1804/67100%/snap/gnome-logs/61100%/snap/corel8/1074100%/snap/gnome-characters/296100%/snap/core/6350100%/snap/gnome-logs/45100%/snap/gnome-calculater/260100%/snap/gnome-characters/139100%/snap/gnome-calculator/4061%/run/user/1211%/run/user/1001100%磁盘故障分析命令。hadoop@bigdata2:~$sudofdisk・l/dev/loopl5Disk/dev/loopl5:140.7MiB,147496960字节,288686个扇区单元:扇区/1*512=512字节扇区大小(逻辑/物理):512字节/512字节I/O大小（最小/最正确）:512字节/512字节hadoop@bigdata2:~$磁盘故障分析命令。•执行"parted新目录"。任务四Hadoop的隐私及平安认证问题•Hadoop并没有向用户提供平安认证的效劳,由此常常会引发一系列平安问题。•用户可直接访问HDFS或MapReduce集群,因此,不仅任何用户都可获取平台上的数据,还容易引发用户强制对整个集群环境执行命令,造成整个集群受影响。•恶意攻击程序可能会伪装成为Hadoop平台上的某项效劳,对效劳器甚至是用户数据造成恶意攻击。授权问题。•Hadoop的数据节点Datanode未提供有效授权机制,来限制访问数据节点的用户和效劳,所以未授权用户也可通过数据ID获取数据,这大大增加了Datanode上存放数据的风险。Hadoop平安验证问题解决方案 Kerberoso•针对上述种种问题,Hadoop提供了Kerberos平安机制。Kerberos时一种分布式身份验证效劳,其搭建在Hadoop集群之上,当客户效劳器访问Hadoop节点数据时,会直接访问Kerberos,而不是直接访问Hadoop节点,由此保证了Hadoop节点的数据不会因被任意访问而收到攻击。任务五Hadoop的未来i.挑战大数据时代,各种大数据平台面临的首要调整便是数据量的不断增长,能否应对不断扩大的数据体量,也是Hadoop系统未来的挑战之一。此外,随着各种大数据公司的不断升级,能否适应难度逐渐加深的数据处理应用场景,并提供相应解决方案,是Hadoop系统面临的又一挑战。通过前面的学习了解到,Hadoop自身也有各种限制,例如实时数据交互、数据处理延迟、平台本身没有提供有效平安机制保