hadoop安装 hadoop+thrift+hive.docx

Hadoop + hive + thrift 安装文档一、主服务器设置1.创建用户#useradd hadoop2.设置密码#passwd hadoop3.切换hadoop登录#su - hadoop4.解压hadoop压缩包#tar zxvf hadoop-1.0.3.tar.gz5.设置目录权限#chown -R hadoop.hadoop hadoop-1.0.36.编辑环境变量#vim hadoop-1.0.3/conf/hadoop-env.sh编辑“JAVA_HOME”路径：export JAVA_HOME=/usr/local/jdk1.7.0_05增加一行（取消过时警告）：export HADOOP_HOME_WARN_SUPPRESS=17.编辑系统环境变量(使用root用户)#vim /etc/profile追加如下内容：export JAVA_HOME=/usr/local/jdk1.7.0_05export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport HADOOP_HOME=/home/hadoop/hadoop-1.0.3export PATH=$PATH:$HADOOP_HOME/bin8.执行环境设置#source /etc/profile9.设置主IP#vim hadoop-1.0.3/conf/masters把localhost替换成4710.设置从IP#vim hadoop-1.0.3/conf/slaves把localhost替换成4811.设置hdfs#vim hadoop-1.0.3/conf/hdfs-site.xml增加如下内容：.dir/home/hadoop/hadoop-1.0.3/namedfs.data.dir/home/hadoop/hadoop-1.0.3/datadfs.replication312.配置任务#vim hadoop-1.0.3/conf/mapred-site.xml增加如下内容：mapred.job.tracker47:9001mapred.local.dir/home/hadoop/hadoop-1.0.3/var13.配置核心文件#vim hadoop-1.0.3/conf/hdfs:/47:9000fs.checkpoint.period3600fs.checkpoint.size67108864hadoop.tmp.dir/home/hadoop/hadoop-1.0.3/tmp14.实现主从自动登录（单方向）创建公钥:#ssh-keygen -t rsa自己可以ssh：#cat .ssh/id_rsa.pub .ssh/authorized_keys设置文件权限：#chmod 700 .ssh/authorized_keys测试一下，不需要密码就表示成功#ssh localhost#exit二、从服务器设置1.创建用户#useradd hadoop2.设置密码#passwd hadoop3.编辑系统环境变量(使用root用户)#vim /etc/profile追加如下内容：export JAVA_HOME=/usr/local/jdk1.7.0_05export PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport HADOOP_HOME=/home/hadoop/hadoop-1.0.3export PATH=$PATH:$HADOOP_HOME/bin4.执行环境设置#source /etc/profile5.解压缩hadoop压缩包#tar zxvf hadoop-1.0.3.tar.gz6.设置目录权限#chown -R hadoop.hadoop hadoop-1.0.3三、回到主服务器1.切换hadoop用户#su - hadoop2.复制公私到从服务器#ssh-copy-id -i .ssh/id_rsa.pub hadoop483.测试#ssh 48配置正确的话是不需要密码就能登录的4.向从服务器发送配置文件#scp -r hadoop-1.0.3/conf hadoop48:/home/hadoop/hadoop-1.0.35.格式化分布式文件系统#hadoop-1.0.3/bin/hadoop namenode -format6.启动hadoop服务#hadoop-1.0.3/bin/start-dfs.sh#hadoop-1.0.3/bin/start-mapred.sh7.查看运行情况web查询界面：47:50030四、安装hive(主服务器)1.压缩包装包(hadoop用户)#tar zxvf hive-0.9.0.tar.gz#mv hive-0.9.0 hadoop-1.0.32.配置hive环境变量#cp hadoop-1.0.3/hive-0.9.0/conf/hive-env.sh.template hadoop-1.0.3/hive-0.9.0/conf/hive-env.sh#vim hadoop-1.0.3/hive-0.9.0/conf/hive-env.sh增加一行：HADOOP_HOME=$HADOOP_HOME3.配置hive元数据保存到mysql建立数据数用户，数据库使用latin1字符集：mysqlCREATE DATABASE hive CHARACTER SET latin1;mysqlGRANT ALL PRIVILEGES ON hive.* TO hive% IDENTIFIED BY hivepasswd;修改配置文件：#cp hadoop-1.0.3/hive-0.9.0/conf/hive-default.xml.template hadoop-1.0.3/hive-0.9.0/conf/hive-site.xml#vim hadoop-1.0.3/hive-0.9.0/conf/hive-site.xml修改四个地方：javax.jdo.option.ConnectionURLjdbc:mysql:/47:3306/hive?createDatabaseIfNotExist=trueJDBC connect string for a JDBC metastorejavax.jdo.option.ConnectionDriverNamecom.mysql.jdbc.DriverDriver class name for a JDBC metastorejavax.jdo.option.ConnectionUserNamehiveusername to use against metastore databasejavax.jdo.option.ConnectionPasswordhivepasswdpassword to use against metastore database5.复制mysql连接库包#cp mysql-connector-java-5.1.11-bin.jar hadoop-1.0.3/hive-0.9.0/lib4.启动hive#hadoop-1.0.3/hive-0.9.0/bin/hive5.去掉log4j过期警告#cp hadoop-1.0.3/hive-0.9.0/conf/perties.template hadoop-1.0.3/hive-0.9.0/conf/perties#vim hadoop-1.0.3/hive-0.9.0/conf/perties找到“log4j.appender.EventCounter”将他的值变更为：“org.apache.hadoop.log.metrics.EventCounter”五、安装php访问hive数据库插件1.解压缩thrift包#tar zxvf thrift-0.8.0.tar.gz2.不要ruby支持否则会报错#cd thrift-0.8.0#./configure -without-ruby#make & make install3.后台启动hive#hadoop-1.0.3/hive-0.9.0/bin/hive -se