hadoop2.2 编译安装（64位）

最近要干正事了，周一要讲论文了。论文有关MapReduce。像我实践派必须要实验一下才能信服。所以我就安装了hadoop，期间经历比较曲折。。为了稳定，我选择了hadoop 2.2。坑爹的是这个一上来就有个错，下面细说。

编译选择：
如果你的机器是32位的，可以不折腾。直接安装编译好的。选个mirror，下载个那个大的就是编译好的。如果64位也直接安装编译好的的话，运行时会报如下Warning：
INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1
INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
这个是因为libhadoop.so.1.0.0是32位的，可以 cd hadoop/lib/native/ ，再 file * ，应该会看到32位。参考文档，只能重新编译。我就整个重新编译了。以下为编译步骤，也可以参考这位大哥的，或者直接看解压后中的BUILDING.txt。

安装依赖:

protobuf : arch可以直接安装protobuf，

findbugs: findbugs网上下载，解压后在shell输入: export FINDBUGS_HOME=/path to your extract directory 。不安装的话会有这个错误:

Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: stylesheet /home/hadoop/hadoop-2.0.4-alpha-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist.

1	Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: stylesheet /home/hadoop/hadoop-2.0.4-alpha-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist.

其他依赖，看文档BUILDING.txt。

修改错误:

参考HADOOP-10110或者这位大哥的。手动将hadoop-common-project/hadoop-auth/pom.xml相应地方改掉，否则会报这个类似错：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:

     <dependency>
       <groupId>org.mortbay.jetty</groupId>
+      <artifactId>jetty-util</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.mortbay.jetty</groupId>
       <artifactId>jetty</artifactId>
       <scope>test</scope>
     </dependency>

<groupId>org.mortbay.jetty</groupId>

+ <artifactId>jetty-util</artifactId>

+ <scope>test</scope>

+ </dependency>

+ <dependency>

+ <groupId>org.mortbay.jetty</groupId>

<artifactId>jetty</artifactId>

</dependency>

还有一个我遇到的问题，就是maven版本，要改成3.3。不是hadoop问题。见这个博客和这个issue。修改hadoop/pom.xml两处maven版本的地方，3.0->3.3。报错：[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null] [ERROR] [ERROR] —————————————————–: org.sonatype.aether.graph.DependencyFilter

编译命令
- 以上处理完成后，编译就简单了，只要输入以下命令。就可以在 hadoop-dist/target/ 找到你要的东西。
  mvn clean install -DskipTests mvn package -Pdist,docs,src,native -DskipTests -Dtar
  1
  2
  mvn clean install -DskipTests
  mvn package -Pdist,docs,src,native -DskipTests -Dtar

基本配置

我使用单点的hadoop，好像现在没psudo模式了。参考这位大哥和这位大哥。我把hadoop安装在/opt/hadoop.要注意自己的路径。

添加用户和组
sudo groupadd hadoop sudo useradd -m -g hadoop -s /bin/bash hadoop passwd hadoop su hadoop ssh-keygen -t rsa -P '' cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
1
2
3
4
5
6
sudo groupadd hadoop
sudo useradd -m -g hadoop -s /bin/bash hadoop
passwd hadoop
su hadoop
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
复制安装包
cp hadoop-dist/target/hadoop-2.2.0.tar.gz /opt/ sudo tar -vxzf hadoop-2.2.0.tar.gz -C /opt sudo mv hadoop-2.2.0 hadoop cd /opt sudo chown -R hadoop:hadoop hadoop
1
2
3
4
5
cp hadoop-dist/target/hadoop-2.2.0.tar.gz /opt/
sudo tar -vxzf hadoop-2.2.0.tar.gz -C /opt
sudo mv hadoop-2.2.0 hadoop
cd /opt
sudo chown -R hadoop:hadoop hadoop

修改配置文件

#注意这是hadoop用户的bashrc,路径各个操作系统不同
export JAVA_HOME=/lib/jvm/java-7-openjdk/
export HADOOP_PREFIX=/opt/hadoop
export HADOOP_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
export YARN_HOME=${HADOOP_HOME}
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
# Native Path
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native"
#Java path
# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_PATH/bin:$HADOOP_HOME/sbin

#注意这是hadoop用户的bashrc,路径各个操作系统不同

export JAVA_HOME=/lib/jvm/java-7-openjdk/

export HADOOP_PREFIX=/opt/hadoop

export HADOOP_HOME=/opt/hadoop

export HADOOP_MAPRED_HOME=${HADOOP_HOME}

export HADOOP_COMMON_HOME=${HADOOP_HOME}

export HADOOP_HDFS_HOME=${HADOOP_HOME}

export YARN_HOME=${HADOOP_HOME}

export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

# Native Path

export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native

export HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native"

#Java path

# Add Hadoop bin/ directory to PATH

export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_PATH/bin:$HADOOP_HOME/sbin

################################################
#core-site.xml
<configuration>
	<property>
	<name>fs.default.name</name>
	<value>hdfs://localhost:9000</value>
	</property>
</configuration>


################################################
#hdfs-site.xml
<configuration>

<property>
	<name>dfs.replication</name>
	<value>1</value>
</property>
<property>
	<name>dfs.namenode.name.dir</name>
	<value>file:/opt/hadoop/yarn_data/hdfs/namenode</value>
</property>
<property>
	<name>dfs.datanode.data.dir</name>
	<value>file:/opt/hadoop/yarn_data/hdfs/datanode</value>
</property>

</configuration>

################################################
#mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
     </property>
</configuration>

################################################
#yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->

<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

################################################

#core-site.xml

<name>fs.default.name</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

################################################

#hdfs-site.xml

<name>dfs.replication</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:/opt/hadoop/yarn_data/hdfs/namenode</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>file:/opt/hadoop/yarn_data/hdfs/datanode</value>

</property>

</configuration>

################################################

#mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

################################################

#yarn-site.xml

<name>yarn.resourcemanager.address</name>

</property>

<name>yarn.resourcemanager.scheduler.address</name>

</property>

<name>yarn.resourcemanager.resource-tracker.address</name>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

</configuration>

建yarn节点目录和format node。
node
mkdir -p /opt/hadoop/yarn_data/hdfs/namenode mkdir -p /opt/hadoop/yarn_data/hdfs/datanode hdfs namenode -format
1
2
3
mkdir -p /opt/hadoop/yarn_data/hdfs/namenode
mkdir -p /opt/hadoop/yarn_data/hdfs/datanode
hdfs namenode -format

写个启动脚本和关闭脚本,或者使用start-all.sh,stop-all.sh, 更好的是start-dfs.sh,start-yarn.sh。还要修改hadoop/etc/hadoop/hadoop-env.sh中的JAVA_HOME。

#start_hadoop.sh
hadoop-daemon.sh start namenode
hadoop-daemon.sh start datanode
yarn-daemon.sh start resourcemanager
yarn-daemon.sh start nodemanager
mr-jobhistory-daemon.sh start historyserver

#stop_hadoop.sh
hadoop-daemon.sh stop namenode
hadoop-daemon.sh stop datanode
yarn-daemon.sh stop resourcemanager
yarn-daemon.sh stop nodemanager
mr-jobhistory-daemon.sh stop historyserver

#start_hadoop.sh

hadoop-daemon.sh start namenode

hadoop-daemon.sh start datanode

yarn-daemon.sh start resourcemanager

yarn-daemon.sh start nodemanager

mr-jobhistory-daemon.sh start historyserver

#stop_hadoop.sh

hadoop-daemon.sh stop namenode

hadoop-daemon.sh stop datanode

yarn-daemon.sh stop resourcemanager

yarn-daemon.sh stop nodemanager

mr-jobhistory-daemon.sh stop historyserver

测试例子

运行启动脚步，看看http://localhost:19888/，http://localhost:50070/,http://localhost:8088/，看看是否正常。

测试wordcount。随便搞几个文件，比如start_hadoop.sh,stop_hadoop.sh

hdfs dfs -copyFromLocal ~/start_hadoop.sh input
hdfs dfs -copyFromLocal ~/stop_hadoop.sh input
hdfs dfs -ls input
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input output
#查看job
mapred job -list
#查看结果
hdfs dfs -cat output/part-r-00000

hdfs dfs -copyFromLocal ~/start_hadoop.sh input

hdfs dfs -copyFromLocal ~/stop_hadoop.sh input

hdfs dfs -ls input

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input output

#查看job

mapred job -list

#查看结果

hdfs dfs -cat output/part-r-00000

今天大概就这么点，累死我了。这结果还可以在web界面查看，自己摸索。上面的路径我没有特别说明，希望能看得懂。我还遇到个问题是hosts文件要规范，127.0.0.1 localhost 一定要在最前面。。

链接：

1. hadoop tutorial

hadoop2.2 编译安装（64位）

编译选择：

安装依赖:

修改错误:

编译命令

基本配置

测试例子

近期文章

近期评论

分类目录