1、下载并安装hadoop(需安装好jdk)
2、创建文件夹
~/dfs/name
~/dfs/data
~/temp
3、修改配置文件
~/hadoop-2.2.0/etc/hadoop/hadoop-env.sh
~/hadoop-2.2.0/etc/hadoop/yarn-env.sh
~/hadoop-2.2.0/etc/hadoop/slaves
~/hadoop-2.2.0/etc/hadoop/core-site.xml
~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml
~/hadoop-2.2.0/etc/hadoop/mapred-site.xml
~/hadoop-2.2.0/etc/hadoop/yarn-site.xml
以上个别文件默认不存在的，可以复制相应的template文件获得。
--hadoop-env.sh
修改javahome
export JAVA_HOME=/home/hduser/jdk1.6.0_45
--yarn-env.sh
同上
--slaves
增加处理节点机器名即可
--core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://cloudn:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hduser/temp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>*</value>
</property>
</configuration>
--hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cloudn:9001</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
</configuration>
--mapred-site.xml
<configuration>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<name>mapreduce.jobhistory.address</name>
<value>cloudn:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>cloudn:19888</value>
</property>
</configuration>
--yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>cloudn:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>cloudn:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>cloudn:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>cloudn:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>cloudn:8088</value>
</property>
</configuration>
4、复制到其他节点，如有64位，32位，不能复制
5、启动准备
格式化namenode: bin/hdfs namenode -format
6、启动hadoop:sbin/start-all.sh
7、hellocount
先在hdfs上创建一个文件夹
./bin/hdfs dfs -mkdir /input
./bin/hdfs dfs -ls /
在namenode上创建个文件夹
files,创建2个文件 echo "Hello World" > file1.txt
more file1.txt file2.txt
hadoop fs -mkdir input
创建输入文件夹
./bin/hdfs dfs -mkdir /input
./bin/hdfs dfs -mkdir input
./bin/hdfs dfs -ls /
上传本地文件到集群的input目录
./bin/hdfs dfs -put ~/files/*.txt /input
./bin/hdfs dfs -ls /input
执行
./bin/hadoop jar /home/hduser/hadoop-2.2.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.2.0-sources.jar org.apache.hadoop.examples.WordCount /input /output
结果
./bin/hadoop dfs -cat /output/part-r-00000
posted on 2014-02-26 16:40
李大嘴 阅读(252)
评论(0) 编辑 收藏