BlogJava-paulwong-随笔分类-AWShttp://www.blogjava.net/paulwong/category/53879.htmlzh-cnSun, 08 Sep 2013 05:49:44 GMTSun, 08 Sep 2013 05:49:44 GMT60Install Hadoop in the AWS cloudhttp://www.blogjava.net/paulwong/archive/2013/09/08/403816.htmlpaulwongpaulwongSun, 08 Sep 2013 05:45:00 GMThttp://www.blogjava.net/paulwong/archive/2013/09/08/403816.htmlhttp://www.blogjava.net/paulwong/comments/403816.htmlhttp://www.blogjava.net/paulwong/archive/2013/09/08/403816.html#Feedback0http://www.blogjava.net/paulwong/comments/commentRss/403816.htmlhttp://www.blogjava.net/paulwong/services/trackbacks/403816.html
  • get the Whirr tar file
    wget http://www.eu.apache.org/dist/whirr/stable/whirr-0.8.2.tar.gz
  • untar the Whirr tar file
    tar -vxf whirr-0.8.2.tar.gz
  • create credentials file
    mkdir ~/.whirr
    cp conf/credentials.sample ~/.whirr/credentials
  • add the following content to credentials file
    # Set cloud provider connection details
    PROVIDER=aws-ec2
    IDENTITY=<AWS Access Key ID>
    CREDENTIAL=<AWS Secret Access Key>
  • generate a rsa key pair
    ssh-keygen -t rsa -P ''
  • create a hadoop.properties file and add the following content
    whirr.cluster-name=whirrhadoopcluster
    whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,2 hadoop-datanode+hadoop-tasktracker
    whirr.provider=aws-ec2
    whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
    whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
    whirr.hadoop.version=1.0.2
    whirr.aws-ec2-spot-price=0.08
  • launch hadoop
    bin/whirr launch-cluster --config hadoop.properties
  • launch proxy
    cd ~/.whirr/whirrhadoopcluster/
    ./hadoop-proxy.sh
  • add a rule to iptables
    0.0.0.0/0 50030
    0.0.0.0/0 50070
  • check the web ui in the browser
    http://<aws-public-dns>:50030
  • add to /etc/profile
    export HADOOP_CONF_DIR=~/.whirr/whirrhadoopcluster/
  • check if the hadoop works
    hadoop fs -ls /



















    paulwong 2013-09-08 13:45 发表评论
    ]]>