随笔-95  评论-31  文章-10  trackbacks-0
oozie版本4.2.0
sqoop版本1.4.6

调用方式:
restful API方式调用oozie然后触发sqoop shell命令

URL: http://192.168.0.201:11000/oozie/v1/jobs?jobtype=sqoop
请求类型:POST
HTTP content:
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://dn-sa:9000</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>dn-sa:8032</value>
    </property>
    <property>
      <name>queueName</name>
      <value>default</value>
    </property>
    <property>
        <name>user.name</name>
        <value>hadoop</value>
    </property>
    <property>
        <name>mapreduce.job.user.name</name>
        <value>hadoop</value>
    </property>
    <property>
        <name>oozie.sqoop.command</name>
        <value>
            import 
            --connect 
            jdbc:mysql://localhost:3306/oozie 
            --username 
            xxx
            --password 
            xxx
            --table 
            WF_JOBS 
            --target-dir 
            /home/hadoop/oozie-4.2.0-deployed/oozie-jobs/output-data/data_3
        </value>
    </property>
    <property>
    <name>oozie.libpath</name>
    <value>hdfs://dn-sa:9000/user/hadoop/share/lib/lib_20161010103517/sqoop</value>
    </property>
    <property>
        <name>oozie.proxysubmission</name>
        <value>true</value>
    </property>
</configuration>

说明:
1:上面有个坑爹的地方,上面sqoop命令行之间必须要加换行符,否则oozie提交过去,日志会打出No such sqoop tool: import --connect  xxxxxxxxxxxxxxxx   日志在hadoop的logs文件夹里看。 换句话说代码里面必须加上\r\n换行符
2:另外一个坑爹的地方,必须要把DB的driver添加到$HADOOP_HOME/share/hadoop/yarn目录下, 否则会提示找不到驱动。

至此整个任务调度完成。
    


posted on 2016-10-10 14:12 朔望魔刃 阅读(943) 评论(0)  编辑  收藏 所属分类: java

只有注册用户登录后才能发表评论。


网站导航: