BlogJava-anchor110-文章分类-bigdatahttp://www.blogjava.net/anchor110/category/54814.htmlzh-cnSun, 26 Feb 2017 03:22:55 GMTSun, 26 Feb 2017 03:22:55 GMT60如何查看hadoop版本http://www.blogjava.net/anchor110/articles/432303.html小一败涂地小一败涂地Mon, 13 Feb 2017 09:42:00 GMThttp://www.blogjava.net/anchor110/articles/432303.htmlhttp://www.blogjava.net/anchor110/comments/432303.htmlhttp://www.blogjava.net/anchor110/articles/432303.html#Feedback0http://www.blogjava.net/anchor110/comments/commentRss/432303.htmlhttp://www.blogjava.net/anchor110/services/trackbacks/432303.htmlhadoop version

小一败涂地 2017-02-13 17:42 发表评论
]]>hadoop查看文件大小命令http://www.blogjava.net/anchor110/articles/432298.html小一败涂地小一败涂地Sat, 11 Feb 2017 16:19:00 GMThttp://www.blogjava.net/anchor110/articles/432298.htmlhttp://www.blogjava.net/anchor110/comments/432298.htmlhttp://www.blogjava.net/anchor110/articles/432298.html#Feedback0http://www.blogjava.net/anchor110/comments/commentRss/432298.htmlhttp://www.blogjava.net/anchor110/services/trackbacks/432298.htmlhadoop dfsadmin -report

小一败涂地 2017-02-12 00:19 发表评论
]]>hbase启动时报错：java.lang.NoClassDefFoundError: org/htrace/Tracehttp://www.blogjava.net/anchor110/articles/424888.html小一败涂地小一败涂地Wed, 06 May 2015 06:26:00 GMThttp://www.blogjava.net/anchor110/articles/424888.htmlhttp://www.blogjava.net/anchor110/comments/424888.htmlhttp://www.blogjava.net/anchor110/articles/424888.html#Feedback2http://www.blogjava.net/anchor110/comments/commentRss/424888.htmlhttp://www.blogjava.net/anchor110/services/trackbacks/424888.htmlhadoop2.6，搭配hbase1.0.1，启动hadoop时，报错如下：

java.lang.NoClassDefFoundError: org/htrace/Trace
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:214)
        at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1526)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1275)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:831)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.htrace.Trace
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 27 more

解决办法：
将$HADOOP_HOME/share/hadoop/common/lib下的htrace-core-3.0.4.jar复制到$HBASE_HOME/lib下。

小一败涂地 2015-05-06 14:26 发表评论
]]>SIMPLE authentication is not enabled. Available:[TOKEN]http://www.blogjava.net/anchor110/articles/424868.html小一败涂地小一败涂地Tue, 05 May 2015 09:48:00 GMThttp://www.blogjava.net/anchor110/articles/424868.htmlhttp://www.blogjava.net/anchor110/comments/424868.htmlhttp://www.blogjava.net/anchor110/articles/424868.html#Feedback0http://www.blogjava.net/anchor110/comments/commentRss/424868.htmlhttp://www.blogjava.net/anchor110/services/trackbacks/424868.html
hbase启动时，查看log日志，报错如下：

2015-05-05 14:58:32,926 FATAL [master:16020.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled.  Available:[TOKEN]
        at org.apache.hadoop.ipc.Client.call(Client.java:1411)
        at org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:602)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
        at com.sun.proxy.$Proxy16.setSafeMode(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2264)
        at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:986)
        at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:970)
        at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:447)
        at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:894)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:416)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:145)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:125)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:594)
        at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:165)
        at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1428)
        at java.lang.Thread.run(Thread.java:745)

经过各方查找，最终发现问题是$HBASE_HOME/conf/hbase-site.xml中的如下属性配置错了：
<property>
                <name>hbase.rootdir</name>
                <value>hdfs://master:8020/hbase</value>
</property>

注：此配置中的IP或域名必须与$HADOOP_HOME/etc/hadoop/core-site.xml中的如下配置保持一致：
<property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:8020</value>
</property>

总结：初入或者即使有经验的hadoop开发人员，在遇到类似错误时，也经常摸不着北，因为错误日志经常误导大家去其他地方查找问题。


小一败涂地 2015-05-05 17:48 发表评论
]]>could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operationhttp://www.blogjava.net/anchor110/articles/424756.html小一败涂地小一败涂地Wed, 29 Apr 2015 07:21:00 GMThttp://www.blogjava.net/anchor110/articles/424756.htmlhttp://www.blogjava.net/anchor110/comments/424756.htmlhttp://www.blogjava.net/anchor110/articles/424756.html#Feedback0http://www.blogjava.net/anchor110/comments/commentRss/424756.htmlhttp://www.blogjava.net/anchor110/services/trackbacks/424756.html
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /aboutyunlog/FlumeData.1430287668990.tmp could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and 2 node(s) are excluded in this operation.

解决办法：
1、关闭hadoop集群中每一台机器的防火墙：iptables -F。

当然上述方法比较暴力，真实情况可能只是放开Flume向hdfs通讯的几个端口就行了。

小一败涂地 2015-04-29 15:21 发表评论
]]>