BlogJava-xiaomage234-随笔分类-rpc

dubbo中的那些“坑"(3)-netty4-rpc网络接口中的高并发的bug

小马歌 — Tue, 17 May 2016 07:25:00 GMT

在几个月前改造dubbo时，netty4已经稳定很久了，一时手痒，按照netty3-rpc的源码克隆了一套netty4，在修正了大量的包、类型不同之后，基本保持了netty3的风格，并发量小或者数据包很小时，一切都很ok, 在进行大并发测试时，结果和netty3完全不同，基本用惨不忍睹来形容。由于当时急于开发php客户端，就把netty4-rpc当做一个失败的组件存档起来，前几天php-dubbo开发基本完成之后，返回过来思考netty4-rpc的问题，经过仔细分析数据包的解析过程，单步跟踪源码
NettyCodecAdapter, TelnetCodec, ExchangeCoedec，发现ByteBuf的缓冲区为1024,当数据超过1024时，会调用多次Decoder.messageReceived函数，第一次分析dubbo的协议头时，是正确的，第二次之后数据就错误了，然后dubbo内部缓冲区的数据越来越长，但是仍然分析不到一个完整的dubbo数据包
因此去看netty4的源码，发现AbstractNioByteChannel中有网络数据接收的代码时这么处理ByteBuf的
ByteBuf byteBuf = null;
            int messages = 0;
            boolean close = false;
            try {
                int totalReadAmount = 0;
                boolean readPendingReset = false;
                do {
                    byteBuf = allocHandle.allocate(allocator);
                    int writable = byteBuf.writableBytes();
                    int localReadAmount = doReadBytes(byteBuf);
                    if (localReadAmount <= 0) {
                        // not was read release the buffer
                        byteBuf.release();
                        close = localReadAmount < 0;
                        break;
                    }
                    if (!readPendingReset) {
                        readPendingReset = true;
                        setReadPending(false);
                    }
                    pipeline.fireChannelRead(byteBuf);
                    byteBuf = null;
看见没，内核是需要ByteBuf.release的，继续通过byteBuf的一个实现PooledByteBuf分析源码，原来是实现了一个基于简单计数应用计数的循环使用的缓冲区，一旦计数变为1，该缓冲区被归还到netty4内核，被后面的数据读取线程重新使用
而我们InternalDecoder的代码为
      message = com.alibaba.dubbo.remoting.buffer.ChannelBuffers.wrappedBuffer(
                    input.toByteBuffer());
直接引用了ByteBuf.toByteBuffer，继续查看源码UnpooledHeapByteBuf, 其toByteBuffer实际是对内部数据的
一个nio封装而已，因此，使用上述函数时，导致dubbo的decode保存了一个某一个ByteBuffer的内部数据，但是虽有该
buffer被归还到netty4缓冲区中被循环引用，下一次可能被其他读写线程重新改写数据，因此，高并发下当缓冲区被重复使用时，bytebuf将由于计数问题不断被使用，而解码器中缺傻傻等待。
解决方案
1.通过byteBuf的retain和release函数保证计数的有效性，通过程序例外或者缓冲区被使用完成时候归还ByteBuf到netty4内核
2.拷贝数据到dubbo的缓冲区中
思考：
netty3 是否也有该问题呢？？？

小马歌 2016-05-17 15:25 发表评论

dubbo中的那些“坑”(1) - 关于MINA传输协议的bug定位及修复

小马歌 — Tue, 17 May 2016 07:24:00 GMT

from:http://my.oschina.net/aruan/blog/351594

同事刘阳使用dubbo服务器中配置mina作为网络传输层，发现大并发情况下，解码发生如下异常

014-12-01 18:00:44,652 [DubboServerHandler-10.1.19.13:20880-thread-164] WARN alibaba.dubbo.remoting.exchange.codec.ExchangeCodec (ExchangeCodec.java:596) - [DUBBO] Fail to encode response: Response [id=8119, version=2.0.0, status=40, event=false, error=Fail to decode request due to: RpcInvocation [methodName=null, parameterTypes=null, arguments=null, attachments={input=242}, headers=null], result=null], send bad_response info instead, cause: null, dubbo version: 2.5.5, current host: 127.0.0.1
java.lang.NullPointerException
   at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.encodeResponseData(DubboCodec.java:301)
   at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encodeResponse(ExchangeCodec.java:560)
   at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:104)
   at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:39)
   at com.alibaba.dubbo.remoting.transport.mina.MinaCodecAdapter$InternalEncoder.encode(MinaCodecAdapter.java:79)
   at org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCodecFilter.java:214)
   at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:361)
   at org.apache.mina.common.support.AbstractIoFilterChain.access$1300(AbstractIoFilterChain.java:53)
   at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterWrite(AbstractIoFilterChain.java:659)
   at org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWrite(AbstractIoFilterChain.java:587)
   at org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterWrite(AbstractIoFilterChain.java:361)
   at org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(AbstractIoFilterChain.java:355)
   at org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSessionImpl.java:166)
   at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:177)
   at org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:168)
   at com.alibaba.dubbo.remoting.transport.mina.MinaChannel.send(MinaChannel.java:95)
   at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
   at

经过对比netty3和netty4作为传输层，却都没有发现类似的问题。

首先排除不是mina本身的问题，mina也没有爆出有这个问题，初步判断dubbo在使用mina时机制有问题

经过对比发现

1.netty是为每一个channel分配了一个NettyCodecAdapter, mina确实在服务器监听前配置了MinaCodecAdapter

2.也就是说，netty的每一个独立的通道的Codec(encoder/decoder）是通道安全的

3.mina的所有通道是共享相同的codec(encoder/decoder)的，因此，解码器中的实例数据时非channel安全的

因此解码器中与netty相同的解码器的缓冲数据算法在并发情况下将会产生数据覆盖问题。

4.解决方案

1.配置acceptor的监听器

codecAdapter = new MinaCodecAdapter(getCodec(), getUrl(), this);
acceptor.getFilterChain().addLast("codec", new ProtocolCodecFilter(codecAdapter));

    acceptor.addListener(new IoServiceListener(){
           @Override
           public void serviceActivated(IoService service,
                   SocketAddress serviceAddress, IoHandler handler,
                   IoServiceConfig config) {
           }

           @Override
           public void serviceDeactivated(IoService service,
                   SocketAddress serviceAddress, IoHandler handler,
                   IoServiceConfig config) {
           }

           @Override
           public void sessionCreated(IoSession session) {
               codecAdapter.sessionCreated(session);
           }

           @Override
           public void sessionDestroyed(IoSession session) {
               codecAdapter.sessionDestroyed(session);
           }
        });

2.监听session的create和destroy事件，传递到decoder中，decoder中，通过session和buffer的键值对保存对不同通道的数据的缓存，

private Map buffers = new ConcurrentHashMap();

       // ChannelBuffers.EMPTY_BUFFER;

       public void sessionCreated(IoSession session) {
           buffers.put(session, ChannelBuffers.EMPTY_BUFFER);
       }

       public void sessionDestroyed(IoSession session) {
           buffers.remove(session);
       }

3.解码时通过session获得当前channel的数据

ChannelBuffer buffer = buffers.get(session);
if(buffer == null) return;

经过测试，问题得以解决

小马歌 2016-05-17 15:24 发表评论

dubbo中的那些“坑"(2)-hessian-lite字符串数据定义改进

小马歌 — Tue, 17 May 2016 07:24:00 GMT

其实这个不算是一个坑，阿里实现的字符串协议挺好的，但是由于我个人的强迫症和在编写php客户端过程中对字符串的输出和解析感觉很别扭，尤其是字符串数据很大时，必须一个字节一个字节的判断处理，让我很郁闷，明显和我当年编写汇编时的哪种精致不符。体现在两个方面

1.字符串的格式定义为字母S或者R + 两个字节的数据长度（MSB）+ utf8格式的字节数组，S表示最后一个块，上面那个长度是unicode

字符串的长度，不是自己数组的长度，即"中国"这个词，长度是2,utf8表示的字节数组确实6个字节

2.我的php客户端是用c混合c++编写的php扩展，输出字符串时首先调用libmbfl库计算unicode字符串的长度，然后输出utf8数据,因为在php中，我们默认采用utf8格式，字符串zval已经是utf8格式了，并且附带一个utf8长度的整数，

3.读取应答分析字符串时，根据上面的长度并不知道该分配多少内存来接受整个字符串，因为长度和utf8的字节长度根本没有关系，还有多个节时防止utf8字母被分配到多个不同的chunk上面

4.因此，我们重新定义了一个字符串数据协议， E +4字节UTF8字节长度（MSB）+ utf8表示一个完整的字符串，4字节时，表示的数据有2^31-1个utf8字节，可以表示好几百兆的汉字，够用了，这样我们可以使用c语言的memcpy函数进行快速数据拷贝和数据缓冲区分配了

5.经过测试，大量字符串的数据传输性能能提高5%

小马歌 2016-05-17 15:24 发表评论

Hessian和Java反序列化问题小结

小马歌 — Tue, 17 May 2016 07:24:00 GMT

摘要: from:http://hittyt.iteye.com/blog/1691772Hessian反序列化问题众所周知，Hessian框架提供的序列化方式，在性能上要优于Java自己的序列化方式。他将对象序列化，生成的字节数组的数量要相对于Java自带的序列化方式要更简洁。目前公司的一个项目中，有RPC调用的需要，这里我们使用了公司自己的开源RPC框架Dubbo作为远程调用框架，进行业务方法的调用和... 阅读全文

小马歌 2016-05-17 15:24 发表评论

近期hessian反序列化问题总结与ThreadPoolExecutor使用心得

小马歌 — Tue, 17 May 2016 07:22:00 GMT

from:http://pfmiles.github.io/blog/recently-hessian-deserialize-problem-and-thread-pool-executor-experience/

最近工作中遇到一个诡异的问题：别人远程调用我们的系统暴露的服务，同步调用，底层使用hessian协议做序列化；
调用方系统报空指针，反序列化失败：

2013-04-18 16:52:10,308 [AvatarRuleChargeService.java:74] [com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService] ERROR com.alibaba.itbu.billing.biz.adaptor.crm.ChargeProxy :: avatar charge sys error com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method match in the service com.alibaba.china.ruleservice.RuleService. Tried 3 times of the providers [172.22.6.83:20980, 172.22.6.80:20980, 172.22.9.76:20980] (3/3) from the registry dubbo-reg1.hst.xyi.cn.alidc.net:9090 on the consumer 172.30.118.26 using the dubbo version 2.4.9. Last error is: Failed to invoke remote method: match, provider: dubbo://172.22.6.83:20980/com.alibaba.china.ruleservice.RuleService?anyhost=true&application=billing&check=false&default.reference.filter=dragoon&dubbo=2.4.9&interface=com.alibaba.china.ruleservice.RuleService&methods=match&pid=18616&revision=1.0-SNAPSHOT&side=consumer&timeout=5000×tamp=1366275108588&version=1.0.0, cause: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155)     at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49)     at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135)     at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)     at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)     at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)     at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271)     ... 28 more Caused by: java.lang.NullPointerException     at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.(RuleServiceImpl.java:163)     ... 33 more      at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)     at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:226)     at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)     at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)     at com.alibaba.dubbo.common.bytecode.proxy1.match(proxy1.java)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)     at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)     at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)     at com.alibaba.itbu.billing.framework.aop.OpenApiLogAspect.logExecuteTime(OpenApiLogAspect.java:38)     at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)     at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)     at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)     at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)     at $Proxy107.match(Unknown Source)     at com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService.chargeByFactor(AvatarRuleChargeService.java:72)     at com.alibaba.itbu.billing.biz.charge.times.RuleChargeByTimesProcessor.getChargeResult(RuleChargeByTimesProcessor.java:62)     at com.alibaba.itbu.billing.biz.charge.times.ChargeByTimesProcessor.charge(ChargeByTimesProcessor.java:117)     at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask.charge(ChargeByIncInstantTimesTask.java:174)     at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask$1.run(ChargeByIncInstantTimesTask.java:109)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: com.alibaba.dubbo.remoting.RemotingException: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155)     at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49)     at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135)     at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)     at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)     at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)     at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271)     ... 28 more Caused by: java.lang.NullPointerException     at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.(RuleServiceImpl.java:163)     ... 33 more      at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:190)     at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:110)     at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)     at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)     at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)     at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke0(DragoonFilter.java:82)     at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke(DragoonFilter.java:34)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)     at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)     ... 32 more

看到这个日志第一反映是觉得RuleServiceImpl.java:163数据错误，抛空指针，但检查那块代码发现那个地方根本不可能抛空指针 —— 所有用到的变量都是new出来的；
而且，依据调用方提供的错误日志的抛出时间，我在被调用方系统的所有机器的日志里查找了一遍，没有发现对应的服务端日志；照理说服务端抛空指针，服务端也会有对应日志，但没有任何线索…

因此只好翻开了JavaDeserializer.java:275作检查，发现有这么一段：

  public JavaDeserializer(Class cl)   {     _type = cl;     _fieldMap = getFieldMap(cl);      _readResolve = getReadResolve(cl);      if (_readResolve != null) {       _readResolve.setAccessible(true);     }      Constructor []constructors = cl.getDeclaredConstructors();     long bestCost = Long.MAX_VALUE;      for (int i = 0; i < constructors.length; i++) {       Class []param = constructors[i].getParameterTypes();       long cost = 0;        for (int j = 0; j < param.length; j++) {     cost = 4 * cost;      if (Object.class.equals(param[j]))       cost += 1;     else if (String.class.equals(param[j]))       cost += 2;     else if (int.class.equals(param[j]))       cost += 3;     else if (long.class.equals(param[j]))       cost += 4;     else if (param[j].isPrimitive())       cost += 5;     else       cost += 6;       }        if (cost < 0 || cost > (1 << 48))     cost = 1 << 48;        cost += param.length << 48;        if (cost < bestCost) {         _constructor = constructors[i];         bestCost = cost;       }     }      if (_constructor != null) {       _constructor.setAccessible(true);       Class []params = _constructor.getParameterTypes();       _constructorArgs = new Object[params.length];       for (int i = 0; i < params.length; i++) {         _constructorArgs[i] = getParamArg(params[i]);       }     }   }

看完这段后，再结合远程调用的返回结果类后恍然大悟：
上面这段代码，是hessian在反序列化的时候，用于在被反序列化的类里面找一个“得分最低”的构造函数，反序列化时会加以调用;
构造函数的“得分”规则大致是：参数越少得分越低；参数个数相同时，参数类型越接近JDK内置类得分越低
而我们的调用返回的类只有一个构造函数，当然只有这个构造函数会被选中
但是，hessian反序列化调用被选中的构造函数时，是这样来创造该构造函数需要的参数的：

protected static Object getParamArg(Class cl)   {     if (! cl.isPrimitive())       return null;     else if (boolean.class.equals(cl))       return Boolean.FALSE;     else if (byte.class.equals(cl))       return new Byte((byte) 0);     else if (short.class.equals(cl))       return new Short((short) 0);     else if (char.class.equals(cl))       return new Character((char) 0);     else if (int.class.equals(cl))       return new Integer(0);     else if (long.class.equals(cl))       return new Long(0);     else if (float.class.equals(cl))       return new Float(0);     else if (double.class.equals(cl))       return new Double(0);     else       throw new UnsupportedOperationException();   }

可以看到，如果参数不是primitive类型，会被null代替；但不巧的是我们的构造函数内部对该参数调用了一个方法，因此抛出了空指针…
也就是说这个空指针是抛在客户端反序列化的时候而不是服务端内部，因此服务端当然找不到对应的错误日志了；
解决的办法也很简单：可以新增一个无参构造函数(无参构造函数肯定“得分”最低一定会被选中)；或者修改代码保证任意参数为null的时候都不会出问题

下面这个问题更有意思，说的是ThreadPoolExecutor的RejectedExecutionHandler的使用：

threadPool = new ThreadPoolExecutor(5, maxThreadNum, 5, TimeUnit.MINUTES, new SynchronousQueue(), tf,             new RejectedExecutionHandler() {                 public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {                     logger.error("Ep thread pool exhausted, epId: " + id + "!!");                     r.run();                 }             });

这个代码是说，当我这个ThreadPool不够用，又不能再新增线程数的时候，由调用方线程自己来执行这个Runnable任务…
本来这看上去没什么问题，问题出在这个Runnable本身的实现上 —— 它内部将执行它的线程block到了一个blocking queue上面，当调用方主线程亲自来执行它时，使得主线程再也回不去做它自己该做的事情了，因此会出大问题…
所以这么看来，“当线程池不够用就让调用方线程自己来干”的这个策略在实际使用时要非常谨慎

这个问题是通过一个方便的thread dump分析工具: tda来查找的，因为出问题的这个应用是个多线程程序，有1600多个常驻线程，thread dump非常之大，肉眼直接看很不方便，但tda能将thread dump变得更友好易读，方便排查问题，在此推荐一下

小马歌 2016-05-17 15:22 发表评论