哈，又一款超级简单的队列（MQ）实现方案来了~

开源的消息队列已经很多了，但大部分很重，实际环境下，很多可能只是使用到了一点功能而已，杀鸡使用牛刀，着实有些浪费了。很多时候，我们只想要一片绿叶，但它们给了我们整个的春天，很难消化。本着DIR精神，也琢磨了一个超级简单的队列实现。

说是超级简单，嗯，绝对是超级简单，队列的存储采用Redis进行持久化存储，采用Netty提供HTTP方式的队列的出/入。Redis的客户端采用的Jedis。然后呢，然后就没了啊。

一。Redis

Redis内置订阅发布模型(Publish/Subscribe),其缺陷是，不存储，一旦订阅者断线，将无法接收到消息，再次连接上之后，在断线期间发布者发布的消息都是无法获取到的。只能采用list数组实现，采用rpush/lpop组合命令来实现先进先出的队列模型，当然redis也提供了阻塞版本的blpush/brpush/blpop/brpop等，就看我们实际环境下如何使用了。

JAVA客户端使用Jedis，提供接口也很丰富。但要注意的是，需要使用连接池，否则在大数量的情况下，有可能jedis的连接不够用。

private static JedisPool pool;
 static {
  ResourceBundle bundle = ResourceBundle.getBundle("redis");
  if (bundle == null) {
   throw new IllegalArgumentException(
     "cannot find the SignVerProp.properties");
  }
  JedisPoolConfig config = new JedisPoolConfig();
  config.setMaxActive(Integer.valueOf(bundle
    .getString("redis.pool.maxActive")));
  config.setMaxIdle(Integer.valueOf(bundle
    .getString("redis.pool.maxIdle")));
  config.setMaxWait(Integer.valueOf(bundle
    .getString("redis.pool.maxWait")));
  pool = new JedisPool(config, bundle.getString("redis.server"),
    Integer.valueOf(bundle.getString("redis.port")));
 }

二。Netty

很成熟的NIO框架，用它来提供HTTP方式的队列的出入。嗯，目前只提供HTTP方式的入队列，出队列等。
HTTP形式为：http://服务器名称:端口/队列操作原语/队列名称?msg=消息内容
队列操作原语，只有get和put；get为出队列，put为入队列。
返回为json： {s:0, m:'错误消息/消息内容'}
s为success的缩写，值为0，意味着false，1对应成功。 m为message的缩写，错误消息/消息内容，添加消息时，会返回插入消息对应的数目默认采用UTF-8。

入队列：http://localhost:8080/put/demo?msg=消息内容
出队列：http://localhost:8080/get/demo HTTP方式，特别适合局域网之间，消息数据的推送。

三。入口解读

采用了QueueDaemon封装了netty代码：调用很简单，程序启动的入口：而HTTP方式队列请求处理器为HttpRequestHandler：

四。队列处理器

这个很简单，直接采用Jedis客户端即可，这里一直占用一个连接，不释放。

五。ab压力测试

本机配置：Pentium(R) Dual-Core CPU E5200 2.50GHz，2.00 GB内存，Windows XP系统，redis-2.4.5-win32 32版本（非linux版本），都在一台机器上运行。 插入 512 bytes 文本消息队列: 1781.24 requests/sec

D:\Apache2.2\bin>ab -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        12 bytes
Concurrency Level:      10
Time taken for tests:   56.141 seconds
Complete requests:      100000
Failed requests:        99991
   (Connect: 0, Receive: 0, Length: 99991, Exceptions: 0)
Write errors:           0
Total transferred:      8188895 bytes
HTML transferred:       1588895 bytes
Requests per second:    1781.24 [#/sec] (mean)
Time per request:       5.614 [ms] (mean)
Time per request:       0.561 [ms] (mean, across all concurrent requests)
Transfer rate:          142.45 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.7      0      16
Processing:     0    5  10.7      0     781
Waiting:        0    5  10.4      0     766
Total:          0    6  10.8      0     781
Percentage of the requests served within a certain time (ms)
  50%      0
  66%     16
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    781 (longest request)

插入 512 bytes 文本消息队列(添加Keep-Alive支持): 1875.18 requests/sec

D:\Apache2.2\bin>ab -k -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        17 bytes
Concurrency Level:      10
Time taken for tests:   53.328 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      8300000 bytes
HTML transferred:       1700000 bytes
Requests per second:    1875.18 [#/sec] (mean)
Time per request:       5.333 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          151.99 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.5      0      16
Processing:     0    5   7.8      0     203
Waiting:        0    5   7.8      0     203
Total:          0    5   7.9      0     203
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    203 (longest request)

获取 512 bytes 消息(With Keep-Alive): 1875.73 requests/sec

D:\Apache2.2\bin>ab -k -c 10 -n 100000 "http://localhost:8080/get/demo"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /get/demo
Document Length:        523 bytes
Concurrency Level:      10
Time taken for tests:   53.313 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      58900000 bytes
HTML transferred:       52300000 bytes
Requests per second:    1875.73 [#/sec] (mean)
Time per request:       5.331 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          1078.91 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.9      0      16
Processing:     0    5   7.5      0      94
Waiting:        0    4   6.9      0      94
Total:          0    5   7.6      0      94
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%     94 (longest request)

其它问题

暂时对安全/授权没有支持，这个其实很容易实现
队列处理器很简单，直接使用jedis即可
对队列数据进行分片(sharding)，写一个QueueService实现即可
对Redis的内存持久化不放心，采用diskstore存储模式好了，在Redis中配置，不涉及到程序
对队列分布式存储，写一个QueueService实现，其它可不用变化
不适合严格意义上的订阅/发布模型，这里适合多个发布者/单个订阅者环境
HTTP请求返回内容为json格式，xml格式，暂时不需要
局域网环境下，系统之间进行消息的推送/通知，

项目下载地址：http://code.google.com/p/nettyqueue/ 参考资料：

httpsqs

posted on 2012-03-20 20:30 nieyong 阅读(24475) 评论(4) 编辑收藏所属分类: Java

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2012-03-21 13:47 3G生活

学习了，呵呵回复更多评论

# 按摩保健器材 2012-03-21 14:32 富尔乐

这一长串的代码羊肠子啊！写的人也忒厉害了！回复更多评论

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2013-07-30 16:13 辉煌

表示很高深呀. 回复更多评论

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2015-02-13 14:23 gggggttttt

为什么要用netty（http）呢？回复更多评论

新用户注册刷新评论列表


只有注册用户登录后才能发表评论。




网站导航: 博客园 IT新闻知识库 C++博客博问管理
相关文章: 哈，又一款超级简单的队列（MQ）实现方案来了~ Fork/Join模式(JSR166y)手记之Fork/Join模式实现浅析1 Fork/Join模式(JSR166y)手记之斐波纳契数列(Fibonacci)求解测试 Fork/Join模式(JSR166y)手记之Phaser Fork/Join模式(JSR166y)手记之ConcurrentLinkedDeque Fork/Join模式(JSR166y)手记之ThreadLocalRandom Fork/Join模式(JSR166y)手记之TransferQueue/LinkedTransferQueue Fork/Join模式(JSR166y)手记之JVAA6搭建环境,开始学习 Servlet 3.0笔记之文件下载的那点事情 Servlet 3.0笔记之Servlet的异步特性支持失效怎么办？

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2012-03-21 13:47 3G生活

# 按摩保健器材 2012-03-21 14:32 富尔乐

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2013-07-30 16:13 辉煌

# re: 哈，又一款超级简单的队列（MQ）实现方案来了~ 2015-02-13 14:23 gggggttttt

聂永的博客