paulwong

Mongodb 查询性能的事情

上一篇：Mongodb VS Mysql 查询性能，测试了 mongodb 与 mysql 的查询性能。结果说明 mongodb 性能可以， 可以代替 mysql 来使用。

但是这个测试都是在百万级别，我的场景在 KW 级别。所以还要对 mongodb 在 kw 级别下测试效果。

我测试环境是 4G 内存（有好些内存被其它程序占用），2kw 数据，查询随机生成 id（一次查询 20 个id）。

在这样的环境中测试不理想，比较失望。平均一次查询 500ms（比 mysql 还差，特别是在并发查询下，性能较差。很底的吞吐量）。查看其索引大小（用 db.mycoll.stats() 可以查询）：2kw 数据中有 1.1G 左右的索引，存储的数据在 11G 左右。

测试过程中发现 iowait 占 50% 左右，看来还是 io 的瓶颈。还看到 mongodb 使用的内存不多（小于索引的大小，看来这机器不足够来测试）。

换了个有可用 6G 内存的机器。在 50 个并发下，可以达到平均 100 ms 左右，算比较满意，但是并发好像能力不够强。但这个性能不能由我控制，还由机器的可用内存控制。原因就是 mongodb 没有指定可占用的内存大小，它把所有空闲内存当缓存使用，既是优点也是缺点：优点--可以最大限度提升性能；缺点--容易受其它程序干扰（占用了它的缓存）。由我测试来看，它抢占内存的能力不强。mongodb 是用内存映射文件 vmm，官方的说明：

Memory Mapped Storage Engine

This is the current storage engine for MongoDB, and it uses memory-mapped files for all disk I/O. Using this strategy, the operating system's virtual memory manager is in charge of caching. This has several implications:

There is no redundancy between file system cache and database cache: they are one and the same.
MongoDB can use all free memory on the server for cache space automatically without any configuration of a cache size.
Virtual memory size and resident size will appear to be very large for the mongod process. This is benign: virtual memory space will be just larger than the size of the datafiles open and mapped; resident size will vary depending on the amount of memory not used by other processes on the machine.
Caching behavior (such as LRU'ing out of pages, and laziness of page writes) is controlled by the operating system: quality of the VMM implementation will vary by OS.
所以这么来看，我觉得 mongodb 没有指定内存大小来保证正常的缓存是个缺点。应该至少保证索引全部能放到内存中。但这个行为不是由启动程序决定，而是由环境决定（美中不足）。

官方也有段内容说到索引放到内存中：

If your queries seem sluggish, you should verify that your indexes are small enough to fit in RAM. For instance, if you're running on 4GB RAM and you have 3GB of indexes, then your indexes probably aren't fitting in RAM. You may need to add RAM and/or verify that all the indexes you've created are actually being used.

还是希望 mongodb 中可以指定内存大小，确保它有足够内存加载索引。

小结：大数据量下（kw级）mongodb 并发查询不够理想（100-200/s）。写数据很快（我的环境，远程提交近 1w/s，估计达到 1.5W/s 是没问题的，基本不受大数据量的影响）。

贴个测试数据：

1 id(内存使用 <1.5g) 10 id(内存使用 2-3g) 20 id(内存使用 >4g)
1 2 3 1 2 3 1 2 3
total time 17.136 25.508 17.387 37.138 33.788 25.143 44.75 31.167 30.678
1 thread thruput 583.5668 392.0339 575.1423 269.266 295.9631 397.725 223.4637 320.8522 325.9665
total time 24.405 22.664 24.115 41.454 41.889 39.749 56.138 53.713 54.666
5 thread thruput 2048.76 2206.142 2073.398 1206.156 1193.631 1257.893 890.6623 930.8733 914.6453
total time 27.567 26.867 28.349 55.672 54.347 50.93 72.978 81.857 75.925
10 thread thruput 3627.526 3722.038 3527.461 1796.235 1840.028 1963.479 1370.276 1221.643 1317.089
total time 51.397 57.446 53.81 119.386 118.015 76.405 188.962 188.034 138.839
20 thread thruput 3891.278 3481.53 3716.781 1675.238 1694.7 2617.63 1058.414 1063.637 1440.517
total time 160.038 160.808 160.346 343.559 352.732 460.678 610.907 609.986 1411.306
50 thread thruput 3124.258 3109.298 3118.257 1455.354 1417.507 1085.357 818.4552 819.6909 354.2818
total time 2165.408 635.887 592.958 1090.264 1034.057 1060.266 1432.296 1466.971 1475.061
100 thread thruput 461.8067 1572.606 1686.46 917.209 967.0647 943.1595 698.1797 681.6767 677.9381
上面的测试分别用三种查询（每次 1,10,20 id），在不同并发下测试3次，每次发出 1w 次查询。第一行数据为所有线程累加时间（单位 ms），第二行数据为吞吐量(1w /(total time / thread num))。测试中内存使用慢慢增加，所以后面的数据可能比较高效的（高效的环境）。

从上表看，10 - 20线程比较高的吞吐量。看到内存使用，前提就是索引加载到内存中，并有些内存作为缓存。

下面有个索引查询优化的 pdf。

Indexing and Query Optimizer

Indexing and Query Optimizer (Aaron Staple)
ps:

默认 mongodb 服务器只有10个并发，如果要提高它的连接数，可以用 --maxConns num 来提高它的接收并发的数据。

mongodb 的 java 驱动默认最多只有 10 并发连接池。要提高它，可以在 mongo.jar 的环境中加入 MONGO.POOLSIZE 系统参数，如 java -DMONGO.POOLSIZE=50 ...

posted @ 2015-01-06 23:39 paulwong 阅读(726) | 评论 (0) | 编辑 收藏

关于MongoDB最大连接数的查看与修改

在Linux平台下，无论是64位或者32位的MongoDB默认最大连接数都是819，WIN平台不知道，估计也没有人在 WIN平台下使用MongoDB做生产环境


途中available显示818少了一个，表示空闲的。current表示已经占用了的连接数，两数一加就等于819，如果我现在在连接一个，那么available就是817，current就是2


819个连接数对于一般的站点我认为已经够用，并且都是现连现取现断。但这个连接数也可以修改，只要在启动的时候加入--maxConns即可
服务器启动


查询最大连接数


发现还是819？其实是Linux默认进程能打开最大文件数有关，可以通过ulimit 解决


再查看最大连接数，搞定


关于ulimit的更多知识大家可以去网上检索检索

客户端程序通常是通过DRIVER来链接，由于每次建立链接的成本都挺高，因此都用链接池来实现，SPRING DATA MONGODB中是如下配置

• autoConnectRetry simply means the driver will automatically attempt to reconnect to the server(s) after unexpected disconnects. In production environments you usually want this set to true.

• connectionsPerHost are the amount of physical connections a single Mongo instance (it's singleton so you usually have one per application) can establish to a mongod/mongos process. At time of writing the java driver will establish this amount of connections eventually even if the actual query throughput is low (in order words you will see the "conn" statistic in mongostat rise until it hits this number per app server).

  There is no need to set this higher than 100 in most cases but this setting is one of those "test it and see" things. Do note that you will have to make sure you set this low enough so that the total amount of connections to your server do not exceed

  db.serverStatus().connections.available

  In production we currently have this at 40.

• connectTimeout. As the name suggest number of milliseconds the driver will wait before a connection attempt is aborted. Set timeout to something long (15-30 seconds) unless there's a realistic, expected chance this will be in the way of otherwise succesful connection attempts. Normally if a connection attempt takes longer than a couple of seconds your network infrastructure isn't capable of high throughput.

• maxWaitTime. Number of ms a thread will wait for a connection to become available on the connection pool, and raises an exception if this does not happen in time. Keep default.

• socketTimeout. Standard socket timeout value. Set to 60 seconds (60000).

• threadsAllowedToBlockForConnectionMultiplier. Multiplier for connectionsPerHost that denotes the number of threads that are allowed to wait for connections to become available if the pool is currently exhausted. This is the setting that will cause the "com.mongodb.DBPortPool$SemaphoresOut: Out of semaphores to get db connection" exception. It will throw this exception once this thread queue exceeds the threadsAllowedToBlockForConnectionMultiplier value. For example, if the connectionsPerHost is 10 and this value is 5 up to 50 threads can block before the aforementioned exception is thrown.

  If you expect big peaks in throughput that could cause large queues temporarily increase this value. We have it at 1500 at the moment for exactly that reason. If your query load consistently outpaces the server you should just improve your hardware/scaling situation accordingly.

• readPreference. (UPDATED, 2.8+) Used to determine the default read preference and replaces "slaveOk". Set up a ReadPreference through one of the class factory method. A full description of the most common settings can be found at the end of this post

• w. (UPDATED, 2.6+) This value determines the "safety" of the write. When this value is -1 the write will not report any errors regardless of network or database errors. WriteConcern.NONE is the appropriate predefined WriteConcern for this. If w is 0 then network errors will make the write fail but mongo errors will not. This is typically referred to as "fire and forget" writes and should be used when performance is more important than consistency and durability. Use WriteConcern.NORMAL for this mode.

  If you set w to 1 or higher the write is considered safe. Safe writes perform the write and follow it up by a request to the server to make sure the write succeeded or retrieve an error value if it did not (in other words, it sends a getLastError() command after you write). Note that until this getLastError() command is completed the connection is reserved. As a result of that and the additional command the throughput will be signficantly lower than writes with w <= 0. With a w value of exactly 1 MongoDB guarantees the write succeeded (or verifiably failed) on the instance you sent the write to.

  In the case of replica sets you can use higher values for w whcih tell MongoDB to send the write to at least "w" members of the replica set before returning (or more accurately, wait for the replication of your write to "w" members). You can also set w to the string "majority" which tells MongoDB to perform the write to the majority of replica set members (WriteConcern.MAJORITY). Typicall you should set this to 1 unless you need raw performance (-1 or 0) or replicated writes (>1). Values higher than 1 have a considerable impact on write throughput.

• fsync. Durability option that forces mongo to flush to disk after each write when enabled. I've never had any durability issues related to a write backlog so we have this on false (the default) in production.

• j *(NEW 2.7+)*. Boolean that when set to true forces MongoDB to wait for a successful journaling group commit before returning. If you have journaling enabled you can enable this for additional durability. Refer to http://www.mongodb.org/display/DOCS/Journaling to see what journaling gets you (and thus why you might want to enable this flag).

ReadPreference The ReadPreference class allows you to configure to what mongod instances queries are routed if you are working with replica sets. The following options are available :

• ReadPreference.primary() : All reads go to the repset primary member only. Use this if you require all queries to return consistent (the most recently written) data. This is the default.

• ReadPreference.primaryPreferred() : All reads go to the repset primary member if possible but may query secondary members if the primary node is not available. As such if the primary becomes unavailable reads become eventually consistent, but only if the primary is unavailable.

• ReadPreference.secondary() : All reads go to secondary repset members and the primary member is used for writes only. Use this only if you can live with eventually consistent reads. Additional repset members can be used to scale up read performance although there are limits to the amount of (voting) members a repset can have.

• ReadPreference.secondaryPreferred() : All reads go to secondary repset members if any of them are available. The primary member is used exclusively for writes unless all secondary members become unavailable. Other than the fallback to the primary member for reads this is the same as ReadPreference.secondary().

• ReadPreference.nearest() : Reads go to the nearest repset member available to the database client. Use only if eventually consistent reads are acceptable. The nearest member is the member with the lowest latency between the client and the various repset members. Since busy members will eventually have higher latencies this should also automatically balance read load although in my experience secondary(Preferred) seems to do so better if member latencies are relatively consistent.

Note : All of the above have tag enabled versions of the same method which return TaggableReadPreference instances instead. A full description of replica set tags can be found here :Replica Set Tags

参考网址：
http://api.mongodb.org/java/2.10.1/com/mongodb/MongoClientOptions.Builder.html#connectionsPerHost(int)
https://github.com/spring-projects/spring-data-mongodb/blob/master/spring-data-mongodb/src/main/resources/org/springframework/data/mongodb/config/spring-mongo-1.5.xsd

posted @ 2015-01-06 22:10 paulwong 阅读(5411) | 评论 (0) | 编辑 收藏

TOMCAT调优

1. 安装AB
   
   rpm -ivh http://repo.webtatic.com/yum/centos/5/`uname -i`/webtatic-release-5-0.noarch.rpm
   yum install httpd-tools
   
   
2. 使用AB
   
   ab -r -n 100000 -c 10000 http://10.120.151.223:8080/
   需加-r，则在收到SOCKET错误的时候不会退出
   这段的意思是发送100000个请求，其中并发是10000个
   
   
3. 修改LINUX能打开的文件的最大数
   
   2、  修改目标
   我们的目标是：让每一个用户登录系统后系统打开的最大文件数都是我们设定好的。
   但我这里不得不说的是：非常遗憾，网上很多这方面关于ulimit设置修改资源限制的文章，但没一篇文章管用。
   把这个目标分解为两个目标：
   
   2.1、设置对root用户登录系统生效
   这个目标可以实现起来不难
   
   2.2、设置对所有用户生效
   这个就非常麻烦了，弄不好还会把你的系统给整坏，因为要重编译Linux的内核才行！
   所以权衡之下，我只实现了第一个目标，因为第二个目标的风险太大，我想如果我之前知道这点，那么我在装系统的时候我会先做这个处理，但现在我觉得已经晚了。
   
   3、  修改的地方
   
   3.1、修改/etc/security/limits.conf
   通过 vi /etc/security/limits.conf修改其内容，在文件最后加入（数值也可以自己定义）：
   * soft  nofile = 65536
   * hard  nofile = 65536
   root soft nofile 65536
   root hard nofile 65536
   * 表示该配置对所有用户均有效,root用户要特别加两行。
   
   3.2、修改/etc/profile
   通过vi /etc/profile修改，在最后加入以下内容
   ulimit -n 65536
   然后重新登录即可生效了。
   说明：
   其实只修改/etc/profile就可以生效了，但我还是建议把/etc/security/limits.conf也修改一下。
   最后强调的是，你如果要使得修改对所有用户都生效，那么现在看来你只能重新编译Linux的内核才行。
   
   
   
4. 安装APR，参考：http://jmchung.github.io/blog/2013/09/06/centos-installing-apache-portable-runtime-apr-for-tomcat/
   
   $ wget http://apache.fayea.com//apr/apr-1.5.1.tar.gz
   $ cd /path/to/tomcat/bin
   $ tar zxvf tomcat-native.tar.gz
   $ cd tomcat-native-x.y.z-src/jni/native
   $ ./configure --with-apr=/usr/local/apr --with-ssl=/usr/lib64/openssl
   $ make install
   
   
5. 修改server.xml
   
   <Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"
           URIEncoding="UTF-8"  
                 enableLookups="false" 
                 tcpNoDelay="true"
           compression="on" compressionMinSize="2048"
           maxThreads="20000" connectionTimeout="-1"
           compressableMimeType="application/json,text/html,text/xml,text/javascript,text/css,text/plain" redirectPort="8443"/>
   
   

https的也要修改：

<Connector SSLEnabled="true" clientAuth="false"
        port="8443" keystoreFile="/root/java/keystore/server.jks" keystorePass="123456"
        protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https"
        secure="true" sslProtocol="TLS" 
    URIEncoding="UTF-8"  minSpareThreads="25" maxSpareThreads="75"
           enableLookups="false" disableUploadTimeout="true" connectionTimeout="20000"
    acceptCount="1000"  maxThreads="1000" maxProcessors="1000" minProcessors="5"
    useURIValidationHack="false" tcpNoDelay="true"
    compression="on" compressionMinSize="2048"
    compressableMimeType="application/json,text/html,text/xml,text/javascript,text/css,text/plain" />


6. JVM启动参数
   
   JAVA_OPTS="-server -Xms2048m -Xmx2048m -Xss512k -XX:+AggressiveOpts -XX:+UseBiasedLocking -XX:PermSize=128M -XX:MaxPermSize=256M -XX:+DisableExplicitGC -XX:MaxTenuringThreshold=31 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m  -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true "

参考网址：
http://www.cnblogs.com/baibaluo/archive/2011/08/23/2150305.html
http://ifeve.com/tomcat-connector-tuning-2/
http://sndapk.blog.51cto.com/5385144/1306278

posted @ 2015-01-06 17:40 paulwong 阅读(522) | 评论 (0) | 编辑 收藏

TomEE资源

Getting started with Apache TomEE
http://jaxenter.com/getting-started-with-apache-tomee-105824.html

posted @ 2015-01-05 00:10 paulwong 阅读(351) | 评论 (0) | 编辑 收藏

Architecture for Redis cache & Mongo for persistence

Caching with Spring Data Redis
http://www.javacodegeeks.com/2013/02/caching-with-spring-data-redis.html

Architecture for Redis cache & Mongo for persistence
http://stackoverflow.com/questions/11218941/architecture-for-redis-cache-mongo-for-persistence

MongoDB with redis
http://stackoverflow.com/questions/10696463/mongodb-with-redis/10721249#10721249

Caching Data in Spring Using Redis
http://caseyscarborough.com/blog/2014/12/18/caching-data-in-spring-using-redis/

Springside Redis
https://github.com/springside/springside4/wiki/Redis

Spring Cache注解+Redis
http://hanqunfeng.iteye.com/blog/2176172

posted @ 2015-01-04 15:50 paulwong 阅读(477) | 评论 (0) | 编辑 收藏

HADOOP各种框架应用领域

***** Data Analytics : Technology Area *****
1. Real Time Analytics : Apache Storm
2. In-memory Analytics : Apache Spark
3. Search Analytics : Apache Elastic search, SOLR
4. Log Analytics : Apache ELK Stack,ESK Stack(Elastic Search, Log
Stash, Spark Streaming, Kibana)
5. Batch Analytics : Apache MapReduce

***** NO SQL DB *****
1. MongoDB
2. Hbase
3. Cassandra

***** SOA *****
1. Oracle SOA
2. JBoss SOA
3. TiBco SOA
4. SOAP, RESTful Webservices 

posted @ 2015-01-04 12:57 paulwong 阅读(533) | 评论 (0) | 编辑 收藏

项目管理资源

Management Tutorials
http://www.tutorialspoint.com/management_tutorials.htm

posted @ 2014-12-29 23:16 paulwong 阅读(362) | 评论 (0) | 编辑 收藏

JAVAEE资源

http://www.importnew.com/12408.html

Java EE + MongoDb with Apache TomEE and Jongo Starter Project
http://www.javacodegeeks.com/2014/09/java-ee-mongodb-with-apache-tomee-and-jongo-starter-project.html

http://www.javacodegeeks.com/2011/11/from-spring-to-java-ee-6.html


Java EE 7 and WebSocket API for Java (JSR 356) with AngularJS on WildFly
http://www.javacodegeeks.com/2014/01/java-ee-7-and-websocket-api-for-java-jsr-356-with-angularjs-on-wildfly.html



lll

posted @ 2014-12-26 16:29 paulwong 阅读(674) | 评论 (0) | 编辑 收藏

Eclipse下svn的创建分支/合并/切换使用

       最近接项目要求，要在svn主干上创建分支，用分支来进行程序的bug修改，而主干上进行新功能的开发。分支上的bug修改完，发布后，可以合并到主干上。项目程序可以在主干和分支之间进行切换，来实现主干和分支的同时维护。

       1.创建分支

        创建分支实际上就是将程序copy一份到指定的分支目录，如下图示：

在项目名称上点击右键，弹出菜单，选择“Team”，再选择“Branch/Tag”，弹出下面的页面： 

 

上图中的“Copy to URL”填写创建新分支的路径地址，后面会将程序copy到该目录下，形成新的分支。点击“Next”:

 

 

选择当前最新的版本，点击“Next”

 

如果勾选了上图下面的switch working copy to new branch/tag，eclipse的程序项目会自动切换到分支下。这里我们不选择，待会自己切换。

这样就创建了一个1.0的分支

         2.合并

         可以从主干合并到分支，也可以从分支合并到主干，根据需要可以选择合适的选项，如下图：

 

上图中的选项：

        1) 从主干合并到分支

        2) 从分支合并到主干

        3) 将主干上的修改合并到分支

        4) 合并2个分支到主干

        5) 从主干到分支，手工指定不需要合并的修改

        6) 从主干到分支，手工指定要合并的修改

 

上图显示没有任何修改，所以不用进行合并。

 

3.切换

在项目名称上点击右键，选择“Team” –> “switch to another Branch/Tag/Revision”。

 

选择需要切换的目的地址，点击ok即可。

 

这样，在项目里就可以在主干和若干分支间进行任意切换，来实现对不同版本/分支的程序进行修改提交操作。

posted @ 2014-12-26 13:19 paulwong 阅读(997) | 评论 (0) | 编辑 收藏

Tuscany 2 的基本配置与流程

实习期间mentor让看看Tuscany——apache的一个顶级项目。之前一直没有接触过，听mentor说了下用途过后，感觉其像是alibaba的分布式服务框架Dubbo，简要看了看，似乎和RMI也有相关性。总之它也是面向服务的分布式框架。经过一下午的尝试，发现Tuscany 2和Tuscany 1还是有比较大的不同（API方面、设计思想没有深入研究不知道），网上关于Tuscany的资料也不多，所以在此写文章mark一下。

Tuscany 1的API参考这位博主http://blog.csdn.net/ajun_studio/article/details/7770023

1. 准备

首先是在工程中引入Tuscany 2.0.1的jar包，从一个Apache的China镜像上不难找到（PS：Tuscany 1在官网上已经不维护了）。

工程需要实现客户端调用远程服务器端的服务的功能。该项目中调用远程完成“加”的功能。

2. 入门

首先添加IAdd接口

然后是实现类Add

加减乘除都属于“计算”类，此处为了简便，只实现“加”方法。然后是“计算”的接口。

然后是“计算”的实现，注意一定要在setter中加入@Reference，否则会报错，这里类似于Spring的注入。

到此，基本的业务逻辑类就编写好了，接下来，是Tuscany中重要的配置文件，即XXX.composite，其本质也是一个xml。如果不是面向分布式服务，该xml文件结构类似Spring的application.xml。下面是Calculator.composite，注意配置文件中reference，name标记为add，应该也是同Spring的大写首字母规则，通过settrt注入。

最后是运行main的类

运行输出如下

3. 远程调用rmi

上面并没有将“计算”类作为一个服务，暴露给远程调用，只是实现了本地的装配。下面将其作为服务给远程调用，后面可以看到，只需要暴露一个包括host、port、serviceName

首先，修改配置文件，修改为如下样子——在本地（127.0.0.1）的8099端口暴露出名字为CalculatorRMIService的服务

服务器端的main主要功能是启动该服务，如下

客户端通过rmi调用服务器端的服务，来实现功能，如下

启动服务端输出

posted @ 2014-12-24 14:07 paulwong 阅读(1851) | 评论 (0) | 编辑 收藏