﻿<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/"><channel><title>BlogJava-banxitan-随笔分类-Mysql</title><link>http://www.blogjava.net/banxitan/category/53406.html</link><description /><language>zh-cn</language><lastBuildDate>Wed, 13 Mar 2013 12:41:00 GMT</lastBuildDate><pubDate>Wed, 13 Mar 2013 12:41:00 GMT</pubDate><ttl>60</ttl><item><title>MySql 的批量操作，要加rewriteBatchedStatements参数 </title><link>http://www.blogjava.net/banxitan/archive/2013/03/13/396409.html</link><dc:creator>MikyTan</dc:creator><author>MikyTan</author><pubDate>Wed, 13 Mar 2013 12:38:00 GMT</pubDate><guid>http://www.blogjava.net/banxitan/archive/2013/03/13/396409.html</guid><wfw:comment>http://www.blogjava.net/banxitan/comments/396409.html</wfw:comment><comments>http://www.blogjava.net/banxitan/archive/2013/03/13/396409.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.blogjava.net/banxitan/comments/commentRss/396409.html</wfw:commentRss><trackback:ping>http://www.blogjava.net/banxitan/services/trackbacks/396409.html</trackback:ping><description><![CDATA[<div>今天在做某项目的POC测试，甲方提供了一个三十万记录的TXT数据文件，需要把该文件的记录插入到数据库中，由于项目部的同事在搭建测试环境中用的是Mysql数据库，在把数据导入到数据库中用的是JDBC的批处理。代码如下
<div style="border-bottom: #cccccc 1px solid; border-left: #cccccc 1px solid; padding-bottom: 4px; background-color: #eeeeee; padding-left: 4px; width: 98%; padding-right: 5px; font-size: 13px; word-break: break-all; border-top: #cccccc 1px solid; border-right: #cccccc 1px solid; padding-top: 4px"><!--<br /><br />Code highlighting produced by Actipro CodeHighlighter (freeware)<br />http://www.CodeHighlighter.com/<br /><br />--><img id="Codehighlighter1_30_999_Open_Image" onclick="this.style.display='none'; Codehighlighter1_30_999_Open_Text.style.display='none'; Codehighlighter1_30_999_Closed_Image.style.display='inline'; Codehighlighter1_30_999_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedBlockStart.gif"><img style="display: none" id="Codehighlighter1_30_999_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_30_999_Closed_Text.style.display='none'; Codehighlighter1_30_999_Open_Image.style.display='inline'; Codehighlighter1_30_999_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedBlock.gif"><span style="color: #0000ff">private</span><span style="color: #000000">&nbsp;</span><span style="color: #0000ff">void</span><span style="color: #000000">&nbsp;batchParseGroup()</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_30_999_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_30_999_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Connection&nbsp;con</span><span style="color: #000000">=</span><span style="color: #000000">&nbsp;</span><span style="color: #0000ff">null</span><span style="color: #000000">;<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PreparedStatement&nbsp;ps&nbsp;</span><span style="color: #000000">=</span><span style="color: #0000ff">null</span><span style="color: #000000">;<br /><img id="Codehighlighter1_92_882_Open_Image" onclick="this.style.display='none'; Codehighlighter1_92_882_Open_Text.style.display='none'; Codehighlighter1_92_882_Closed_Image.style.display='inline'; Codehighlighter1_92_882_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockStart.gif"><img style="display: none" id="Codehighlighter1_92_882_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_92_882_Closed_Text.style.display='none'; Codehighlighter1_92_882_Open_Image.style.display='inline'; Codehighlighter1_92_882_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedSubBlock.gif">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span style="color: #0000ff">try</span><span style="color: #000000">&nbsp;</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_92_882_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_92_882_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;con</span><span style="color: #000000">=</span><span style="color: #000000">&nbsp;DbConnectionManager.getConnection();<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;con.setAutoCommit(</span><span style="color: #0000ff">false</span><span style="color: #000000">);<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;String&nbsp;sql&nbsp;</span><span style="color: #000000">=</span><span style="color: #000000">&nbsp;</span><span style="color: #000000">"</span><span style="color: #000000">&nbsp;insert&nbsp;into&nbsp;jivegroup(uri,groupname,pgroupid,description,creationdate,modificationdate,priority,selfpriority)&nbsp;values(?,?,?,?,?,?,?,?)</span><span style="color: #000000">"</span><span style="color: #000000">;<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps</span><span style="color: #000000">=</span><span style="color: #000000">&nbsp;con.prepareStatement(sql);<br /><img id="Codehighlighter1_394_839_Open_Image" onclick="this.style.display='none'; Codehighlighter1_394_839_Open_Text.style.display='none'; Codehighlighter1_394_839_Closed_Image.style.display='inline'; Codehighlighter1_394_839_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockStart.gif"><img style="display: none" id="Codehighlighter1_394_839_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_394_839_Closed_Text.style.display='none'; Codehighlighter1_394_839_Open_Image.style.display='inline'; Codehighlighter1_394_839_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedSubBlock.gif">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span style="color: #0000ff">for</span><span style="color: #000000">(</span><span style="color: #0000ff">int</span><span style="color: #000000">&nbsp;i</span><span style="color: #000000">=</span><span style="color: #000000">0</span><span style="color: #000000">;i</span><span style="color: #000000">&lt;</span><span style="color: #000000">groupList.size();i</span><span style="color: #000000">++</span><span style="color: #000000">)</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_394_839_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_394_839_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Group&nbsp;group&nbsp;</span><span style="color: #000000">=</span><span style="color: #000000">&nbsp;groupList.get(i);<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">1</span><span style="color: #000000">,&nbsp;group.getUri());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">2</span><span style="color: #000000">,&nbsp;group.getName());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">3</span><span style="color: #000000">,&nbsp;group.getPgroupId());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">4</span><span style="color: #000000">,&nbsp;group.getName());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">5</span><span style="color: #000000">,&nbsp;</span><span style="color: #000000">""</span><span style="color: #000000">+</span><span style="color: #000000">System.currentTimeMillis());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setString(</span><span style="color: #000000">6</span><span style="color: #000000">,&nbsp;</span><span style="color: #000000">""</span><span style="color: #000000">+</span><span style="color: #000000">System.currentTimeMillis());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setInt(</span><span style="color: #000000">7</span><span style="color: #000000">,&nbsp;group.getPriority());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.setInt(</span><span style="color: #000000">8</span><span style="color: #000000">,&nbsp;group.getPriority());<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.addBatch();<br /><img id="Codehighlighter1_804_834_Open_Image" onclick="this.style.display='none'; Codehighlighter1_804_834_Open_Text.style.display='none'; Codehighlighter1_804_834_Closed_Image.style.display='inline'; Codehighlighter1_804_834_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockStart.gif"><img style="display: none" id="Codehighlighter1_804_834_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_804_834_Closed_Text.style.display='none'; Codehighlighter1_804_834_Open_Image.style.display='inline'; Codehighlighter1_804_834_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedSubBlock.gif">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</span><span style="color: #0000ff">if</span><span style="color: #000000">(i</span><span style="color: #000000">%</span><span style="color: #000000">100</span><span style="color: #000000">==</span><span style="color: #000000">0</span><span style="color: #000000">)</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_804_834_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_804_834_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.executeBatch();<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockEnd.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span></span><span style="color: #000000"><br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockEnd.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span></span><span style="color: #000000"><br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;con.commit();<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ps.executeBatch();<br /><img id="Codehighlighter1_907_935_Open_Image" onclick="this.style.display='none'; Codehighlighter1_907_935_Open_Text.style.display='none'; Codehighlighter1_907_935_Closed_Image.style.display='inline'; Codehighlighter1_907_935_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockStart.gif"><img style="display: none" id="Codehighlighter1_907_935_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_907_935_Closed_Text.style.display='none'; Codehighlighter1_907_935_Open_Image.style.display='inline'; Codehighlighter1_907_935_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedSubBlock.gif">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span></span><span style="color: #000000">&nbsp;</span><span style="color: #0000ff">catch</span><span style="color: #000000">&nbsp;(SQLException&nbsp;e)&nbsp;</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_907_935_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_907_935_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;e.printStackTrace();<br /><img id="Codehighlighter1_943_996_Open_Image" onclick="this.style.display='none'; Codehighlighter1_943_996_Open_Text.style.display='none'; Codehighlighter1_943_996_Closed_Image.style.display='inline'; Codehighlighter1_943_996_Closed_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockStart.gif"><img style="display: none" id="Codehighlighter1_943_996_Closed_Image" onclick="this.style.display='none'; Codehighlighter1_943_996_Closed_Text.style.display='none'; Codehighlighter1_943_996_Open_Image.style.display='inline'; Codehighlighter1_943_996_Open_Text.style.display='inline';" align="top" src="http://www.blogjava.net/images/OutliningIndicators/ContractedSubBlock.gif">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span></span><span style="color: #0000ff">finally</span><span style="border-bottom: #808080 1px solid; border-left: #808080 1px solid; background-color: #ffffff; display: none; border-top: #808080 1px solid; border-right: #808080 1px solid" id="Codehighlighter1_943_996_Closed_Text"><img src="http://www.blogjava.net/Images/dot.gif"  alt="" /></span><span id="Codehighlighter1_943_996_Open_Text"><span style="color: #000000">{<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/InBlock.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DbConnectionManager.closeConnection(ps,&nbsp;con);<br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedSubBlockEnd.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}</span></span><span style="color: #000000"><br /><img align="top" src="http://www.blogjava.net/images/OutliningIndicators/ExpandedBlockEnd.gif"  alt="" />&nbsp;&nbsp;&nbsp;&nbsp;}</span></span></div><br />在测试时，发现三十万的数据居然需要十分钟左右的时间。首先想到的就是Mysql的相关配置是不是有问题，反复修改了Mysql的相应配置参数，收效甚微。<br /><br />在Mysql的官网上查到如下：<br />&nbsp;<a href="http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-configuration-properties.html">http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-configuration-properties.html</a><br /><br />
<p>关于rewriteBatchedStatements参数，Mysql官方的说明：</p>
<p>Should the driver use multiqueries (irregardless of the setting of "allowMultiQueries") as well as rewriting of prepared statements for INSERT into multi-value inserts when executeBatch() is called? Notice that this has the potential for SQL injection if using plain java.sql.Statements and your code doesn't sanitize input correctly. Notice that for prepared statements, server-side prepared statements can not currently take advantage of this rewrite option, and that if you don't specify stream lengths when using PreparedStatement.set*Stream(), the driver won't be able to determine the optimum number of parameters per batch and you might receive an error from the driver that the resultant packet is too large. Statement.getGeneratedKeys() for these rewritten statements only works when the entire batch includes INSERT statements.</p><br /><br />解决办法：<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 下载最新的JDBC的驱动程序。<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; MYSQL URL的配置参数如下：<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; jdbc:mysql://54.200.190.80:3306/ccb_ucstar?rewriteBatchedStatements=true<br /><br />经过测试。三十多万的数据。70秒内搞定！</div><img src ="http://www.blogjava.net/banxitan/aggbug/396409.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.blogjava.net/banxitan/" target="_blank">MikyTan</a> 2013-03-13 20:38 <a href="http://www.blogjava.net/banxitan/archive/2013/03/13/396409.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item></channel></rss>