Geek asks and answers

Different ways to do bulk paste into a database from a Java application

I am looking for various ways to perform bulk insertion into a database (e.g. SQL Server 2012) from a Java application. I need to insert many objects into the database very efficiently without making as many calls in the database as there are entities.

My requirement is to perform bulk insertion of objects, where inserting an object into a database may include inserting data into one or more tables. The following are two ways I can think:

  • Dynamically generate a package of SQL statements and execute it against the database using the native JDBC support.

  • Build an XML representation of all entities, and then call the stored procedure by passing the generated XML. The stored procedure takes care of parsing the XML and injecting the objects into the database.

I am new to Java and do not have enough knowledge of the available frameworks. IMO, the above two approaches are apparently very naive and do not use the available framework. I ask experts to share various ways to ensure mass insertion along with their pros and cons. I am open to MyBatis, Spring -MyBatis, Spring -JDBC, JDBC, etc., which solves problems in an efficient way.

Thank.

+5
source share
3 answers

I have a demo, batch processing JDBC File: demo.txt Content

1899942, demo1
1899944, demo2
1899946, demo3
1899948, demo4

,

:

public class Test2 {
        public static void main(String[] args) {
            long start = System.currentTimeMillis();
            String sql = "insert into mobile_place(number,place) values(?,?)";
            int count=0;
            PreparedStatement pstmt = null;
            Connection conn = JDBCUtil.getConnection();
            try {
                pstmt = conn.prepareStatement(sql);

                InputStreamReader is = new InputStreamReader(new FileInputStream(new File("D:/CC.txt")),"utf-8");
                BufferedReader br = new BufferedReader(is);
                conn.setAutoCommit(false);

                String s1 = null;
                String s2 = null;
                while(br.readLine() != null){
                    count++;
                    String str = br.readLine().toString().trim();
                    s1 = str.substring(0, str.indexOf(","));
                    s2 = str.substring(str.indexOf(",")+1,str.length());

                    pstmt.setString(1, s1);
                    pstmt.setString(2, s2);
                    pstmt.addBatch();

                    if(count%1000==0){
                        pstmt.executeBatch();
                        conn.commit();
                        conn.close();
                        conn = JDBCUtil.getConnection();
                        conn.setAutoCommit(false);
                        pstmt = conn.prepareStatement(sql);
                    }
                    System.out.println("insert "+count+"line");
                }
                if(count%1000!=0){
                    pstmt.executeBatch();
                    conn.commit();
                }
                long end = System.currentTimeMillis();

                System.out.println("Total time spent:"+(end-start));
            } catch (Exception e) {
                e.printStackTrace();
            }finally{
                try {
                    pstmt.close();
                    conn.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                }
            }

        }
    }

    //getConnection()//get jdbc Connection
    public static Connection getConnection(){ 
    try { 
    Class.forName("com.mysql.jdbc.Driver"); 
    } catch (ClassNotFoundException e) { 
    e.printStackTrace(); 
    } 
    try { 
    conn = DriverManager.getConnection(url, userName, password); 
    } catch (SQLException e) { 
    e.printStackTrace(); 
    } 
    return conn; 
    }

, ,

. PreparedStatement [ PreparedStatement]

JDBC 3 1.use PreparedStatement :

try {   
      Class.forName("com.mysql.jdbc.Driver");   
      conn = DriverManager.getConnection(o_url, userName, password);   
      conn.setAutoCommit(false);   
      String sql = "INSERT adlogs(ip,website,yyyymmdd,hour,object_id) VALUES(?,?,?,?,?)";   
      PreparedStatement prest = conn.prepareStatement(sql,ResultSet.TYPE_SCROLL_SENSITIVE,ResultSet.CONCUR_READ_ONLY);   
      for(int x = 0; x < size; x++){   
         prest.setString(1, "192.168.1.1");   
         prest.setString(2, "localhost");   
         prest.setString(3, "20081009");   
         prest.setInt(4, 8);   
         prest.setString(5, "11111111");   
         prest.addBatch();   
      }   
      prest.executeBatch();   
      conn.commit();   
      conn.close();   
} catch (SQLException ex) {   
   Logger.getLogger(MyLogger.class.getName()).log(Level.SEVERE, null, ex);   
}

2.use Statement.addBatch :

conn.setAutoCommit(false);   
 Statement stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE, ResultSet.CONCUR_READ_ONLY);   
 for(int x = 0; x < size; x++){   
   stmt.addBatch("INSERT INTO adlogs(ip,website,yyyymmdd,hour,object_id) VALUES('192.168.1.3', 'localhost','20081009',8,'23123')");   
 }   
stmt.executeBatch();   
conn.commit();

3. :

conn.setAutoCommit(false);   
Statement stmt = conn.createStatement(ResultSet.TYPE_SCROLL_SENSITIVE,   
                                    ResultSet.CONCUR_READ_ONLY);   
for(int x = 0; x < size; x++){   
   stmt.execute("INSERT INTO adlogs(ip,website,yyyymmdd,hour,object_id) VALUES('192.168.1.3', 'localhost','20081009',8,'23123')");   
}   
conn.commit();

. 100 000 . : 1:17,844s 2: 18.421s 3: 16.359s

+9

MS JDBC 4.1 SQLServerBulkCopy, , , .NET, , bcp. https://msdn.microsoft.com/en-us/library/mt221490%28v=sql.110%29.aspx

+2

you can customize your code using JDBC, there is no frame support for your requirement

+1
source

More articles:


All Articles