ResultSet Performance Optimization (Apache Spring, MySQL)

My problem is this: I am trying to process about 1.5 million rows of data in Spring through a JDBCTemplate coming from MySQL. With so many rows, I use the RowCallbackHandler class as suggested here

The code really works, but SLOW ... The fact is that no matter what I set the sample size for, I seem to get about 350 records per sample, with a delay of 2 to 3 seconds between samples (from observing my logs ) I tried to comment on the repository command and confirmed that the behavior remained unchanged, so the issue is not related to the record.

There are 6 columns, only 1, which is varchar, and that it has a length of only 25 characters, so I don't see a bandwidth problem.

Ideally, I would like to get more than 30000-50000 rows at a time. Is there any way to do this?

Here is my code:


    protected void runCallback(String query, Map params, int fetchSize, RowCallbackHandler rch) 
            throws DatabaseException {
        int oldFetchSize = getJdbcTemplate().getFetchSize();
        if (fetchSize > 0) {
            getJdbcTemplate().setFetchSize(fetchSize);
        }
        try {
            getJdbcTemplate().query(getSql(query), rch);
        }
        catch (DataAccessException ex) {
            logger.error(ExceptionUtils.getStackTrace(ex));
            throw new DatabaseException( ex.getMessage() );         
        }
        getJdbcTemplate().setFetchSize(oldFetchSize);
    }

and the handler:

public class SaveUserFolderStatesCallback implements RowCallbackHandler {
        @Override
        public void processRow(ResultSet rs) throws SQLException {
            //Save each row sequentially.
            //Do NOT call ResultSet.next() !!!!

            Calendar asOf = Calendar.getInstance();
            log.info("AS OF DATE: " + asOf.getTime());
            Long x = (Long) rs.getLong("x");
            Long xx = (Long) rs.getLong("xx");
            String xxx = (String) rs.getString("xxx");
            BigDecimal xxxx = (BigDecimal)rs.getBigDecimal("xxxx");
            Double xxxx = (budgetAmountBD == null) ? 0.0 : budgetAmountBD.doubleValue();
            BigDecimal xxxxx = (BigDecimal)rs.getBigDecimal("xxxxx");
            Double xxxxx = (actualAmountBD == null) ? 0.0 : actualAmountBD.doubleValue();           

            dbstore(x, xx, xxx, xxxx, xxxxx, asOf);
        }       

    }
+3
source share
3 answers

The answer actually is to do setFetchSize (Integer.MIN_VALUE), while it completely violates the stated Statement.setFetchSize contract, the java mysql connector uses this value for the result set thread. This leads to huge performance improvements.

, (Spring) JdbcTemplate, ... , , fetchSize (Integer.MIN_VALUE)

http://javasplitter.blogspot.com/2009/10/pimp-ma-jdbc-resultset.html

!

0

? indexex , /. .

: . .

.

0

  • , DB. ASYNC_NETWORK_IO .

  • , Spring

0

All Articles