My problem is this: I am trying to process about 1.5 million rows of data in Spring through a JDBCTemplate coming from MySQL. With so many rows, I use the RowCallbackHandler class as suggested here
The code really works, but SLOW ... The fact is that no matter what I set the sample size for, I seem to get about 350 records per sample, with a delay of 2 to 3 seconds between samples (from observing my logs ) I tried to comment on the repository command and confirmed that the behavior remained unchanged, so the issue is not related to the record.
There are 6 columns, only 1, which is varchar, and that it has a length of only 25 characters, so I don't see a bandwidth problem.
Ideally, I would like to get more than 30000-50000 rows at a time. Is there any way to do this?
Here is my code:
protected void runCallback(String query, Map params, int fetchSize, RowCallbackHandler rch)
throws DatabaseException {
int oldFetchSize = getJdbcTemplate().getFetchSize();
if (fetchSize > 0) {
getJdbcTemplate().setFetchSize(fetchSize);
}
try {
getJdbcTemplate().query(getSql(query), rch);
}
catch (DataAccessException ex) {
logger.error(ExceptionUtils.getStackTrace(ex));
throw new DatabaseException( ex.getMessage() );
}
getJdbcTemplate().setFetchSize(oldFetchSize);
}
and the handler:
public class SaveUserFolderStatesCallback implements RowCallbackHandler {
@Override
public void processRow(ResultSet rs) throws SQLException {
Calendar asOf = Calendar.getInstance();
log.info("AS OF DATE: " + asOf.getTime());
Long x = (Long) rs.getLong("x");
Long xx = (Long) rs.getLong("xx");
String xxx = (String) rs.getString("xxx");
BigDecimal xxxx = (BigDecimal)rs.getBigDecimal("xxxx");
Double xxxx = (budgetAmountBD == null) ? 0.0 : budgetAmountBD.doubleValue();
BigDecimal xxxxx = (BigDecimal)rs.getBigDecimal("xxxxx");
Double xxxxx = (actualAmountBD == null) ? 0.0 : actualAmountBD.doubleValue();
dbstore(x, xx, xxx, xxxx, xxxxx, asOf);
}
}
source
share