How to properly handle the interrupt waiting for logs?

I have started an import task that works very well, until a couple of days ago, when the number of objects increased sharply.

What will happen is that I got an excess of waiting for a lock wait. Then the application retries and an exception is thrown, as I call em.getTransaction (). Begin (); again.

To get rid of this problem, I changed innodb_lock_wait_timeout to 120 and reduced the party side to 50 objects.

I cannot figure out how to properly handle all this in code. I do not want all imports to fail due to blocking. How would you handle this? Do you have an example code? Maybe some other thoughts? Please go nuts!

My BatchPersister:

public class BatchPersister implements Persister {

    private final static Log log = getLog(BatchPersister.class);
    private WorkLogger workLog = WorkLogger.instance();

    private static final int BATCH_SIZE = 500;

    private int persistedObjects;
    private long startTime;
    private UpdateBatch batch;
    private String dataSource;


    public BatchPersister(String dataSource) {
        this.dataSource = dataSource;        
    }

    public void persist(Persistable obj) {

        persistedObjects++;
        logProgress(100);

        if (batch == null)
            batch = new UpdateBatch(BATCH_SIZE, dataSource);

        batch.add(obj);

        if (batch.isFull()) {
            batch.persist();
            batch = null;
        }
    }
}

Updatebatch

public class UpdateBatch {

    private final static Log log = LogFactory.getLog(UpdateBatch.class);
    private WorkLogger workLogger = WorkLogger.instance();

    private final Map<Object, Persistable> batch;
    private final EntityManager em;
    private int size;

    /**
     * Initializes the batch and specifies its size.
     */
    public UpdateBatch(int size, String dataSource) {
        this.size = size;
        batch = new LinkedHashMap<Object, Persistable>();
        em = EmFactory.getEm(dataSource);
    }    

    public void persist() {
        log.info("Persisting " + this);
        em.getTransaction().begin();    
        persistAllToDB();
        em.getTransaction().commit();

        WorkLog batchLog = new WorkLog(IMPORT_PERSIST, IN_PROGRESS);
        batchLog.setAffectedItems(batch.size());
        workLogger.log(batchLog);
        em.close();
   }

/**
  * Persists all data in this update batch
  */
    private void persistAllToDB() {
        for (Persistable persistable : batch.values())
            em.persist(persistable);
        }

        @Override
        public String toString() {
            final ArrayList<Persistable> values = new ArrayList<Persistable>(batch.values());
            Persistable first = values.get(0);
            Persistable last = values.get(values.size() - 1);
            return "UpdateBatch[" +
                first.getClass().getSimpleName() + "(" + first.getId() + ")" +
                " - " +
                last.getClass().getSimpleName() + "(" + last.getId() + ")" +
                "]";
         }
    }
}
+3
source share
2 answers

Solution 1. Do not use JPA, it is not designed to work with massive database operations. Since you have access to your DataSource and you manage transactions manually, nothing prevents you from using plain old SQL.

Solution 2. Perhaps there was a performance problem associated with the cache of the first level of the persistence context - each stored object is stored in this cache, when this cache becomes large, it can damage the performance (mainly memory)

, hibernate.jdbc.batch_size ( , Hibernate- JPA) 20 - , 20 .

-, 20 , .

private void persistAllToDB() {
    int counter = 0;
    for (Persistable persistable : batch.values())
        em.persist(persistable);
        counter++;
        if(counter % 20 == 0){
           em.flush();
           em.clear();
        }
    }
}

3. MySQL InnoDB engine [http://dev.mysql.com/doc/refman/5.1/en/insert-speed.html, http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html]. , .

, , - .

+1

Pitor . , " 2" Hibernate StatelessSession api .

, - , , , , , . , , , , . . , jdbc- Hibernate; .

0

All Articles