Repeating obvious deadlocks with c3p0, Oracle and Hibernate

We have repeated apparent deadlocks in our production system that we seem to be unable to understand. We were not able to track any correlation with the number of users on the Internet, and it seems that we are running out of available connections.

We have a Java EE application connecting to Oracle through Hibernate and c3p0. Our c3p0 configuration:

minPoolSize=10
maxPoolSize=300
initialPoolSize=30
acquireIncrement=10
maxIdleTime=1800
maxStatementsPerConnection=0
numHelperThreads=5

The obvious output of the lock log always looks something like this:

[com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-1) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@4c9f1b4d -- APPARENT DEADLOCK!!! Complete Status:
    Managed Threads: 5
    Active Threads: 5
    Active Tasks:
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7fe1ab86 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@38c42c01 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@572512c4 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4)
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42f32e8e (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@6b758ef8 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3)
    Pending Tasks:
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@72fd72e5
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5d82535d
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@172f2ea1
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1a9e57eb
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@20ee5a35

The actual traces of the thread stream stack are different, and I added a few examples below:

Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,jboss]
            java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
            java.lang.StringBuilder.<init>(StringBuilder.java:80)
            oracle.net.ns.Packet.<init>(Packet.java:513)
            oracle.net.ns.ConnectPacket.<init>(ConnectPacket.java:64)
            oracle.net.ns.NSProtocol.connect(NSProtocol.java:278)
            oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1042)
            oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:301)
            oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:531)
            oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:221)
            oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
            oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)

Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,jboss]
            oracle.jdbc.driver.T4CTTIoauthenticate.processRPA(T4CTTIoauthenticate.java:491)
            oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:295)
            oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:186)
            oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:390)
            oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:356)
            oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:531)
            oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:221)
            oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
            oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:1

Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,jboss]
            oracle.net.ns.NSProtocol.connect(NSProtocol.java:346)
            oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1042)
            oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:301)
            oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:531)
            oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:221)
            oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
            oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
            com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
            com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
            com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
            com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
            com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
            com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
            com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

Any suggestions on where we should conduct our investigation? Is it c3p0, our queries, our code, database?

+5
source share
1

Connection. , . , , , .

, c3p0-0.9.2.1, "" , . , c3p0, numHelperThreads, -, , , 3 5, . , , "", , . [ . !]

, , , ( APPARENT DEADLOCK, - 10 ). .

0.9.2 numHelperThreads. [ numHelperThreads, , ; , , IO, , , ], , .

!

p.s. , , , APPARENT DEADLOCK. , , APPARENT DEADLOCK , c3p0 . , IO.

+3

All Articles