Java / jetty does not respond to connections after a period of heavy load

We have a very difficult problem with our Java servlet running under Jetty. It works fine when the load is low and moderate, but when the load reaches a certain level, it will stop responding after 10-20 minutes.

If we follow the main java process when connecting to it using curl, we will see that it establishes a connection, receives a request, parses it, and does what it usually does (it requests the Solr server, executes some MySQL queries, etc. ), but the result is never sent back to the client.

When it freezes, it will hang endlessly. No amount of time will cause it to "break out of it," BUT if we kill any random thread under the main java / jetty process, the number of threads will decrease and will begin to respond to requests again.

Here's what it looks like:

# curl http://localhost:8080/some-servlet-url

(Does not respond at this point)

# ps -efL | grep qserv | wc -l
243
# ps -efL | grep qserv | wc -l
243
# ps -efL | grep qserv | wc -l
243

(Number of threads remain seemingly constant)

# kill 29760   <--- random thread under the main java/jetty process
# ps -efL | grep qserv | wc -l
26

(Number of threads immediately decreases sharply)

# curl http://localhost:8080/some-servlet-url
... HTTP response ...

(Responds to connections again)

The server is an m2.2xlarge instance of Amazon EC2 with Ubuntu 12.04.1 LTS and Jetty 8.1.7.v20120910

$ java -version
java version "1.7.0_07"
OpenJDK Runtime Environment (IcedTea7 2.3.2) (7u7-2.3.2-1ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

We tested several versions of java and Jetty. We also tried using Tomcat instead of Jetty - the same problem.

Profiling your application with YourKit does not show explicit thread blocking or excessive CPU usage.

Any ideas?

Edit: We were able to get the stack trace of a hanging java process, and it looks like all the HTTP streams are in this state:

"http-bio-8080-exec-5" daemon prio=10 tid=0x00007fe518007800 nid=0x1fc5 in Object.wait() [0x00007fe57934f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x000000076ee9d230> (a org.apache.commons.pool.impl.GenericObjectPool)
        at java.lang.Object.wait(Object.java:503)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810)
        - locked <0x000000076ee9d230> (a org.apache.commons.pool.impl.GenericObjectPool)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
        at net.acmecorp.active.QueryResultXMLFormatter.selectBestHitsAndRunDocumentCompletion(QueryResultXMLFormatter.java:362)
        at net.acmecorp.active.QueryResultXMLFormatter.queryResultToXMLRootElement(QueryResultXMLFormatter.java:167)
        at net.acmecorp.active.QueryPrepareAndExecuter.prepareParametersAndExecuteQuery_AndInvokeFormatter(QueryPrepareAndExecuter.java:239)
        at net.acmecorp.servlets.MultiQueryServlet.handle(MultiQueryServlet.java:470)
        at net.acmecorp.servlets.MultiQueryServlet.doGet(MultiQueryServlet.java:85)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1002)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        - locked <0x0000000700dd70d0> (a org.apache.tomcat.util.net.SocketWrapper)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
        - <0x000000076ed40990> (a java.util.concurrent.ThreadPoolExecutor$Worker)

java-, , , , , - ...