I am launching an online store that is experiencing a very sharp movement. Recently, the whole system came to a standstill during a big sale event, because the payment gateway had some kind of problem, and the responses from their API took 17 seconds to respond instead of the usual 2-3 seconds. Hundreds of people tried to shop at the same time. As a result, all the web worker threads on each web server in the production cluster were tied to waiting for API responses from the payment gateway. The entire production cluster was blocked and could not serve a single page.
One solution to this problem would be to use Resque to process payments in the background. Web servers will immediately return a response to the user, who would say something like: "Your payment is being processed ..." The web server will be able to proceed to the next web request.
The problem is to continue verification after processing the payment. I cannot use AJAX to query whether a transaction has completed because many clients do not have JavaScript. I can not depend on it. I do not want to use JavaScript technology or any type of server technology with a low degree of reliability, because this would create an expensive customer support problem if the percentage of checks stopped at the "processing ..." page.
How can I process a page stream so that transactions can be processed in the background, reliably?