How to save application state for node.js application, which consists mainly of HTTP request?
I have a script in node.JS that works with a RESTful API to import a large number (10,000+) of products into an e-commerce application. The API has a limit on the number of requests that can be made, and we are trying to expand this limit. In a previous run, the script exited with Error: connect ETIMEDOUT, probably due to exceeding the API limits. I would like to be able to try to connect 5 times, and if this does not resume after an hour, when the limit is restored.
It would also be useful to maintain progress in the event of a failure (power drops, network outages, etc.). And be able to resume the script from the moment it stopped.
I know that node.js works like a giant event queue, all HTTP requests and their callbacks fall into this queue (along with some other events). This makes it the main target for maintaining the state of the current execution. Other pleasant (not quite necessary for this project) could distribute the work between several machines on different networks to increase throughput.
So is there an existing way to do this? Perhaps a framework? Or I need to implement this on my own, in this case any useful resources on how to do this will be useful.
source
share