I'm working on a task manager for appengine, and the default scheduler always finishes launching 3-4 instances that do all the work, some overflow instances that can take thousands of tasks, or just a couple, and then sitting there the burning processor does nothing.
My task includes processing tasks for different domains of different sizes, sometimes a huge bandwidth, and in other cases - one user with an update of 10,000 models; if I turn off the normal appengine task scheduler, it will fail in two ways: 1) the backends never close, and when the memory gets into the cap, java gc makes a trash instance and acts like almost a zombie, but never closes {and still occupying / holding tasks} and 2) many domains have one user who takes much more time than all the others to process, and this allows the backend to be supported long before the rest of the domain is completed.
These tasks must be completed during the day, and several backends are required to process the branches, so I cannot just reset them all to B8 and call it day. Therefore, we need a dispatcher to control the distribution of tasks to the backends.
Now I don’t want to pay for the datastore operators for each task in order to save several minutes of processor time, so my plan of attack {please criticize} is to use the static ConcurrentHashMap in RAM, start each run () in an attempt, each delayed task put it [hashcode , startTime] during startup and deleted (hashcode) at the end. There will be one such map to the internal instance, which runs the tasks wrapped in the method, BackendCounter.addToLiveMap (this); it .size () serves as a total of how many jobs live on this backend {with a timestamp to detect zombie jobs that work> 10 minutes}. The task manager can disable the workflow per instance to keep track of how many jobs, excluding itself, are running in that instance,and save the ranked list in memcache, from which the instances have how many tasks are expected. If one instance falls below the X threshold of live tasks, select an overflow instance to delay it until then, use the BackendCounter.addToLiveMap method (this) to raise an exception, which I can catch, to specify jobs, just to assign myself to a new one instance of {ChangeInstanceException # getNewTarget ()}. Thus, I can prevent unsuitable instances from receiving new jobs so that they can disconnect by paying only for some memcache operations, and the splitter only pays for writing and deleting to a static card.to defer it until then, use the BackendCounter.addToLiveMap method (this) to raise an exception that I can catch, to specify jobs, just to assign myself to a new instance of {ChangeInstanceException # getNewTarget ()}. Thus, I can prevent instances of little use from receiving new jobs so that they can disconnect by paying only for some memcache operations, and the splitter only pays for writing and deleting to a static card.to defer it until then, use the BackendCounter.addToLiveMap method (this) to raise an exception that I can catch, to specify jobs, just to assign myself to a new instance of {ChangeInstanceException # getNewTarget ()}. Thus, I can prevent instances of little use from receiving new jobs so that they can disconnect by paying only for some memcache operations, and the splitter only pays for writing and deleting to a static card.so that they have the opportunity to disconnect by paying only for certain memcache operations, and the splitter only pays for writing and deleting to a static card.so that they have the opportunity to disconnect, paying only for some memcache operations, and the splitter only pays for writing and deleting to a static card.
, -. , , , ( 0 1) , .
, BackendCounter.addToLiveMap(this), ChangeInstanceException :
if (((float) Runtime.getRuntime(). freeMemory()/Runtime.getRuntime(). totalMemory()) < 0.9) throw new ChangeInstanceException (getOverflowInstance());
, , .
, 0 1 { , , }, 2+, , 10 15 . , 2, 3 , , 4, , .
, , , , , , , throwInstanceException.
.