Geek asks and answers

Getting a lot of data from a very slow external data source

I need to get a large amount of data from an external source. The problem is that the external source sends data very slowly. The workflow is as follows:

  • A user initiates a process from the application interface (usually it retrieves data from a local XML file). This is a pretty quick process.
  • After that, we need to download the information related to the extracted data from an external source (mainly these are external statistics for data from xml). And it is very slow. But the user needs this additional information to continue working. For example, it can perform filtering according to external data or something else.

So we need to do this asynchronously. The main idea is to show external data as it appears. The question is, how can we organize this asynchronous process? Maybe some kind of problem or something else? We use php + mysql as the backend and jquery in the interface. Many thanks!

+3
source share
2 answers

Your two possible strategies:

  • Stream to the backend using a PHP script that wraps a large external resource into a database or memcache and responds to period requests for new data, clearing their db line or cache in response.

  • , JavaScript , . Gecko WebKit XmlHttpRequest.onreadystatechange , JavaScript. IE iframe, Ajax Patterns, SO-.

+1

cURL system(), . , PHP , . PHP system():

, , . PHP .

. , :

  • iFrame , (, , - ),
  • jQuery AJAX ,
  • CGI script, , DB ,
  • ...
0

More articles:


All Articles