Download a CSV file from the Internet (with redirects) in python

Let me start with the fact that I know that there are several topics that discuss issues like mine, but the proposed solutions for some reason do not work for me for some reason. Also, I am new to downloading files from the Internet using scripts. So far I have mainly used python as a replacement for Matlab (using numpy / scipy). Therefore, if I make stupid mistakes, please be with me.

My goal: I want to download many .csv files from an internet database (http://dna.korea.ac.kr/vhot/) automatically using python. I want to do this because it is too cumbersome to download the 1000+ CSV files that I need manually. Access to the database can only be obtained using the user interface, where you need to select several options from the drop-down menu, so that finally, with links to CSV files, after some steps. I found out that the URL that you get after filling out the drop-down menus and clicking "search" contains all the options for the drop-down menu. This means that I can just change them and not use the drop-down menu, which helps a lot.

An example url from this site (let's call it url1 ): url1 = http://dna.korea.ac.kr/vhot/search.php?species=Human&selector=drop&mirname=&mirname_drop=hbv-miR-B2RC&pita=on&set=and&miranda_th=-- 5 & ​​rh_th = -10 & ts_th = 0 & mt_th = 7.3 & pt_th = 99999 & gene =

On this page I can select 5 csv files, one example directs me to the following URL:

URL2 = http://dna.korea.ac.kr/vhot/download.php?mirname=hbv-miR-B2RC&species_filter=species_id+%3D+9606&set=and&gene_filter=&method=pita&m_th=-5&rh_th=-10&ts_th=0&mt_th=7.3&pt_th = 99999 & targetscan = & miranda = & rnahybrid = & microt = & pita = on

csv , , "" ( , googeling, , ).

. , , url1 , url2 ( , . Url2 , . url1 ...). url1 url2, " " csv . - , ?

, csv python. urllib, urllib2 request, . , , Requests , .

- ( ):

stackoverflow.com/questions/7603044/how-to-download-a-file-returned-indirectly-from-html-form-submission-pyt

stackoverflow.com/questions/9419162/python-download-returned-zip-file-from-url

techniqal.com/blog/2008/07/31/python-file-read-write-with-urllib2/

, , :

import urllib2
import csv
import sys

url = 'http://dna.korea.ac.kr/vhot/download.php?mirname=hbv-miR-B2RC&species_filter=species_id+%3D+9606&set=or&gene_filter=&method=targetscan&m_th=-5&rh_th=-10&ts_th=0&mt_th=7.3&pt_th=-10&targetscan=on&miranda=&rnahybrid=&microt=&pita='

#1
u = urllib2.urlopen(url)
localFile = open('file.csv', 'w')
localFile.write(u.read())
localFile.close()

#2
req = urllib2.Request(url)
res = urllib2.urlopen(req)
finalurl = res.geturl()
pass
# finalurl = 'http://dna.korea.ac.kr/vhot/download.php?mirname=hbv-miR-B2RC&species_filter=species_id+%3D+9606&set=or&gene_filter=&method=targetscan&m_th=-5&rh_th=-10&ts_th=0&mt_th=7.3&pt_th=-10&targetscan=on&miranda=&rnahybrid=&microt=&pita='

#3
import requests
r = requests.get(url)
r.content
pass
#r.content =  "< s c r i p t > location.replace('download_send.php?name=qgN9Th&type=targetscan'); < / s c r i p t >"

#4
import requests
r = requests.get(url, 
allow_redirects=True,
data={'download_open': 'Download', 'format_open': '.csv'})
print r.content
# r.content = "

#5
import urllib
test1 = urllib.urlretrieve(url, "test.csv")
test2 = urllib.urlopen(url)
pass

# 2, # 3 # 4 . # 1 # 5 CSV </script>'

№3 , , ?

- ?