mardi 5 janvier 2016

python/sqlite 3 average time per insert from webscraper

I wrote a program that for every stock in russell 2000 (stored in a table) it pings yahoo finance and scrapes the historical info (open, hi, low, close) for the last 3 days then inserts it into a separate table. I dont know what limits yahoo has so today I only ran 1100 stocks and it took 1070 seconds so about 1 per second. Does that seem like a normal time for this type of program? For the whole list it would take almost a half hour which seems like a lot but I have no reference. Obviously professionals dont use yahoo finance and have a data service that can do it in like 1 second but just curious if anyone has experience with this...

Aucun commentaire:

Enregistrer un commentaire