lundi 9 mars 2015

Fast data moving from CSV to SQLite by Python

I have a problem. There are hundreds of CSV files, ca. 1,000,000 lines each one. I need to move that data in a specific way, but script working very slow (it passing few ten of tousands per hour).


My code:



import sqlite3 as lite
import csv
import os



my_file = open('file.csv', 'r')
reader = csv.reader(my_file, delimiter=',')


date = '2014-09-29'

con = lite.connect('test.db', isolation_level = 'exclusive')
for row in reader:

position = row[0]
item_name = row[1]



cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS [%s] (Date TEXT, Position INT)" % item_name)
cur.execute("INSERT INTO [%s] VALUES(?, ?)" % item_name, (date, position))
con.commit()


I found an information saying about isolation_level and single accessing to database, but it didn't work well.


Lines CSV files have a structure: 1,item1 | 2,item2


Does anyone could to help me? Thanks!


Aucun commentaire:

Enregistrer un commentaire