dimanche 3 avril 2016

Python/SQLite iterate through rows w/ BLOB data

all! First time posting here, long-time reader/follower.

Anyway, I'm trying to iterate through many rows and conduct a rather intensive comparison of file hex data. This data is stored in a BLOB field in my SQLite table. I wanted to use BLOB because the larger files get, the more hex data needs to be saved in the column.

I can iterate through the rows just fine, but what I'm worried about is loading too much data into memory. If I have a multi-GB sized file that I had run the hex for previously, that data is a bit too much to put into memory.

Right now I'm using:

conn = sqlite3.connect('test.sqlite')
c = conn.cursor()
conn.text_factory = str 

rows = c.execute("SELECT * FROM HashValues")

for row in rows:
*Grab 4th column data - haven't written this yet*
*Conduct comparison with preloaded hex data*
c.execute("INSERT INTO HashValues VALUES (?, ?, ?, ?, ?)", (ProcessDate, theFile, MD5Hex, SHA1Hex, rd))
conn.close()

What's the best method to do this, and how would I go about it? Thanks in advance!

Aucun commentaire:

Enregistrer un commentaire