dimanche 8 février 2015

log parsing and sqlite peformance

When parsing huge log files into a sqlite database is there any major performance improve if we don't call connection.commit() after each insert or does that not make any difference. I guess the question is if it's only for separating transactions or if it does anything that needs time.


Also are foreign key constraints checked and can we turn those checks off to improve performance? how?


And for parsing the lines in the log file we currently use this parse module. But it seems quite slow. Is there any other awesome fast parsing module for this purpose?


Aucun commentaire:

Enregistrer un commentaire