I am trying to work around the 999 variable limit inherent in SQLite when wrapped by ORMs such as peewee
. I am trying to construct a few dozen tables with ~50k rows and ~20 columns each. However, due to the 999 limit, I have to limit my inserts to ~50 rows per insert statement. This is extremely slow.
How can I make this faster? If I do not have a primary key constraint, then this requirement goes away, as I can just use pandas
to dump directly to SQL, but then modifying later to have a primary key is a pain.
Here is an example:
from peewee import *
database = SqliteDatabase(None)
class Base(Model):
class Meta:
database = database
colnames = ["A", "B", "C", "D", "E", "F", "G", "H"]
cols = {x: TextField() for x in colnames}
table = type('mytable', (Base,), cols)
database.init('test.db')
database.create_tables([table])
data = []
for x in range(150): # if this number is any higher this crashes
data.append({x: 1 for x in colnames})
with database.atomic() as txn:
table.insert_many(data).execute()
How can I get around this limitation? In the peewee
documentation, they mention using apsw
, which has the ability to modify the SQLite max_variables variable, but I am worried about the effects of increasing this variable to some huge number.
Aucun commentaire:
Enregistrer un commentaire