Using Python, I try to create a new SQLite table in a database from a query result which is executed on two other tables in that database.
For the record, the query is
CREATE TABLE results AS
SELECT table_1.*, table_2.*
FROM table_1
LEFT JOIN table_2
ON table_1.ID_1 = table_2.ID_2
UNION ALL
SELECT table_1.*, table_2.*
FROM table_2
LEFT JOIN table_1
ON table_1.ID_1 = table_2.ID_2
WHERE table_1.ID_1 IS NULL
which is supposed to be a workaround for a FULL OUTER JOIN which is not directly available in SQLite (this method can be found on different SO threads on this topic).
However, this operation is slow on my tables with ~1 million rows each... so slow that I get the impression it's going to take hours when hitting refresh in the Explorer Window showing the database's file size while updating.
How can I get this faster? I already did a lot of research on this and most of the time, people are talking about using transactions because otherwise, each row will open up a new connection to the database or whatever... however, I could not find a working example on how to use this.
My two approaches so far which are way too slow:
Using Python's sqlite3 module:
# open sqlite database
conn = sqlite3.connect('Database.sqlite')
# get a cursor
cursor = conn.cursor()
# start query
cursor.execute("""
CREATE TABLE results AS
SELECT allShapesMerged.*, allDbfsMerged.*
FROM allShapesMerged
LEFT JOIN allDbfsMerged
ON allShapesMerged.ID = allDbfsMerged.NETWORK_ID
UNION ALL
SELECT allShapesMerged.*, allDbfsMerged.*
FROM allDbfsMerged
LEFT JOIN allShapesMerged
ON allShapesMerged.ID = allDbfsMerged.NETWORK_ID
WHERE allShapesMerged.ID IS NULL
;""")
Using sqalchemy:
from sqlalchemy import create_engine, event
from sqlite3 import dbapi2 as sqlite
# create database engine
engine = create_engine('sqlite:///Database.sqlite')
# open sqlite database
connection = engine.connect()
# query
with connection.begin() as trans:
connection.execute("BEGIN TRANSACTION;")
connection.execute("""
CREATE TABLE results AS
SELECT allShapesMerged.*, allDbfsMerged.*
FROM allShapesMerged
LEFT JOIN allDbfsMerged
ON allShapesMerged.ID = allDbfsMerged.NETWORK_ID
UNION ALL
SELECT allShapesMerged.*, allDbfsMerged.*
FROM allDbfsMerged
LEFT JOIN allShapesMerged
ON allShapesMerged.ID = allDbfsMerged.NETWORK_ID
WHERE allShapesMerged.ID IS NULL
;""")
trans.commit()
Aucun commentaire:
Enregistrer un commentaire