I have a very complex sql query - the logic is simple, but I need to join 17 tables so there're a lot of (LEFT) JOINs and WHERE clauses.
SELECT table1.column_A
table2.column_B
table3.column_C
table4.column_D
....
FROM table1
LEFT JOIN table2 ON table1.column_a = table2.column_b
JOIN table3 ON table3.column_c = table1.column_d
LEFT JOIN table4.column_e = table3.column_f
AND LENGTH(table4.column_g) > 6 AND (table4.column_h IN (123,234))
LEFT JOIN ....
....
WHERE table1.column_i = 21
AND (table1.column_j IS NULL OR DATE(table1.column_k) <> DATE(table1.column_l))
The above query only takes 5 seconds to run in MySQL. But when I run it in sqlite in-memory db (using Perl on Linux), it takes about 20 min. This is still acceptable.
When I add a ORDER BY clause (I do need this), the execution time increases dramatically.
ORDER BY table1.column_m, table6.column_n, table7.column_o IS NULL;
It'll take 40 seconds in MySQL. In sqlite in-memory db (using Perl on Linux), I waited for over an hour, but it still didn't finish.
What kind of tuning do I need to do to make the query faster? My threshold is within 1 hour.
The reason I'm making it an in-memory db is that I receive SQL generated normalized data but we need to load data into a non-SQL db in the end, so I don't want to create a intermediate SQL db just for data loading - that makes the code ugly and increases maintenance complexity. Plus, the current timing issue I'm facing is just a one-time thing. In the future on a daily basis, the data volume we receive will be much much smaller (less than 1% of what I have today)
Thanks in advance for your help!!
Aucun commentaire:
Enregistrer un commentaire