mercredi 15 juillet 2015

How to make sqlite in-memory db join query as fast as MySQL

I have a very complex sql query - the logic is simple, but I need to join 17 tables so there're a lot of (LEFT) JOINs and WHERE clauses.

SELECT table1.column_A
       table2.column_B
       table3.column_C
       table4.column_D
       ....
FROM table1
LEFT JOIN table2 ON table1.column_a = table2.column_b
JOIN table3 ON table3.column_c = table1.column_d
LEFT JOIN table4.column_e = table3.column_f
    AND LENGTH(table4.column_g) > 6 AND (table4.column_h IN (123,234))
LEFT JOIN ....
....
WHERE table1.column_i = 21 
    AND (table1.column_j IS NULL OR DATE(table1.column_k) <> DATE(table1.column_l))

The above query only takes 5 seconds to run in MySQL. But when I run it in sqlite in-memory db (using Perl on Linux), it takes about 20 min. This is still acceptable.

When I add a ORDER BY clause (I do need this), the execution time increases dramatically.

ORDER BY table1.column_m, table6.column_n, table7.column_o IS NULL;

It'll take 40 seconds in MySQL. In sqlite in-memory db (using Perl on Linux), I waited for over an hour, but it still didn't finish.

What kind of tuning do I need to do to make the query faster? My threshold is within 1 hour.

The reason I'm making it an in-memory db is that I receive SQL generated normalized data but we need to load data into a non-SQL db in the end, so I don't want to create a intermediate SQL db just for data loading - that makes the code ugly and increases maintenance complexity. Plus, the current timing issue I'm facing is just a one-time thing. In the future on a daily basis, the data volume we receive will be much much smaller (less than 1% of what I have today)

Thanks in advance for your help!!

Aucun commentaire:

Enregistrer un commentaire