Python Write / Read Big Data?

I got at least 1 000 000 000 of lines of text to process (i do time series analysis). These lines are distributed into 45 000 textes files. The files correspond to forcast(F) / real(R) datas for items (I have approx 3500 item) . We have multi F files for 1 R file.

Type of datas (4 columns) per text files : text, date, float, test

• The 1 process consist on mapping real vs forcast datas for each for each item and append the result into corresponding item textfile (each item file have approx 150 000/200 000 line of text with 10 columns)

  • The 2 process consist of read items files and process statistical analysis

My question : I would like to store the results of the 1 process into a container (sqlite or hdf or...) in order to append new datas, and to speed my queries (i use matplotlib) for the second process.

info : I have a hp z600 with 16 go ram and 2x4 cores

My first write/read performance test are (w: write; r=read) with 1 000 000 of data (random distributed on 5 columns

• sql-w : 13.612 secondes
• sql-r : 18.184
• hdf-fixe-w : 18.185
• hdf-fixe-r : 18.185
• hdf-fixe-compress-w : 18.185
• hdf-fixe-compress-r : 18.185
• hdf-table-w : 18.185
• hdf-table-r : 18.185
• hdf-table-compress-w : 18.185
• hdf-table-compress-r : 18.185
• csv-w : 18.185
• csv-r : 18.185

I willing to store my datas into sqlite or mysql (1 table for each item in order to speed the writing process ?!) on a regular basic (twice a week) I willing to do my reporting 1 time/month (I would like to concatenate all database for my statistical analysis) and export result in pdf or with QT4

Do you have any recommandations to give me ?

Thanks in advance