dimanche 21 décembre 2014

Storing millions of record as in-memory in c# application [on hold]

I am developing a graphical analysis application in c# and wpf, for that I have a written custom parser to process csv and txt files and I'll keep those processed data in-memory to use it later.


Right now I am using Data Table to keep all the data in-memory, However I have serious concern about memory usage of Data Table, a input data of 150 MB will become 1 GB after storing it into Data Table.


Obviously I cannot mange my data with Data Table, so I have created my own custom Data Table class and filled the data to it. But I don't find any significant improvement in a memory usage. Consider all my data will be in a string.


I have tried Object data base like velocity db and perst, both were some or other reason I cannot use it in my application.


So, lastly I thought I can keep the process data in Sqlite database. However I suspect using SQL query will definitely slow down the application performance.


Give me some insight to handled the millions of record in an c# application.


Aucun commentaire:

Enregistrer un commentaire