jeudi 19 février 2015

Properly Handling Large Amounts of Data

I've ventured into extremely unknown territory and I need some advice. I'm building an IRC bot for Twitch that could potentially contain hundreds of thousands of accounts. Previously I used Dictionary(of String, Integer) to store these accounts and their balance in memory, and just wrote them to a text file and saved it to disk. Now I'm storing a lot more data like their online time, whether they are a follower, or subscriber, how much they've donated. So I've started utilizing SQLite to store the accounts and I believe I need to utilize DataGridView with virtualmode to display these accounts and allow the user to edit any data. Am I on the right track so far? My aim is to keep it simple while managing larges amount of data, hopefully that wasn't an oxymoron.


Now I imagine the accounts in the SQLite database will need to be cached so that it's not constantly reading the database from the hard drive when a change needs to be made or when the user is scrolling through his list of accounts in the datagridview(which is in virtualmode). Am I right so far?


If there is another, more simpler method I could utilize please let me know, otherwise can someone help comprehend how I could cache these accounts? I did consider writing them to a Dictionary(of String, String) and then splitting the string when I needed to work with a specific value. Anyways I await your suggestions or comments. Thank you.


Aucun commentaire:

Enregistrer un commentaire