vendredi 30 janvier 2015

Building a distributed bittorrent-SQL database

I have an idea for a distributed SQL database using the bittorrent protocol for pulling and writing its data.


For the sake of argument, lets say this is a messaging application, where thousands of users run a program that contains a messaging window, and an input box for them to write messages.


Each message written does a INSERT to their own sqlite DB.


How it could be done



  • Download a .torrent file that essentially contains the schema/DDL for creating the DB, and create it on the local machines.

  • Anytime a 'write' action is done(like a user wants to send a message), that INSERT line(which is kinda like a delta) does two things:

    • Writes to their own internal DB

    • Creates a .torrent file out of that line, named something like, messaging-[my-ip]-[UTC_timestamp].torrent, and posts it to a tracker



  • Everyone running the app is continually scanning the tracker for files of this certain name(and possibly after a certain date), downloads the .torrent and hosts it, and runs the INSERT commands on their local DB.


What you'd then have is a ton of delta-files, all P2P hosted for redundancy, updating local .sqlite DBs on a lot of machines.


Some issues I'm having




  • How do I scrape for torrents of a certain file-name? I've read through the http bittorrent tracker spec, but you seem to only be able to query files based on their specific info name. Is there no way to query for a group of files, or based on file name?




  • How do I download a .torrent file from a tracker? Will I need to host the files on a centralized server, or can I use the tracker to download the files in some way? And if I have to host the .torrent files myself...



    • Wouldn't this defeat the purpose of a decentralized DB, since if my website goes down, the application would stop getting updates?




Thanks for the help in advance.


Aucun commentaire:

Enregistrer un commentaire