mercredi 20 janvier 2016

sqllite3 each function event loop

I'm trying to read a large number of records from a sqlite database (~2 million) and insert some of the information from each row into mongodb.

var db = new sqlite3.Database(db_path, sqlite3.OPEN_READONLY, function (err) {
    if (err) {
        console.log('Error opening SQLite Database')
        process.exit(1)
    }

    var url = 'mongodb://localhost:27017/test'
    MongoClient.connect(url, function (err, db) {
        if (err) {
            console.log('Error connecting to MongoDB')
            process.exit(1)
        }
        db.each(sql, rowParser)
    })
})

rowParser is a function that takes a row builds up a json document and inserts it into mongodb like so.

function create(doc) {
    var collection = db.collection('test')
    collection.insertOne(doc, function(err, result) {
        console.log('NEVER GETS CALLED')
    })
}

I think the problem is the each function runs, gets to the mongodb insert, places a new event on the queue, and execution of each continues with the next row, this keeps going adding new events to the queue which eventually fills up and the program crashes as it never gets a chance to reduce the events in the queue. Is this correct?

Is there a better way to achieve this or force execution of each to pause and let some of the insert callbacks get executed?

Aucun commentaire:

Enregistrer un commentaire