dimanche 18 octobre 2015

Modifying database from Pool does not work (but no Exceptions)

I've created a method, which downloads a product from one web page and then, this product is stored into the database SQLite3.

This function works good when it's called normally but I want to make a pool and do it parallel (because of sending parallel requests) (the web page allows bots to send 2000 requests/ minute).

The problem is that when I try to put it into the pool it does not store the data into the database nor raises some error or exception.

Here is the code of main function:

if __name__ == '__main__':
    pu = product_updater() # class which handles almost all, this class also has database manager class as an attribute
    pool = Pool(10) 

    for line in lines[0:100]: # list lines is a list of urls
        # pu.update_product(line[:-1]) # this works correctly 
        pool.apply_async(pu.update_product, args=(line[:-1],)) # this works correctly but does not store products into the database

    pool.close()
    pool.join()

def update_product(self,url): # This method belongs to product_updater class

    prod = self.parse_product(url)
    self.man.insert_product(prod) # man is a class to handling database

I use this pool: from multiprocessing.pool import ThreadPool as Pool

Do you know what could be wrong?

EDIT: I think that it could be caused because there is only one cursor which is shared between workers but I think that if this would be a problem it would raises some Exception.

EDIT2: The weird is that I tried to make Pool which has only 1 worker so there shouldn't be a problem with concurrency but the same result - no new rows in a database.

Aucun commentaire:

Enregistrer un commentaire