I've created a method, which downloads a product from one web page and then, this product is stored into the database SQLite3
.
This function works good when it's called normally but I want to make a pool and do it parallel (because of sending parallel requests) (the web page allows bots to send 2000 requests/ minute).
The problem is that when I try to put it into the pool it does not store the data into the database nor raises some error or exception
.
Here is the code of main function:
if __name__ == '__main__':
pu = product_updater() # class which handles almost all, this class also has database manager class as an attribute
pool = Pool(10)
for line in lines[0:100]: # list lines is a list of urls
# pu.update_product(line[:-1]) # this works correctly
pool.apply_async(pu.update_product, args=(line[:-1],)) # this works correctly but does not store products into the database
pool.close()
pool.join()
def update_product(self,url): # This method belongs to product_updater class
prod = self.parse_product(url)
self.man.insert_product(prod) # man is a class to handling database
I use this pool: from multiprocessing.pool import ThreadPool as Pool
Do you know what could be wrong?
EDIT: I think that it could be caused because there is only one cursor which is shared between workers but I think that if this would be a problem it would raises some Exception
.
EDIT2: The weird is that I tried to make Pool which has only 1 worker so there shouldn't be a problem with concurrency but the same result - no new rows in a database.
Aucun commentaire:
Enregistrer un commentaire