I'm very desperate because I tried everything I could to make it work but no success... I've created a pool which is processing some text lines and then inserting them into the database. The problem is that it does not insert into the table nothing.
I know that it is probably because multiple workers trying to modify the database but I've already tried many things to avoid that.
For example this (this is the method for inserting url into the index table):
def insert_into_index(self, url):
self.lock.acquire()
self.cur.execute('INSERT INTO index VALUES(?)', (url,))
self.conn.commit()
self.lock.release()
Or this (each new use of the method has it's own cursor):
def insert_into_index(self, url):
cur = self.conn.cursor()
cur.execute('INSERT INTO index VALUES(?)', (url,))
self.conn.commit()
The snippet where I create the Pool:
""" CREATE A POOL """
pool = Pool(50)
for category in categories[1:2]:
task = pool.apply_async(index, args=(category.strip('\n'),))
task.get()
pool.close()
pool.join()
""" POOL END """
Index() is a method which creates an instance of class leaf(text)
, which do some things with the text, obtains list of strings
and try to insert
them into the database.
class leaf():
def __init__(self, url):
self.url = url
self.results = self.get_all_hrefs(url)
self.dbm = database_file.manager()
for res in self.results:
self.dbm.insert_into_index(res)
But I don't have nothing in table index. I've tried everything I knew about with no success. Could you give me an advice?
Aucun commentaire:
Enregistrer un commentaire