mardi 8 septembre 2015

Reading without errors from a SQLlite database while another process is writing to it (via R)

I'm new to SQL, and trying to do some reading of a database I'm building while I have another process writing to it. I'm strictly reading stuff that is not being written by the write process (in my real example).

When I try to read from the database, more often than not the write process stops and the database gets locked -- I need to kill R and restart it.

I sort of intuitively understand why this would be problematic. But there seems to be workarounds ("no lock"?) that I can't figure out how to implement.

MWE below: run the first process, and then go start another session and try to run the second process a few times:

1> library(dplyr)
1> library(RSQLite)
1> system("rm dummy.sqlite3")
1> db <- src_sqlite("dummy.sqlite3", create = T)
1> df = data.frame(x = rnorm(2),y = rnorm(2))
1> table = copy_to(db, df = df, temporary = FALSE)
1> #Write process
1> for (i in 1:1e6){
1+ x = data.frame(x = rnorm(1),y = rnorm(1))
1+ db_insert_into(con = db$con, table = 'df', values = x) 
1+ }

And then over in the other session...

3> db <- src_sqlite("dummy.sqlite3", create = F)
3> df = tbl(db,'df')
3> x1<-filter(df, x>3)
3> collect(x1)
Source: local data frame [2 x 2]

         x          y
     (dbl)      (dbl)
1 3.445299 -0.2531794
2 3.235710 -1.2147918
3> library(dplyr)
3> library(RSQLite)
3> setwd('/home/andrew/Dropbox/weirding_data')
3> db <- src_sqlite("dummy.sqlite3", create = F)
3> df = tbl(db,'df')
Error in sqliteFetch(res, n = n) : 
  rsqlite_query_fetch: failed first step: database is locked
3> x1<-filter(df, x>3)
3> collect(x1)
Source: local data frame [3 x 2]

         x          y
     (dbl)      (dbl)
1 3.445299 -0.2531794
2 3.235710 -1.2147918
3 3.457522  0.9358973
3> library(dplyr)
3> library(RSQLite)
3> setwd('/home/andrew/Dropbox/weirding_data')
3> db <- src_sqlite("dummy.sqlite3", create = F)
3> df = tbl(db,'df')
3> x1<-filter(df, x>3)
3> collect(x1)
Source: local data frame [5 x 2]

         x          y
     (dbl)      (dbl)
1 3.445299 -0.2531794
2 3.235710 -1.2147918
3 3.457522  0.9358973
4 3.265626 -0.7512677
5 3.052190 -0.1328862
3> library(dplyr)
3> library(RSQLite)
3> setwd('/home/andrew/Dropbox/weirding_data')
3> db <- src_sqlite("dummy.sqlite3", create = F)
3> df = tbl(db,'df')
Error in sqliteFetch(rs, n = -1) : 
  rsqlite_query_fetch: failed first step: database is locked
3> x1<-filter(df, x>3)
3> collect(x1)
Error in sqliteFetch(res, n = n) : 
  rsqlite_query_fetch: failed first step: database is locked

It works a couple of times before it fails.

So, how can I read from something that is being written to? And why isn't the failure deterministic? And how big of a hit will the write speed take if there are queries extracting the same amount of data as the write process is doing?

Aucun commentaire:

Enregistrer un commentaire