jeudi 8 octobre 2015

ValueError: Cannot convert identifier to UTF-8: 'Id'

I have a large Pandas dataframe (over 2 millions rows), with the following columns:

Id,CandidateRegistrationID,CandidateID,OurReference,QualificationCode,ExamCode,ExamDate,QualificationName,DataSource,QuestionNo,CandidateResponse,CorrectAnswerChoice,UniquePaperNo,QuestionCode

I have a function to write the dataframe to sqlite:

def writeDF(df,db,table):
    conn = sqlite3.connect(db)
    conn.text_factory = str  # allows utf-8 data to be stored

    df.to_sql(table, conn, flavor='sqlite', schema=None, if_exists='replace', index=False, index_label=None, chunksize=None, dtype=None)

    conn.close()

On cut-down versions of the data this works fine. On the full data-set I get the following error:

ValueError: Cannot convert identifier to UTF-8: 'Id'

The Id field is just an integer.

I'd welcome any insights. Googling just takes me to the line in Pandas that makes the error.

Traceback (most recent call last):
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
  return self.wsgi_app(environ, start_response)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
  response = self.make_response(self.handle_exception(e))
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
  reraise(exc_type, exc_value, tb)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
  response = self.full_dispatch_request()
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
  rv = self.handle_user_exception(e)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
  reraise(exc_type, exc_value, tb)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
  rv = self.dispatch_request()
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
  return self.view_functions[rule.endpoint](**req.view_args)
File "/py-csv-jmetrik/app/routes.py", line 69, in index
  writeDF(data_df,db,table)
File "/py-csv-jmetrik/app/routes.py", line 27, in writeDF
  df.to_sql(table, conn, flavor='sqlite', schema=None, if_exists='replace', index=True, index_label=None, chunksize=None, dtype=None)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/core/generic.py", line 982, in to_sql
  dtype=dtype)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 549, in to_sql
  chunksize=chunksize, dtype=dtype)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 1565, in to_sql
  dtype=dtype)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 627, in __init__
  self.table = self._create_table_setup()
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 1377, in _create_table_setup
  for cname, ctype, _ in column_names_and_types]
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 1297, in _get_valid_sqlite_name
  uname = _get_unicode_name(name)
File "/py-csv-jmetrik/venv/lib/python2.7/site-packages/pandas/io/sql.py", line 1271, in _get_unicode_name
  raise ValueError("Cannot convert identifier to UTF-8: '%s'" % name)

Aucun commentaire:

Enregistrer un commentaire