mercredi 2 décembre 2015

Apache Spark SQL identifier expected exception

My question is quite similar to this one: Apache Spark SQL issue : java.lang.RuntimeException: [1.517] failure: identifier expected But I just can't figure out where my problem lies. I am using SQLite as database backend. Connecting and simple select statements work fine.

The offending line:

val df = tableData.selectExpr(tablesMap(t).toSeq:_*).map(r => myMapFunc(r))

tablesMap contains the table name as key and an array of strings as expressions. Printed, the array looks like this:

WrappedArray([My Col A], [ColB] || [Col C] AS ColB)

The table name is also included in square brackets since it contains spaces. The exception I get:

Exception in thread "main" java.lang.RuntimeException: [1.1] failure: identifier expected

I already made sure not to use any Spark Sql keywords. In my opinion there are 2 possible reasons why this code fails: 1) I somehow handle spaces in column names wrong. 2) I handle concatenation wrong.

Aucun commentaire:

Enregistrer un commentaire