report the error as well as the line that caused it#258
report the error as well as the line that caused it#258abridgett wants to merge 8 commits intodatabricks:masterfrom
Conversation
There was a problem hiding this comment.
Please check travis. [error] /home/travis/build/databricks/spark-csv/src/main/scala/com/databricks/spark/csv/CsvRelation.scala:176: File line length exceeds 100 characters
|
For me I think this is reasonable because I sometimes see some users have some difficulties to find out why it was dropped or malformed. |
Current coverage is
|
|
Thanks Hyukjin, I've fixed that overlong line |
There was a problem hiding this comment.
Personal opinion: since this is an warning message, shouldn't we maybe just print out the message only by e.getMessage() excluding the class name of the exception?
There was a problem hiding this comment.
In addition, you can write s"Number format exception (${a.getMessage})." instead of concatenating them with + which is not used generally in this library and Spark codes as well.
There was a problem hiding this comment.
This line exceeds 100 characters as well.
I don't know Scala (at all!) so there's almost certainly cleaner ways - my apologies. The logging at the moment is sometimes unhelpful as it's hard to see the real issue - with DROPMALFORMED you see the line, with another parsing mode you get the error but not the line.