:cast-fns stack traces hide which cast-fn caused error

[NoahTheDuke]

Set up

A csv file titled "example.csv" that contains:

first-column,second-column
X,1

A clojure file called example/csv.clj that contains:

(ns example.csv
  (:require [semantic-csv.core :as sc]))

(defn manually-cast-columns
  [col]
  (assoc col
         :first-column (sc/->int (:first-column col))
         :second-column (sc/->int (:second-column col))))

(defn process-with-cast-fns []
  (sc/slurp-csv "example.csv"
                :cast-fns {:first-column sc/->int
                           :second-column sc/->int}))

(defn process-manually []
  (->> (sc/slurp-csv "example.csv")
       (mapv manually-cast-columns)))

(defn -main
  [& args]
  (case (first args)
    "cast" (process-with-cast-fns)
    "man" (process-manually)))

Issue

If I have the malformed csv file above (an "X" in a column that is supposed to be a number) and call process-with-cast-fns, it will fail. The stack trace outputs a NullPointerException, and the line indicating where the failure happened is just the slurp-csv call.

Caused by: java.lang.NullPointerException
	at clojure.core$partial$fn__5561.invoke(core.clj:2616)
	at clojure.lang.AFn.applyToHelper(AFn.java:154)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at clojure.core$apply.invokeStatic(core.clj:659)
	at clojure.core$update_in$up__6562.invoke(core.clj:6105)
	at clojure.core$update_in.invokeStatic(core.clj:6106)
	at clojure.core$update_in.doInvoke(core.clj:6092)
	at clojure.lang.RestFn.invoke(RestFn.java:445)
	at semantic_csv.impl.core$row_val_caster$fn__250.invoke(core.cljc:43)
	at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
	at clojure.core.protocols$fn__7841.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__7841.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6748)
	at clojure.core$reduce.invoke(core.clj:6730)
	at semantic_csv.impl.core$cast_row.invokeStatic(core.cljc:62)
	at semantic_csv.impl.core$cast_row.doInvoke(core.cljc:46)
	at clojure.lang.RestFn.invoke(RestFn.java:521)
	at semantic_csv.transducers$cast_with$fn__333$fn__334.invoke(transducers.cljc:167)
	at semantic_csv.transducers$mappify$fn__313$fn__314.invoke(transducers.cljc:42)
	at clojure.core$filter$fn__5610$fn__5611.invoke(core.clj:2798)
	at clojure.lang.TransformerIterator.step(TransformerIterator.java:79)
	at clojure.lang.TransformerIterator.hasNext(TransformerIterator.java:97)
	at clojure.lang.RT.chunkIteratorSeq(RT.java:510)
	at clojure.core$sequence.invokeStatic(core.clj:2654)
	at clojure.core$sequence.invoke(core.clj:2639)
	at semantic_csv.core$parse_and_process.invokeStatic(core.cljc:285)
	at semantic_csv.core$parse_and_process.doInvoke(core.cljc:277)
	at clojure.lang.RestFn.invoke(RestFn.java:439)
	at clojure.core$partial$fn__5561.invoke(core.clj:2617)
	at clojure.lang.AFn.applyToHelper(AFn.java:156)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at clojure.core$apply.invokeStatic(core.clj:657)
	at clojure.core$apply.invoke(core.clj:652)
	at semantic_csv.impl.core$apply_kwargs.invokeStatic(core.cljc:17)
	at semantic_csv.impl.core$apply_kwargs.doInvoke(core.cljc:13)
	at clojure.lang.RestFn.invoke(RestFn.java:439)
	at semantic_csv.core$slurp_csv.invokeStatic(core.cljc:305)
	at semantic_csv.core$slurp_csv.doInvoke(core.cljc:298)
	at clojure.lang.RestFn.invoke(RestFn.java:439)
	at example.csv$process_with_cast_fns.invokeStatic(core.clj:11)
	at example.csv$process_with_cast_fns.invoke(core.clj:10)
	at example.csv$_main.invokeStatic(core.clj:22)
	at example.csv$_main.doInvoke(core.clj:19)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.lang.Var.invoke(Var.java:381)
	at user$eval149.invokeStatic(form-init6465213592991168308.clj:1)
	at user$eval149.invoke(form-init6465213592991168308.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:7062)
	at clojure.lang.Compiler.eval(Compiler.java:7052)
	at clojure.lang.Compiler.load(Compiler.java:7514)
	... 12 more

If I call process-manually, it will fail as well. The error that is thrown is instead a java.lang.NumberFormatException, and the line indicating where the failure happened points directly to the correct sc/->int call.

Caused by: java.lang.NumberFormatException: For input string: "X"
	at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
	at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
	at java.lang.Double.parseDouble(Double.java:538)
	at semantic_csv.casters$__GT_int.invokeStatic(casters.cljc:41)
	at semantic_csv.casters$__GT_int.invoke(casters.cljc:31)
	at semantic_csv.casters$__GT_int.invokeStatic(casters.cljc:38)
	at semantic_csv.casters$__GT_int.invoke(casters.cljc:31)
	at example.csv$manually_cast_columns.invokeStatic(core.clj:7)
	at example.csv$manually_cast_columns.invoke(core.clj:4)
	at clojure.core$mapv$fn__8088.invoke(core.clj:6832)
	at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:58)
	at clojure.core.protocols$fn__7847.invokeStatic(protocols.clj:136)
	at clojure.core.protocols$fn__7847.invoke(protocols.clj:124)
	at clojure.core.protocols$fn__7807$G__7802__7816.invoke(protocols.clj:19)
	at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
	at clojure.core.protocols$fn__7835.invokeStatic(protocols.clj:75)
	at clojure.core.protocols$fn__7835.invoke(protocols.clj:75)
	at clojure.core.protocols$fn__7781$G__7776__7794.invoke(protocols.clj:13)
	at clojure.core$reduce.invokeStatic(core.clj:6748)
	at clojure.core$mapv.invokeStatic(core.clj:6823)
	at clojure.core$mapv.invoke(core.clj:6823)
	at example.csv$process_manually.invokeStatic(core.clj:17)
	at example.csv$process_manually.invoke(core.clj:15)
	at example.csv$_main.invokeStatic(core.clj:23)
	at example.csv$_main.doInvoke(core.clj:19)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.lang.Var.invoke(Var.java:381)
	at user$eval149.invokeStatic(form-init6579145103885863466.clj:1)
	at user$eval149.invoke(form-init6579145103885863466.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:7062)
	at clojure.lang.Compiler.eval(Compiler.java:7052)
	at clojure.lang.Compiler.load(Compiler.java:7514)
	... 12 more

This is a contrived example to show the difference, but in my actual app, I have 20+ columns and hundreds of row, so figuring out exactly which column is failing is much more difficult. I suspect it's much faster to use :cast-fns, but the errors returned by the stack traces from manual casting are way more helpful, as they point directly to the function call where I attempt to cast bad input and display the given bad value.

I don't know what's possible in this library, but it would be very helpful if somehow the :cast-fns processing could return the raw exceptions instead of swallowing them and outputting only an NPE with no indication of which cast-fn or which value it failed on.

Thanks so much.

[metasoarous]

Hi @NoahTheDuke. Thanks for bringing up this issue.

You're right that this is unhelpfully obfuscating. This may have to do with the fact that the implementation uses transducers under the hood. Previously, the implementation looked pretty close to a generalized version of what you implemented, so not sure what else it could be.

I think the right thing to do here would be to catch in the transducer that casts values, since then you'd be able to print out the column name and value. I think it would also be helpful to have an :on-cast-error option that lets you decide whether to keep going or not, since sometimes a couple of missing values isn't a big deal.

Thanks again!