OkJson is super slow when parsing long strings

[runnedrun]

I've been trying to pass long html strings in QC, and its been hanging. I investigated a bit and it looks like OkJson is the problem. When passing in a hash containing the entire html of a wikipedia page OkJson takes about 10 minutes to encode, whereas ruby's to_json takes 500 ms. OkJson is pretty awesome because of it's portability, but it's made it infeasible to pass long strings as arguments to QC. That being said, there may also be a better way to pass long strings to QC that I don't know about.

Thanks!

[joevandyk]

Why does queue_classic use okjson?

[ryandotsmith]

@joevandyk I chose okjson because dealing with all of Ruby's JSON dependencies is a nightmare.

[ryandotsmith]

@runnedrun you could consider writing the HTML to a table. Then pass then enqueue the id of that row into QC. Then your worker can just look up the HTML when he picks up the job.

[joevandyk]

@ryandotsmith isn't json in the stdlib?

[ryandotsmith]

@joevandyk good call. I don't have a lot of experience with the stdlib JSON implementation. Is there any reason not to use it?

http://www.ruby-doc.org/stdlib-1.9.2/libdoc/json/rdoc/JSON.html

[joevandyk]

@ryandotsmith json isn't in ruby 1.8's stdlib... that's the only downside i know of.

[ezkl]

@ryandotsmith Have you seen Intridea's MultiJSON?

MultiJSON tries to have intelligent defaulting. That is, if you have any of the supported engines already loaded, it will utilize them before attempting to load any. When loading, libraries are ordered by speed. First Oj, then Yajl, then the JSON gem, then JSON pure. If no other JSON library is available, MultiJSON falls back to OkJson, a simple, vendorable JSON parser.

It supports and is tested against most rubies.

I've been meaning to contribute to QC. Let me know if you'd like me to fork and implement.

[ryandotsmith]

closing the issue as @ezkl has a PR with a fix.