Coercion refactor

[STRML]

This fixes some of the issues in #208 and builds on top of #231 which was rebased out without warning.

There are two types of coercion Strong-Remoting should be doing, and they should not share code / be intermixed.

1. Coercing data from loose input methods like querystrings, HTTP headers, and form-encoded data, where all data is received as a string.
   • We should do the least-surprising thing here as much as possible. http-coerce will coerce to the expected type as closely as possible, while special-casing 'undefined' and '' to undefined.
     • I chose undefined instead of null so it plays nicely with ES6 default parameters.
     • Note: Should input like '?toggle' (the empty string) coerce to true? If so, we should special-case this querystring behavior.
   • If the type is 'any', we attempt to match as a number, boolean, or simply return the string as-is if no match.
   • No coercion should be done on JSON, which preserves types (except Dates).
2. In SharedMethod, coercion should only be done via Dynamic.
   • If no Dynamic convert is available, some legacy code attempts to parse JSON strings.
   • Erroring is better than silently doing the wrong thing. Errors should be thrown more often on malformed input.

For context, this grew out of some very frustrating behavior with master. Most of that behavior is documented in #231. This additionally fixes some bad behavior in #231, like the 'boolean' type coercing the string "false" to true.

[STRML]

Should we special-case arrays as something we coerce regardless? In remoting methods, it is awfully nice to be sure an array type is passed, regardless of whether or not the input was null - an empty array is simpler to deal with. This is the current behavior.

[bajtos]

@STRML thank you for the patch, could you please rebase it on top of the current master, which includes your older patch #231?

Also PTAL at strongloop/loopback#1806, how will the changes in this pull request affect that issue?

Are you aware of any edge cases where this patch may break existing applications?

[bajtos]

Moved Dynamic() invocation into shared-method. This puts the bulk of the coercion into the http handlers, assuming that otherwise coercion is not desired in favour of strictness.

I find this a bit surprising. In my opinion, the coercion problem is transport-specific and thus it should be handled by transport-specific code (e.g. HttpContext). Imagine we had support for a transport that can preserve all type information (e.g. something like https://github.com/cognitect/transit-js). In that case no coercion should be made, which would be difficult to achieve with SharedMethod performing the coercion.

However, I am not very familiar with this part of strong-remoting codebase. If the solution proposed here is the best what we can achieve now, then I can live with that.

@ritch @fabien @raymondfeng could you PTAL and review this patch?

[kblcuk]

FYI integration test in project I work on started to fail with latest release because of incorrect parsing of boolean request params (where myParam=false would evaluate to true in remote method). Rolling back to 2.22.2 solved this, so it's something introduced in latest (probably linked here) release.

[STRML]

Yes, this is a bug fixed in this PR. I'll be back in the office soon and
can better address the concerns so we can get this merged.
On Dec 14, 2015 11:17 AM, "Alexei Mikhailov" [email protected]
wrote:

FYI integration test in project I work on started to fail with latest
release because of incorrect parsing of boolean request params (where
myParam=false would evaluate to true in remote method). Rolling back to
2.22.2 solved this, so it's something introduced in latest (probably linked
here) release.

—
Reply to this email directly or view it on GitHub
#264 (comment)
.

[STRML]

@bajtos It seems odd to me, to make http coercion user-configurable. It should simply follow simple rules, and if the user wants to coerce further, he can do so for all possible input via Dynamic.

This lets the user configure nice things, like auto-coercing strings into arrays of strings if the method only accepts arrays, and so on. As-is, I find it confusing how many coercion paths there are:

• Dynamic()
• http-coerce.js (which should only coerce form encoding and qs)
• Additional coercion in SharedMethod that doesn't use Dynamic.

My first thought was, we could simply put all coercion logic in Dynamic, but this would make it impossible to e.g. properly coerce Dates, which can't be communicated over JSON.

So instead, I've settled on:

• Transport-specific coercion for:
  • Form/QS (heavy coercion because everything's a string).
  • JSON (next to nothing, except possibly ISO Date string -> Date)
• Remoting-global coercion via user-configurable Dynamic, which coerces all args regardless of source
  • Useful for single item -> Array, Date -> Moment, Number -> int32, etc

Does this make sense?

[bajtos]

I am reluctant to remove request values from error messages. By including the request value, we make troubleshooting easier, e.g. when one doesn't have access to request bodies, only the error messages.

We already had a user asking to include values in validation errors, see loopbackio/loopback-datasource-juggler#389

method.invoke('ctx', { obj: '<script>alert(1)</script>' }, function(err) {

The use case where the request value contains <script> element should be IMO handled by the client processing the responses, it's the client responsibility to html-encode any error messages before rendering them in the UI.

I am also not convinced that the argument value is the only place that can be abused for code injection, i.e. this change may be giving a false sense of security. However, if you insist, then I am willing to consider adding a feature flag to exclude request values from error messages, the flag should be disabled by default.

@raymondfeng @ritch thoughts?

[STRML]

This is classic reflection, and production APIs simply shouldn't be echoing request data back. Yes, there are ways to mitigate it, and yes, you shouldn't be displaying it in the UI, but there are always edge cases (like MIME sniffing) that can make this dangerous. With this reflection hole open, in certain cases, XSS would be possible via a specially-crashed API link.

[bajtos]

@kblcuk FYI integration test in project I work on started to fail with latest release because of incorrect parsing of boolean request params (where myParam=false would evaluate to true in remote method). Rolling back to 2.22.2 solved this, so it's something introduced in latest (probably linked here) release.

I am going to revert the patch that introduced the problem, to buy us more time to deal with these issues properly. See #269

@STRML Does this make sense?

I am afraid I'll need a bit more time to digest what you wrote (and the code changes proposed here) to answer.

[kblcuk]

@bajtos ok, thanks!

[bajtos]

Update: I am working on bringing the changes proposed here to the current 2.x series, you can track my progress in https://github.com/strongloop/strong-remoting/tree/feature/coercion-refactor

[bajtos]

Closing in favour of #343