Scim users get distorted between construct, post, get.

[fisx]

Case handling in scim is a bit of a mess.

One of our the bigger issues was this: we parse RichInfo data in scim from a json schema that contains the key/value pairs both as a json object, and for when an ordering of keys is required, an assoc assoc array. The parser constructs the union of the unordered and the ordered map. The problem is that jsonLower (a relatively new helper function that attempts to deal with the criminally insane idea that scim json needs to be case insensitive in its object attributes. jsonLower honours this requirement, but of course does not go through all the string values in richFieldType in the assoc list that correspond to the object attributes in the map. Fixed eg. here (calling the smart constructor that eliminates duplicates rather than the ADT constructor).

To make things more fun, there is this issue with lower-casing in haskell, which we fix here by running all lower-casers in sequence, and overall by using case-insensitive consistently over the various lower-casing functions in text or base.

This PR also introduces a function normalizeLikeStored that normalizes Scim users. It is used a lot in tests, but also to make sure that the spar responses are a bit more "normal".

I also add some new tests, and increase the entropy when creating scim data in /services/spar/test-integration/Util/Scim.hs (because I ran into collisions there).

Sorry, this isn't very coherent. Maybe the changes will make more sense than this attempt at summarizing them? You can skim through the commit history, but I don't recommend reading them in order for a review.

Checklist

• The PR Title explains the impact of the change.
• The PR description provides context as to why the change should occur and what the code contributes to that effect. This could also be a link to a JIRA ticket or a Github issue, if there is one.
• changelog.d contains the following bits of information:
  • A file with the changelog entry in one or more suitable sub-sections. The sub-sections are marked by directories inside changelog.d.

[fisx]

failure on concourse:

when service can be discovered and the URLs change:             FAIL
>         test/unit/Test/Brig/Calling.hs:169:
>         servers should get overwritten
>         expected: Discovered (SFTServers (SrvEntry {srvPriority = 0, srvWeight = 0, srvTarget = SrvTarget {srvTargetDomain = "sft4.foo.example.com.", srvTargetPort = 443}} :| [SrvEntry {srvPriority = 0, srvWeight = 0, srvTarget = SrvTarget {srvTargetDomain = "sft5.foo.example.com.", srvTargetPort = 443}}]))
>          but got: Discovered (SFTServers (SrvEntry {srvPriority = 0, srvWeight = 0, srvTarget = SrvTarget {srvTargetDomain = "sft1.foo.example.com.", srvTargetPort = 443}} :| [SrvEntry {srvPriority = 0, srvWeight = 0, srvTarget = SrvTarget {srvTargetDomain = "sft2.foo.example.com.", srvTargetPort = 443}},SrvEntry {srvPriority = 0, srvWeight = 0, srvTarget = SrvTarget {srvTargetDomain = "sft3.foo.example.com.", srvTargetPort = 443}}]))

failure when running tests locally:

  test-integration/Test/Spar/APISpec.hs:769:5:
  1) WireIdPAPIV2.Spar.API, DELETE /identity-providers/:idp, zuser has no team, responds with 'insufficient permissions'
       uncaught exception: ErrorCall
       Assertions failed:
        1: 202 =/= 403

       Response was:

       Response {responseStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, responseVersion = HTTP/1.1, responseHeaders = [("Transfer-Encoding","chunked"),("Date","Fri, 24 Sep 2021 20:44:39 GMT"),("Server","Warp/3.3.13"),("Content-Encoding","gzip"),("Content-Type","application/json")], responseBody = Just "{\"code\":403,\"message\":\"The given phone number has been blacklisted due to suspected abuse or a complaint.\",\"label\":\"blacklisted-phone\"}", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}
       CallStack (from HasCallStack):
         error, called at src/Bilge/Assert.hs:89:5 in bilge-0.22.0-2K7mJKDUfGc17BkrPwuOOW:Bilge.Assert
         <!!, called at src/Bilge/Assert.hs:107:19 in bilge-0.22.0-2K7mJKDUfGc17BkrPwuOOW:Bilge.Assert
         !!!, called at test-integration/Util/Core.hs:580:3 in main:Util.Core
         createRandomPhoneUser, called at test-integration/Test/Spar/APISpec.hs:684:26 in main:Test.Spar.APISpec
         testGetPutDelete, called at test-integration/Test/Spar/APISpec.hs:769:5 in main:Test.Spar.APISpec

I predict both are flakes and won't reproduce for quite a while, but I'm not entirely confident about the latter.

[fisx]

I saw it one more time, then couldn't reproduce it in 20 or so other local runs, or on concourse. Can't think of any connection between this PR and that test. Hm...

[fisx]

we might get away with only this:

Suggested change

	nubber = Text.toLower . Text.toCaseFold . CI.foldedCase . richFieldType
	nubber = richFieldType

haven't tried.

[pcapriotti]

Yeah, Text.toCaseFold should be exactly the same as CI.foldedCase, since CI wraps a Text here, if I'm not mistaken. And we shouldn't call both toLower and toCaseFold.

[fisx]

Nope, this will trigger that thing that I think is a bug in case-insensitive again: basvandijk/case-insensitive#31 (comment)

[pcapriotti]

Looks reasonably clean. I'm not super confident, but it looks like we are now normalising everything at every stage, so it's more likely to be correct (especially if we decide to just ignore locale-specific issues and uncommon unicode characters like Cherokee letters), although I suspect some normalisation steps are unnecessary. I left a few comments below.

[pcapriotti]

Maybe adding a comment that this is just for testing is a good idea.

[pcapriotti]

It's probably not ideal to use case-folded strings as JSON keys (Unicode recommends to use case-folding only for comparison). Why is this JSON normalisation still needed? I would guess that once all the case comparisons on the haskell side are done correctly, JSON could be generated just using the "original" strings. Am I thinking about this wrong?

[fisx]

The original idea of jsonLower was to run it in json parsers initially, so that the actual parser could rely on everything being lower-case. This was an easy way of working around the fact that json and therefore aeson is strictly case-sensitive.

So morally, this is just lower-casing Values that are about to be deconstructed. But I will double-check and add this to the haddocks.

[fisx]

I double-checked, and I was right:

git grep -Hn jsonLower
libs/hscim/src/Web/Scim/Capabilities/MetaSchema.hs:84:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Capabilities/MetaSchema.hs:95:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Capabilities/MetaSchema.hs:114:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Class/Group.hs:63:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Class/Group.hs:76:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/AuthenticationScheme.hs:70:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/Common.hs:97:jsonLower :: Value -> Value
libs/hscim/src/Web/Scim/Schema/Common.hs:98:jsonLower (Object o) = Object . HM.fromList . fmap lowerPair . HM.toList $ o
libs/hscim/src/Web/Scim/Schema/Common.hs:100:    lowerPair (key, val) = (CI.foldCase key, jsonLower val)
libs/hscim/src/Web/Scim/Schema/Common.hs:101:jsonLower (Array x) = Array (jsonLower <$> x)
libs/hscim/src/Web/Scim/Schema/Common.hs:102:jsonLower same@(String _) = same -- (only object attributes, not all texts in the value side of objects!)
libs/hscim/src/Web/Scim/Schema/Common.hs:103:jsonLower same@(Number _) = same
libs/hscim/src/Web/Scim/Schema/Common.hs:104:jsonLower same@(Bool _) = same
libs/hscim/src/Web/Scim/Schema/Common.hs:105:jsonLower same@Null = same
libs/hscim/src/Web/Scim/Schema/ListResponse.hs:62:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/Meta.hs:70:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/ResourceType.hs:59:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Address.hs:39:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Certificate.hs:32:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Email.hs:47:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/IM.hs:32:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Name.hs:39:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Phone.hs:32:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/hscim/src/Web/Scim/Schema/User/Photo.hs:32:  parseJSON = genericParseJSON parseOptions . jsonLower
libs/wire-api/test/unit/Test/Wire/API/User/RichInfo.hs:171:    jsonroundtrip = unsafeParse . Scim.jsonLower . Aeson.toJSON

[fisx]

... aaand the haddocks already pretty much say this much. @pcapriotti if you can think of anything to add, please let me know (i can also do it in a separate PR).

[pcapriotti]

Yeah, Text.toCaseFold should be exactly the same as CI.foldedCase, since CI wraps a Text here, if I'm not mistaken. And we shouldn't call both toLower and toCaseFold.

[fisx]

although I suspect some normalisation steps are unnecessary.

very possible! i wanted to mention that my priority was soundness, not efficiency, but forgot.

[fisx]

hm...

    RichInfo Examples
      Empty rich info:                                                                              OK
      Old RichInfoMapAndList:                                                                       OK
      case insensitive 'richinfo':                                                                  OK
      RichInfoMapAndList as only assoc list:                                                        OK
      RichInfoMapAndList Map:                                                                       OK
      Without Old RichInfoMapAndList:                                                               OK
      wrong version:                                                                                OK
      drop empty fields:                                                                            OK
    'toRichInfoAssocList', 'fromRichInfoAssocList'
      works (counter-example of earlier bug):                                                       OK
      works (property):                                                                             FAIL (0.17s)
        *** Failed! Falsified (after 37 tests and 10 shrinks):
        RichInfoAssocList {unRichInfoAssocList = [RichField {richFieldType = "\43889", richFieldValue = "`"}]}
        RichInfoAssocList {unRichInfoAssocList = [RichField {richFieldType = "\43889", richFieldValue = "`"},RichField {richFieldType = "\5025", richFieldValue = "`"}]} /= RichInfoAssocList {unRichInfoAssocList = [RichField {richFieldType = "\43889", richFieldValue = "`"}]}
        Use --quickcheck-replay=719615 to reproduce.

looks like this doesn't work out of the box?