update-space-users erroring with pagination

[msawant060]

update-space-users job failing with an error after the 1.0.60 release .
error: got errors processing update space users [role for type space with GUID[ab5e8f16-5ae3-4a35-8388-2db030ccc31c] is returned multiple times, pagination for v3 roles is not working role for type space with GUID[ed0d4cb5-7e52-4f52-ba60-9f78629aa230] is returned multiple times, pagination for v3 roles is not working]

To Reproduce
Steps to reproduce the behavior:
Currently not reproducible

Expected behavior
job should pass without the pipeline failing

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
With release 1.0.59 there seem to be no issue , but since then for any further release we are noticing this error in the build logs.

This PR seems to have introduced the issue
#386

in this release
v1.0.59...v1.0.60

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[calebwashburn]

Added this logic on one of the test environments as found errors in v3 api pagination, rebuilt environment and could not re-produce. This appeared to be an error with v3 api. Would be interested to see if this script shows duplicates as used to to troubleshoot when seeing this before

#!/bin/bash -e
guids=$(mktemp)
sortedGuids=$(mktemp)
sortedUniquGuids=$(mktemp)
api_endpoint=https://api.<your system domain/
result=$(cf curl "/v3/roles?order_by=updated_at&per_page=5000&types=org_user,space_developer")
next_page=$(echo $result | jq .pagination.next.href -r)
echo $result | jq '.resources[].guid' -r >> $guids
while [ true ]; do
    if [ "$next_page" == "null" ]   
    then
        break
    fi 
    next_url=$(echo "${next_page##$api_endpoint}")
    result=$(cf curl $next_url)
    echo $result | jq '.resources[].guid' -r >> $guids
    next_page=$(echo $result | jq .pagination.next.href -r)
    if [ "$next_page" == "null" ]
    then
        break
    fi 
done

echo "Results"
cat $guids 
echo "Duplicates"
sort $guids | uniq -cd

[calebwashburn]

Also would be interested to see if anything insightful is returned from

cf curl /v3/roles/ab5e8f16-5ae3-4a35-8388-2db030ccc31c

[msawant060]

cf curl /v3/roles/ab5e8f26-5ae3-4a35-8388-2db030ccc31c
{
"guid": "ab5e8f26-5ae3-4a35-8388-2db030ccc31c",
"created_at": "2023-03-30T13:53:52Z",
"updated_at": "2023-03-30T13:53:52Z",
"type": "space_developer",
"relationships": {
"user": {
"data": {
"guid": "c41c5869-ee29-4519-8dd4-5731d9f4a3a1"
}
},
"space": {
"data": {
"guid": "c4050468-b77d-4746-8e07-8dbb25b90c7b"
}
},
"organization": {
"data": null
}
},
"links": {
"self": {
"href": "https://api.system.intrepidstr.dsglobal.org/v3/roles/ab5e8f26-5ae3-4a35-8388-2db030ccc31c"
},
"user": {
"href": "https://api.system.intrepidstr.dsglobal.org/v3/users/c41c5869-ee29-4519-8dd4-5731d9f4a3a1"
},
"space": {
"href": "https://api.system.intrepidstr.dsglobal.org/v3/spaces/c4050468-b77d-4746-8e07-8dbb25b90c7b"
}
}
}

[msawant060]

Added this logic on one of the test environments as found errors in v3 api pagination, rebuilt environment and could not re-produce. This appeared to be an error with v3 api. Would be interested to see if this script shows duplicates as used to to troubleshoot when seeing this before

#!/bin/bash -e
guids=$(mktemp)
sortedGuids=$(mktemp)
sortedUniquGuids=$(mktemp)
api_endpoint=https://api.<your system domain/
result=$(cf curl "/v3/roles?order_by=updated_at&per_page=5000&types=org_user,space_developer")
next_page=$(echo $result | jq .pagination.next.href -r)
echo $result | jq '.resources[].guid' -r >> $guids
while [ true ]; do
    if [ "$next_page" == "null" ]   
    then
        break
    fi 
    next_url=$(echo "${next_page##$api_endpoint}")
    result=$(cf curl $next_url)
    echo $result | jq '.resources[].guid' -r >> $guids
    next_page=$(echo $result | jq .pagination.next.href -r)
    if [ "$next_page" == "null" ]
    then
        break
    fi 
done

echo "Results"
cat $guids 
echo "Duplicates"
sort $guids | uniq -cd

tmp.txt

[calebwashburn]

This script will detect if there is the same issue with pagination that cf-mgmt is checking for. So if running on a foundation and it shows duplicates cf-mgmt should also see those duplicates.

[calebwashburn]

@msawant060 are you still having this issue or have you fixed it? Wanting to know if we can close this issue or need to do more analysis

[supp429]

hello @calebwashburn We ran into the same issue with 1.0.69. As indicated by another user above, there are no duplicates after running your script. But I have noticed something odd about the roles that the script printed... most of them have "cac2-11ed-bcec-005056a01210" common at the end. See below.

071cb86d-cac2-11ed-bcec-005056a01210
0752ac77-cac2-11ed-bcec-005056a01210
076f3fb4-cac2-11ed-bcec-005056a01210
0739c74b-cac2-11ed-bcec-005056a01210
0742a2d0-cac2-11ed-bcec-005056a01210

Update: Reverted to old version (1.0.43) that we were using previously and this issue does not exist.

[calebwashburn]

@supp429 if you can recreate the problem I would like to provide a build of this to better capture what is happening thru some debug output to see if we can isolate why this pagination issue is happening.

[calebwashburn]

@supp429 I have create a development build to see if we can better trap this scenario as unable to duplicate in any of our labs. If you could change to use pivotalservices/cf-mgmt:develop vs one of tagged versions as your docker image that would be helpful as outputting more data in the error message to get more insights. No need to enable debug level logging for this.

[calebwashburn]

Closing this issue and should be resolved in https://github.com/vmware-tanzu-labs/cf-mgmt/releases/tag/v1.0.85

[supp429]

Hello @calebwashburn I apologize for no response on the earlier posts. we ran into this again with v1.0.88. We did an upgrade today to the latest version to take advantage of the fixes related to log running sync jobs and API timeout issues. I believe it was the same environment that we had seen this issue lat year. Also, there were no duplicates returned by your script above.

Please let me know if you want me to capture any additional traces. Thank you!

UPDATE: Debug logs below.

2024/05/29 17:28:40 D0529 17:28:40.861327 52 initialize.go:78] Using Version: [1.0.88], Commit: [01a5d93ea44bf4c2cea85238f7b8a90431a9f8b2] of cf-mgmt
2024/05/29 17:28:46 D0529 17:28:46.747409 52 manager.go:87] Found 8328 roles from space API
2024/05/29 17:28:46 Sending quit message and waiting for confirmation
2024/05/29 17:28:46 Shutting down - quit message received
2024/05/29 17:28:46 Closing network connection
error: got errors processing update space users [role for type space with GUID[07548fe0-cac2-11ed-bcec-005056a01210] is returned multiple times, prior role [{"guid":"07548fe0-cac2-11ed-bcec-005056a01210","created_at":"2023-03-25T04:03:36Z","updated_at":"2023-03-25T04:03:40Z","type":"space_developer","relationships":{"space":{"data":{"guid":"dfc721d9-471e-4100-a2af-2439d48943b9"}},"user":{"data":{"guid":"1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}},"organization":{"data":null}},"links":{"self":{"href":"https://api.acme.net/v3/roles/07548fe0-cac2-11ed-bcec-005056a01210"},"space":{"href":"https://api.acme.net/v3/spaces/dfc721d9-471e-4100-a2af-2439d48943b9"},"user":{"href":"https://api.acme.net/v3/users/1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}}}] and current role [{"guid":"07548fe0-cac2-11ed-bcec-005056a01210","created_at":"2023-03-25T04:03:36Z","updated_at":"2023-03-25T04:03:40Z","type":"space_developer","relationships":{"space":{"data":{"guid":"dfc721d9-471e-4100-a2af-2439d48943b9"}},"user":{"data":{"guid":"1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}},"organization":{"data":null}},"links":{"self":{"href":"https://api.acme.net/v3/roles/07548fe0-cac2-11ed-bcec-005056a01210"},"space":{"href":"https://api.acme.net/v3/spaces/dfc721d9-471e-4100-a2af-2439d48943b9"},"user":{"href":"https://api.acme.net/v3/users/1b53a4b2-bb19-471b-bf9b-1393b28a46eb"}}}]]

[calebwashburn]

@supp429 if you are having issues with the latest version please open a new issue describing the problem and include and output from the specific step that shows the error. Ideally adding DEBUG output is helpful.

[supp429]

I will create a new issue. Thanks!