Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] users/list api call returns 500 entries and DSL does not loop through the next offset of 500 if more than 500 users present in command calm update cache #230

Closed
glover-chris opened this issue May 2, 2022 · 0 comments · Fixed by #260
Assignees
Labels
bug Something isn't working

Comments

@glover-chris
Copy link

glover-chris commented May 2, 2022

Describe the bug
The users/list api call returns 500 and DSL does not loop through the next offset of 500 if more than 500 users present.

Branch/Container
master

To Reproduce
Steps to reproduce the behavior:

  1. run calm update cache against PC where, say 1000, users exist.
  2. Users above the 500 count will not be updated in DSL cache.

Expected behavior
All users should be included in the update cache command.

Findings/Troubleshooting Done
Issue may be in the code here where the user api is called but there is no logic for more than 500 users.

res, err = Obj.list({"length": 1000})

Line 1200
client = get_api_client()
Obj = get_resource_api("users", client.connection)
res, err = Obj.list({"length": 1000})

Screenshots
If applicable, add screenshots to help explain your problem.

** System Configuration:**

  • OS: MAC/Windows
  • Python: 3.9.7

Additional context
Temporary fix in code at line 1195 of calm/dsl/db/table_config.py is working.

        client = get_api_client()
        Obj = get_resource_api("users", client.connection)
        offset = 0
        length = 500
        while True:
            res, err = Obj.list({"length": length, "offset": offset})
            if err:
                raise Exception("[{}] - {}".format(err["code"], err["error"]))
            res = res.json()
            for entity in res["entities"]:
                name = entity["status"]["name"]
                uuid = entity["metadata"]["uuid"]
                display_name = entity["status"]["resources"].get("display_name") or ""
                directory_service_user = (
                    entity["status"]["resources"].get("directory_service_user") or dict()
                )
                directory_service_ref = (
                    directory_service_user.get("directory_service_reference") or dict()
                )
                directory_service_name = directory_service_ref.get("name", "LOCAL")
                if directory_service_name:
                    cls.create_entry(
                        name=name,
                        uuid=uuid,
                        display_name=display_name,
                        directory=directory_service_name,
                    )
            if offset + length >= res['metadata']['total_matches']:
                break
            else:
                offset += length
@glover-chris glover-chris added the bug Something isn't working label May 2, 2022
abhijeetkaurav1st pushed a commit that referenced this issue Aug 22, 2023
…#230)

**Problem Statement:**

When creating a single VM bp, if you have parallel tasks that then call
different sub tasks within each parallel step, all sub tasks are run
from each parallel task.

**Fix:**

Earlier branch() helper was only exposed for runbooks and also made
changes to include branch() on decompilation

General topological sorting algo would not work here as we will have to
build the tree here recursively.
So following steps are included which can sum up the algo:
1) We first do a level traversal and store a list of all the children
corresponding to a given task in a list
2) Before rendering the task_tree template we do pre-computation of
ordered levelled tasks list which contains all the tasks in the order
they would be present after decompilation along with the indentation of
their parallel() and branch() helpers.
3) generate_levelled_task_indent_list() generated that list recursively
by keeping a track of parent of the current task, we check if the parent
of a task has already been visited or not, if it has been visited then
the current task would be present as a branch and its branch indentation
would be depth of the task from root node and its parallel indentation
would be none.
4) If the task node is not visited then it will have a parallel indent
as depth and branch indent as depth-1.
5) For finding the branch belongs to which parallel we store the index
in parallel_index_map.
6) Finally these levelled_task_indent_list and parallel_index_map are
passed as user_attrs while rendering the template for decompilation.

Eg decompilation:

![Screenshot 2023-05-05 at 3 42 44
PM](https://user-images.githubusercontent.com/123080484/236995118-1fa1300c-b135-4fd7-b65f-a1ddc857c8da.png)

```
with parallel() as p0:

            with branch(p0):
                CalmTask.Exec.escript(
                    name="Task1",
                    filename=os.path.join(
                        "scripts", "Package_Package1_Action___uninstall___Task_Task1.py"
                    ),
                    target=ref(Service1),
                )

                with parallel() as p1:

                    with branch(p1):
                        CalmTask.Exec.escript(
                            name="Task1_1",
                            filename=os.path.join(
                                "scripts", "Package_Package1_Action___uninstall___Task_Task1_1.py",
                            ),
                            target=ref(Service1),
                        )

                    with branch(p1):
                        CalmTask.Exec.escript(
                            name="Task1_2",
                            filename=os.path.join(
                                "scripts", "Package_Package1_Action___uninstall___Task_Task1_2.py",
                            ),
                            target=ref(Service1),
                        )

            with branch(p0):
                CalmTask.Exec.escript(
                    name="Task2",
                    filename=os.path.join(
                        "scripts", "Package_Package1_Action___uninstall___Task_Task2.py"
                    ),
                    target=ref(Service1),
                )

                CalmTask.Exec.escript(
                    name="Task2_1",
                    filename=os.path.join(
                        "scripts", "Package_Package1_Action___uninstall___Task_Task2_1.py",
                    ),
                    target=ref(Service1),
                )

```

**Regression Test Result:**
<img width="1383" alt="Screenshot 2023-06-14 at 10 06 15 AM"
src="https://github.com/ideadevice/calm-dsl/assets/123080484/5154cd08-c756-48dc-a7b2-ecc6d19a07bc">

---------

Co-authored-by: Abhishek Patwardhan <[email protected]>
(cherry picked from commit 2344fac4b382d79affb773a1788af1d5b314b5d2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants