Query filter and smart inventory; host membership of indirect group

[bartmeuris]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API

SUMMARY

Nested groups cannot be found using host_filter API or smart inventories.

ENVIRONMENT

• AWX version: 1.0.3.14
• AWX install method: awx-container from geerlingguy
• Ansible version: 2.4.3.0

STEPS TO REPRODUCE

Preparation: create groups:

• Create a group in AWX, for example dnsservers
• In that group, create 2 other groups: dns-authoritative and dns-recursive
• Add diffrerent hosts to the dns-authoritative and dns-recursive groups, and none directly in the dnsservers group.

Using the API, go to:

/api/v2/hosts/?host_filter=groups__name=dnsservers

EXPECTED RESULTS

Expected results are that when searching for dnsservers would return all hosts in both the dns-authoritative and dns-recursive groups.

ACTUAL RESULTS

Actual result is that an empty result is returned:

{
    "count": 0,
    "next": null,
    "previous": null,
    "results": []
}

Both the /api/v2/hosts/?host_filter=groups__name=dns-authoritative and /api/v2/hosts/?host_filter=groups__name=dns-recursive requests do return the expected hosts in those groups.

ADDITIONAL INFORMATION

• When in the UI, you go to an the inventory, select this dnsservers group, and look at the hosts there, you do see all hosts I would expect to be returned.
• Using /api/v2/hosts/?groups__search= does not make any difference.
• This also does not work in Smart Inventories. A smart host filter for dns-authoritative or dns-recursive works, but dnsservers results in an empty result.

[jaredevantabor]

@bartmeuris just a thought, but could you try group.name instead of group_name? The documentation here suggests that syntax for group searches.

[jaredevantabor]

@bartmeuris the host_filter is primarily for filtering host facts, and so for your query for groups, you probably don't need the host_filter, and can query the host endpoint directly. So /api/v2/hosts/?groups__name=dns-authoritative should work. Let me know either way.

[bartmeuris]

@jaredevantabor the problem isn't that /api/v2/hosts/?groups__name=dns-authoritative doesn't work, it's that when I search for /api/v2/hosts/?groups__name=dnsservers that it doesn't include any hosts from the sub-groups.

This is also an issue with Smart Inventories, which is how I bumped into the issue, but since this also had this issue when using the API, I would expect the problem to be the same.

[chrismeyersfsu]

Querying the API for group membership must be done with the tree structure in mind.
i.e.
/api/v2/hosts/?groups__parents__name=dnsservers or /api/v2/hosts/?host_filter=groups__parents__name=dnsservers

Note that the above query will only work for hosts that exist in a group 1 level below dnsservers.

[bartmeuris]

Ok, the groups__parents__name seems to work, and groups.parents.name seems to be working in the Smart inventory too.

This is very confusing however, I would expect hosts from sub-groups to be included in the parent. This also has the problem that this works for hosts in sub-groups, but would not include any hosts included directly in the dnsservers group but not present in any of the sub-groups.

I can use ?host_filter=(groups__parents__name=dnsservers or groups__name=dnsservers) through the API, however in the Smart inventory, at least from the UI, I can't to filter like this, since it adds an "and" between expressions. Through the API I can patch the
"host_filter" for this smart inventory to be "groups__parents__name=dnsservers or groups__name=dnsservers", but that's quite annoying and would make it much harder to use.

Now this non-listing of sub-group member hosts is weird and counter-intuitive, but I suspect that when AWX/Tower expects it to work this way, changing this could have a serious impact and changing this behavior would be a no-go. Fixing the Smart Inventory UI to allow for OR expressions could be part of the solution and is imho needed, but that's besides this issue. It would still be ugly and counter-intuitive in my opinion, since you would still have to specify that it's a parent folder or a direct folder membership. I have no idea how 3 levels or more should be handled, I haven't tested this - but I wouldn't be surprised if this turned out to be problematic too.

However, after some searching, there seems to be an all_groups present in the related field of a host, which I would expect to be able to search - but ?host_filter=all_groups__name=dnsservers results in an error:

{
    "error": "Cannot resolve keyword 'all_groups' into field. Choices are: activitystream, ad_hoc_command_events, ad_hoc_commands, ansible_facts, ansible_facts_modified, created, created_by, created_by_id, description, enabled, facts, groups, has_active_failures, has_inventory_sources, id, insights_system_id, instance_id, inventory, inventory_id, inventory_sources, job_events, job_events_as_primary_host, job_host_summaries, jobs, last_job, last_job_host_summary, last_job_host_summary_id, last_job_id, modified, modified_by, modified_by_id, name, smart_inventories, tagged_items, tags, variables"
}

As expected, this also doesn't work using the smart inventories, since they use the same filtering. The documentation however states (emphasis is mine):

• __ to reference related fields in relational fields
• __ is used on ansible_facts to separate keys in a JSON key path

My interpretation here is that this should be query-able, but isn't If this would be, this would be the best solution in my opinion. This would mean I would be able to filter in the smart inventory using all_groups.name:dnsservers and use ?host_filter=all_groups__name=dnsservers through the API.

[chrismeyersfsu]

@bartmeuris You've understood the general design of the API filtering well. However, not all fields that we show are queriable. Manifested fields are not queriable.

all_groups and all_hosts is not queriable because it is a generated field (property to be exact

awx/awx/main/models/inventory.py

Line 930 in 1582fcb

@property

and

awx/awx/main/models/inventory.py

Line 634 in 1582fcb

def all_groups(self):

)

To your point that arbitrary nested host group membership is not queriable. This is not easy when thinking about a relational database. The group -> group -> ... -> host relationships are stored as foreign keys. If you have united_states east and west groups with hosts mail-east and web-west. It would be stored with entries east -> united_states west -> united_states mail-east -> east web-west -> west (actually they are many to many relationships, but for simplicity lets think of it as a foreign key).

So you can see, there is no direct relationship from mail-east to united_states nor is there a direct relationship from mail-west to united_states. Hence the query limitation.

We could solve this problem by (1) creating a link from every host to every group that it's in. This would require a pretty pricey update anytime the inventory topology changes or (2) upon viewing a particular group's host list, we can compute that host list. all_hosts endpoint does this.

I think we can make (2) queriable since it returns a QuerySet. all_hosts and all_groups would then be usable in host_filter and other queries.

[bartmeuris]

@chrismeyersfsu yes I already dove into the code and had pinpointed it to that @property in the Host model, but I'm not familiar with Django or the AWX architecture.

Glad to hear this could possibly be solved, I currently know the limitations, so I can work with this, and for my smart inventories use the API to fix the host_filter but it would be more intuitive and easier if this wasn't necessary.

[chrismeyersfsu]

Postgres LTREE column type could be leveraged to represent the Ansible inventory group structure. With this, I think we could provide the kinds of queries that we want here like "is HostA a member of group us-east".

[shanemcd]

We are moving away from smart inventories and recently introduced a new feature called Constructed Inventory. That merged in #13448 and got released with AWX 22. Constructed Inventories allow for querying host vars. I'm going to go ahead and close this and can also follow up with a link to a blog post that @AlanCoding is currently working on.

Docs to new constructed inventory source:

https://github.com/ansible/awx/blob/devel/docs/inventory/constructed_inventory.md