remove computed inventory fields from Host and Group #5448

ryanpetrello · 2019-12-04T18:04:15Z

see: https://github.com/ansible/tower/issues/2592

ryanpetrello · 2019-12-04T18:05:06Z

awx/api/serializers.py

                                           'has_inventory_sources',
                                           'total_inventory_sources',
                                           'inventory_sources_with_failures',
                                           'organization_id',
                                           'kind',
                                           'insights_credential_id',),
-    'host': DEFAULT_SUMMARY_FIELDS + ('has_active_failures',
-                                      'has_inventory_sources'),
-    'group': DEFAULT_SUMMARY_FIELDS + ('has_active_failures',


Group loses all of these - assuming we remove the green/red icons and don't care about API backwards compat at /api/v2/dashboard/, I can't see any tangible reason we spend our time calculating these.

We deprecated these fields recently.

assuming we ... don't care about API backwards compat at /api/v2/dashboard/

I can't imagine that we do

assuming we remove the green/red icons

That is a removal of user-facing functionality. There is very non-trivial meaning in those orbs, because the Ansible notion of membership includes indirect via children groups. At the same time, merely a red/green indicator is a pretty gross over-simplification. Anyway, the final call is within the UX wheelhouse.

ryanpetrello · 2019-12-04T18:05:25Z

awx/api/serializers.py

                                           'has_inventory_sources',
                                           'total_inventory_sources',
                                           'inventory_sources_with_failures',
                                           'organization_id',
                                           'kind',
                                           'insights_credential_id',),
-    'host': DEFAULT_SUMMARY_FIELDS + ('has_active_failures',


Host loses all of these, and they're replaced w/ queries that generate the data from other sources:

https://github.com/ansible/awx/pull/5448/files#diff-a81324c523b41de7296fdd5ff9063d10R1748

What's the performance differential when handling this in the list views?

@matburt the prior case is going to be slightly faster on list views (it's just two boolean field lookups).

The after version has two new generated fields on the list view now:

def get_has_active_failures(self, obj): return bool( obj.last_job_host_summary and obj.last_job_host_summary.failed ) def get_has_inventory_sources(self, obj): return obj.inventory_sources.exists()

Here's the /api/v2/hosts/ list view before I preloaded my database with a bunch of data. Note that I've set up an inventory w/ a bunch of hosts:

awx-dev=> SELECT COUNT(*) FROM main_host; count ------- 22809 (1 row)

Next, I made a bunch of inventory sources, and a ton of JobHostSummary objects associated with every host:

In [1]: i = Inventory.objects.first() In [2]: for s in InventorySource.objects.all(): ...: for h in Host.objects.all(): ...: h.inventory_sources.add(s) ...: In [24]: for i in range(MANY): ...: j = Job.objects.create() ...: for h in Host.objects.all(): ...: j.job_host_summaries.get_or_create(host=h, host_name=h.name, defaults=host_stats) ...: ...:

awx-dev=> SELECT COUNT(*) FROM main_jobhostsummary; count -------- 633365 (1 row) awx-dev=> SELECT COUNT(*) FROM main_host_inventory_sources; count ------- 132610 (1 row)

First pass on this shows that it's definitely slower, but not that much slower:

I'm going to keep running my data generator all day and see if I can get some of these tables into the millions and see if I notice any additional performance degradation.

I'll also see if I can sprinkle in some ORM prefetching to cut down on these queries a bit.

Update: I added some more prefetching to speed this up a bit, and it cut the query count in half:

https://github.com/ansible/awx/pull/5448/files#r362574316

ryanpetrello · 2019-12-04T18:07:43Z

awx/api/views/__init__.py

@@ -204,20 +204,14 @@ def get(self, request, format=None):
                                            'failed': ec2_inventory_failed.count()}

        user_groups = get_user_queryset(request.user, models.Group)
-        groups_job_failed = (


This makes group-based failure tracking no longer exist/supported at /api/v2/dashboard/.

It feels like this was for the old inventory model when groups and inventory sources were tied together.

Yep, that's my impression, too.

ryanpetrello · 2019-12-04T18:13:52Z

Inventory failure tracking still works:

ryanpetrello · 2019-12-04T18:14:14Z

Host state and recent jobs tracking works:

ryanpetrello · 2019-12-04T18:14:26Z

Groups have no status indicators anymore:

ryanpetrello · 2019-12-04T18:14:41Z

Dashboard loads, and the top failure links work when clicked on:

AlanCoding · 2019-12-04T19:19:45Z

awx/api/serializers.py

@@ -1756,6 +1745,14 @@ def to_representation(self, obj):
            ret['last_job_host_summary'] = None
        return ret

+    def get_has_active_failures(self, obj):
+        return bool(
+            obj.last_job_host_summary and obj.last_job_host_summary.failed


Oh, so I guess we're already fetching this information and showing it in summary_fields. And fetching it... and even more

awx/awx/main/access.py

Lines 908 to 909 in 90d38a5

select_related = ('created_by', 'modified_by', 'inventory',

'last_job__job_template', 'last_job_host_summary__job',)

AlanCoding · 2019-12-04T19:22:56Z

awx/api/views/__init__.py

        data['hosts'] = {'url': reverse('api:host_list', request=request),
-                         'failures_url': reverse('api:host_list', request=request) + "?has_active_failures=True",
+                         'failures_url': reverse('api:host_list', request=request) + "?last_job_host_summary__failed=True",


Doing this filter, I get different answers in my own server compared to doing /api/v2/hosts/?last_job__failed=true. Reason still unclear.

Okay, this means that the host had failed tasks.

last job failed means that any host in the last job this host was a part of failed.

softwarefactory-project-zuul · 2019-12-04T19:24:25Z

Build failed.

awx-api-lint : SUCCESS in 2m 48s
awx-api : FAILURE in 8m 10s
awx-ui : FAILURE in 3m 01s
awx-ui-next : SUCCESS in 6m 10s
awx-swagger : SUCCESS in 10m 28s
awx-detect-schema-change : FAILURE in 10m 24s (non-voting)
awx-ansible-modules : SUCCESS in 3m 03s

AlanCoding · 2019-12-04T19:44:55Z

awx/api/serializers.py

+        )
+
+    def get_has_inventory_sources(self, obj):
+        return obj.inventory_sources.exists()


This will probably incur an additional query. However, I see it as a valid candidate for prefetch_related, because there will be a large amount of duplication in many cases.

Yep, I checked, and this does add more queries, so I added it to prefetch_related and watched the query count go down.

Thanks. That might exhaust the actionable comments I had. I will go back over the diff one last time.

https://github.com/ansible/awx/pull/5448/files#diff-c14c9d695bfbeb70889f2b42b18ed42dR910

AlanCoding · 2019-12-04T19:52:16Z

I was worried about this, but much lower concern after seeing this. I would still like to profile the queries from the hosts endpoints.

ryanpetrello · 2019-12-04T20:30:34Z

I was worried about this, but much lower concern after seeing this. I would still like to profile the queries from the hosts endpoints.

I do think we should add an index to main_jobhostsummary.failed

softwarefactory-project-zuul · 2019-12-04T20:43:49Z

Build failed.

awx-api-lint : SUCCESS in 3m 23s
awx-api : FAILURE in 10m 28s
awx-ui : FAILURE in 3m 35s
awx-ui-next : SUCCESS in 6m 27s
awx-swagger : SUCCESS in 10m 13s
awx-detect-schema-change : FAILURE in 10m 37s (non-voting)
awx-ansible-modules : SUCCESS in 3m 48s

softwarefactory-project-zuul · 2019-12-04T21:26:23Z

Build failed.

awx-api-lint : SUCCESS in 3m 38s
awx-api : SUCCESS in 8m 32s
awx-ui : FAILURE in 3m 01s
awx-ui-next : SUCCESS in 6m 32s
awx-swagger : SUCCESS in 8m 49s
awx-detect-schema-change : FAILURE in 10m 14s (non-voting)
awx-ansible-modules : SUCCESS in 4m 14s

softwarefactory-project-zuul · 2019-12-04T22:24:56Z

Build succeeded.

awx-api-lint : SUCCESS in 3m 07s
awx-api : SUCCESS in 9m 18s
awx-ui : SUCCESS in 8m 52s
awx-ui-next : SUCCESS in 8m 02s
awx-swagger : SUCCESS in 10m 32s
awx-detect-schema-change : FAILURE in 11m 40s (non-voting)
awx-ansible-modules : SUCCESS in 3m 39s

kdelee

tested the API changes as they effect us, looks good to me! Will leave it up to @dsesami for final approval from UI perspective

softwarefactory-project-zuul · 2020-01-08T17:17:58Z

Build succeeded.

awx-api-lint : SUCCESS in 5m 19s
awx-api : SUCCESS in 11m 35s
awx-ui : SUCCESS in 7m 15s
awx-ui-next : SUCCESS in 9m 04s
awx-swagger : SUCCESS in 11m 29s
awx-detect-schema-change : FAILURE in 13m 24s (non-voting)
awx-ansible-modules : SUCCESS in 6m 04s

dsesami

UI changes look good, thanks for this.

softwarefactory-project-zuul · 2020-01-08T17:31:50Z

Build succeeded.

awx-api-lint : SUCCESS in 3m 39s
awx-api : SUCCESS in 10m 14s
awx-ui : SUCCESS in 5m 19s
awx-ui-next : SUCCESS in 8m 46s
awx-swagger : SUCCESS in 9m 26s
awx-detect-schema-change : FAILURE in 11m 37s (non-voting)
awx-ansible-modules : SUCCESS in 3m 19s

softwarefactory-project-zuul · 2020-01-13T15:40:28Z

Build failed.

awx-api-lint : SUCCESS in 11m 33s
awx-api : FAILURE in 17m 39s
awx-ui : SUCCESS in 14m 33s
awx-ui-next : SUCCESS in 15m 53s
awx-swagger : SUCCESS in 20m 40s
awx-detect-schema-change : FAILURE in 17m 26s (non-voting)
awx-ansible-modules : SUCCESS in 12m 11s

softwarefactory-project-zuul · 2020-01-13T16:20:19Z

Build succeeded.

awx-api-lint : SUCCESS in 5m 28s
awx-api : SUCCESS in 11m 44s
awx-ui : SUCCESS in 5m 33s
awx-ui-next : SUCCESS in 8m 45s
awx-swagger : SUCCESS in 11m 25s
awx-detect-schema-change : FAILURE in 11m 14s (non-voting)
awx-ansible-modules : SUCCESS in 4m 42s

Remove the status col from any group list that used the now-removed computed fields.

softwarefactory-project-zuul · 2020-01-14T21:55:22Z

Build failed.

awx-api-lint : SUCCESS in 4m 21s
awx-api : FAILURE in 9m 58s
awx-ui : SUCCESS in 5m 51s
awx-ui-next : SUCCESS in 11m 06s
awx-swagger : SUCCESS in 12m 24s
awx-detect-schema-change : FAILURE in 12m 33s (non-voting)
awx-ansible-modules : SUCCESS in 4m 53s

softwarefactory-project-zuul · 2020-01-14T22:11:28Z

Build succeeded.

awx-api-lint : SUCCESS in 2m 44s
awx-api : SUCCESS in 9m 33s
awx-ui : SUCCESS in 5m 34s
awx-ui-next : SUCCESS in 7m 42s
awx-swagger : SUCCESS in 9m 57s
awx-detect-schema-change : FAILURE in 12m 07s (non-voting)
awx-ansible-modules : SUCCESS in 3m 58s

softwarefactory-project-zuul · 2020-01-15T05:20:26Z

Build succeeded.

awx-api-lint : SUCCESS in 3m 29s
awx-api : SUCCESS in 7m 51s
awx-ui : SUCCESS in 6m 23s
awx-ui-next : SUCCESS in 7m 24s
awx-swagger : SUCCESS in 10m 09s
awx-detect-schema-change : FAILURE in 10m 39s (non-voting)
awx-ansible-modules : SUCCESS in 3m 42s

softwarefactory-project-zuul · 2020-01-15T17:24:49Z

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

awx-api-lint : SUCCESS in 13m 15s
awx-api : FAILURE in 17m 21s
awx-ui : SUCCESS in 14m 37s
awx-ui-next : SUCCESS in 18m 32s
awx-swagger : SUCCESS in 22m 02s
awx-detect-schema-change : FAILURE in 22m 23s (non-voting)
awx-ansible-modules : SUCCESS in 13m 16s
awx-push-new-schema : SKIPPED (non-voting)

softwarefactory-project-zuul · 2020-01-15T19:31:40Z

Build succeeded.

awx-api-lint : SUCCESS in 3m 51s
awx-api : SUCCESS in 9m 11s
awx-ui : SUCCESS in 7m 25s
awx-ui-next : SUCCESS in 7m 58s
awx-swagger : SUCCESS in 10m 55s
awx-detect-schema-change : FAILURE in 9m 23s (non-voting)
awx-ansible-modules : SUCCESS in 4m 41s

softwarefactory-project-zuul · 2020-01-15T19:53:28Z

Build succeeded (gate pipeline).

awx-api-lint : SUCCESS in 4m 35s
awx-api : SUCCESS in 9m 49s
awx-ui : SUCCESS in 4m 34s
awx-ui-next : SUCCESS in 7m 18s
awx-swagger : SUCCESS in 10m 59s
awx-detect-schema-change : FAILURE in 9m 39s (non-voting)
awx-ansible-modules : SUCCESS in 3m 38s
awx-push-new-schema : SUCCESS in 9m 58s (non-voting)

Move only_transmit_kwargs calculation out of thread

ryanpetrello requested review from matburt, AlanCoding, wenottingham, jakemcdermott and mabashian December 4, 2019 18:04

ryanpetrello commented Dec 4, 2019

View reviewed changes

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch 2 times, most recently from 1a3fccd to 2384ccf Compare December 4, 2019 18:11

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 2384ccf to 2db4181 Compare December 4, 2019 18:18

AlanCoding reviewed Dec 4, 2019

View reviewed changes

ryanpetrello changed the title ~~remove computed inventory fields from Host and Group~~ WIP: remove computed inventory fields from Host and Group Dec 4, 2019

AlanCoding reviewed Dec 4, 2019

View reviewed changes

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 2db4181 to 0171f8a Compare December 4, 2019 20:32

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 0171f8a to 057334a Compare December 4, 2019 20:51

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 057334a to 1a53f82 Compare December 4, 2019 22:05

jakemcdermott assigned jakemcdermott and ryanpetrello Dec 5, 2019

kdelee approved these changes Jan 8, 2020

View reviewed changes

jakemcdermott force-pushed the remove-computed-group-and-host-fields branch from af97959 to c2d3357 Compare January 8, 2020 17:19

dsesami approved these changes Jan 8, 2020

View reviewed changes

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from c2d3357 to dfffa87 Compare January 13, 2020 15:09

ryanpetrello and others added 7 commits January 14, 2020 16:37

remove computed inventory fields from Host and Group

568606d

Remove status icons from host nested groups

107d2da

Use related hosts and group counts for delete modal

2bc6521

Remove status icons from group nested groups

ec1c2a8

Remove unused host status service

1220847

Remove dead code

773b976

Remove group status cols

44c0eb8

Remove the status col from any group list that used the now-removed computed fields.

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 588d460 to ed9222a Compare January 14, 2020 21:38

reorder migrations after a rebase

be68a19

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from ed9222a to be68a19 Compare January 14, 2020 21:58

ryanpetrello added the mergeit label Jan 15, 2020

send inv computed tasks *after* commit to avoid a race condition

0f0d9ba

ryanpetrello force-pushed the remove-computed-group-and-host-fields branch from 512244d to 0f0d9ba Compare January 15, 2020 19:15

ryanpetrello mentioned this pull request Jan 15, 2020

Remove two unused parent relationships from JobEvent model #5669

Merged

softwarefactory-project-zuul bot merged commit 6d075b8 into ansible:devel Jan 15, 2020

AlexSCorey pushed a commit to AlexSCorey/awx that referenced this pull request Dec 7, 2021

Merge pull request ansible#5448 from AlanCoding/out_of_thread

6fe5730

Move only_transmit_kwargs calculation out of thread

	select_related = ('created_by', 'modified_by', 'inventory',
	'last_job__job_template', 'last_job_host_summary__job',)

remove computed inventory fields from Host and Group #5448

remove computed inventory fields from Host and Group #5448

Conversation

ryanpetrello commented Dec 4, 2019 • edited Loading

ryanpetrello Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Jan 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanpetrello commented Dec 4, 2019

ryanpetrello commented Dec 4, 2019

ryanpetrello commented Dec 4, 2019

ryanpetrello commented Dec 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Dec 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlanCoding commented Dec 4, 2019

ryanpetrello commented Dec 4, 2019

softwarefactory-project-zuul bot commented Dec 4, 2019

softwarefactory-project-zuul bot commented Dec 4, 2019

softwarefactory-project-zuul bot commented Dec 4, 2019

kdelee left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Jan 8, 2020

dsesami left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Jan 8, 2020

softwarefactory-project-zuul bot commented Jan 13, 2020

softwarefactory-project-zuul bot commented Jan 13, 2020

softwarefactory-project-zuul bot commented Jan 14, 2020

softwarefactory-project-zuul bot commented Jan 14, 2020

softwarefactory-project-zuul bot commented Jan 15, 2020

softwarefactory-project-zuul bot commented Jan 15, 2020

softwarefactory-project-zuul bot commented Jan 15, 2020

softwarefactory-project-zuul bot commented Jan 15, 2020

ryanpetrello commented Dec 4, 2019 •

edited

Loading

ryanpetrello Dec 4, 2019 •

edited

Loading

ryanpetrello Jan 2, 2020 •

edited

Loading

ryanpetrello Dec 4, 2019 •

edited

Loading

ryanpetrello commented Dec 4, 2019 •

edited

Loading