revert wait_for_splunk_instance.yml

[jmeixensperger]

Allows wait_for_splunk_instance.yml to target remote hosts instead of only localhost.

Using the previous uri module also always guarantees a response.status value.

[jmeixensperger]

Verified that this change works on an S3 SVA (3 search heads w/ shc, 1 indexer, 1 deployer)

[jmeixensperger]

@jonathan-vega-splunk I am still seeing the occasional error on deployers when the shc captain is not ready yet (see #824). This seems to occur when the shc captain is in the middle of restarting, however it is NOT caused by the wait_for_splunk_instance.yml error that this PR aims to resolve. We may want to take a look at increasing the retry window for deployers (the retry count and/or the retry delay).

[hendolim]

Can we keep using splunk_api to ensure UDS support. And just fix the url param to accommodate remote instance?

[jmeixensperger]

Can we keep using splunk_api to ensure UDS support. And just fix the url param to accommodate remote instance?

There are 2 major reasons I didn't use splunk_api:

1. This task always targets a remote host, and it is impossible to target remote UDS endpoints. This task is also never called against forwarders.
2. When targeting remote hosts, splunk_api can break the retry functionality of tasks with statements like changed when: response.status_code == 200 or until: response.status_code == 200. If the target host is unreachable, the first attempt/response does not have a status_code, triggers an ansible failure, and prevents all further retries.

[hendolim]

Can we keep using splunk_api to ensure UDS support. And just fix the url param to accommodate remote instance?

There are 2 major reasons I didn't use splunk_api:

1. This task always targets a remote host, and it is impossible to target remote UDS endpoints. This task is also never called against forwarders.
2. When targeting remote hosts, splunk_api can break the retry functionality of tasks with statements like changed when: response.status_code == 200 or until: response.status_code == 200. If the target host is unreachable, the first attempt/response does not have a status_code, triggers an ansible failure, and prevents all further retries.

Good callout on 1. I totally forgot about that. When I was debugging this, it appeared as if it is waiting for itself to come up rather than a remote instance, so I was under the impression that the task applies generically for both local and remote host. We might need to clarify the task name or put a comment