-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
opensearch play fails Exiting script [deploy_logging.sh] #520
Comments
@cumcke any idea what the issue could be ? |
@biohazd Were you deploying into a entirely new cluster or were you updating SAS Viya Monitoring on an existing cluster? The messages you shared look like they were attempting to update an existing deployment of things. |
It is a new cluster. I ran the script a few times. I have tried it on a new cluster as well, and still fails. |
Is there any way to debug it better or get some more verbose error message ? |
Unfortunately, when deploying our project via the Viya 4 Deployment project, we lose some visibility and/access. If the cluster is up and running, you could deploy the monitoring components directly by using our deployment tooling directly. All that you would need is a Linux shell with access to the cluster and a kube config file with full admin access to the cluster. That would probably make it easier to debug this problem. In the meantime, I have some additional questions:
I would recommend you remove any of the logging components already deployed before trying again. To do that, run the uninstall task that corresponds to the cluster_logging deploy task you've been running. Or, alternatively, you should be able to delete the "logging" namespace. That will give you a clean environment. Once you've done that, re-run the deployment task and, if it fails, share the log output you get at that point. That might give us a better idea of what's going on. |
Thanks. I will try that. I’m not using any custom USERDIR and there should be no network restrictions. |
Thnaks, it did deploy and is working, but it does still show those errors. TASK [monitoring : cluster-logging - deploy] ***********************************
|
@biohazd Unfortunately, the messages indicate things did not actually deploy completely successfully in your environment. While some pods may be up and running, the ERROR messages indicate that some of the content (e.g. the pre-built OpenSearch Dashboards, saved queries, etc.) couldn't be loaded. In addition, I suspect the Fluent Bit pods which collect the log messages from the various Kubernetes nodes/pods were not deployed either. So, you will have no log messages to review in OpenSearch Dashboards. The message about "not being able to identify the temporary port used for port-forwarding" is a revealing one. I suspect OpenSearch was not up (and may still not be up) when the script ran. I suspect you are running into resource issues on your cluster. Is it possibly under-sized? We've seen these sorts of messages when that is the case. If you can monitor the cluster during the deployment process, check the OpenSearch pod logs for error messages and/or events indicating there was a problem scheduling the pod onto a node. Another possibility is that the PVCs needed by OpenSearch weren't provisioned for some reason. All of that should be detectable by monitoring the cluster during the deployment process using a tool like OpenLens or even just using kubectl describe pod commands. |
@biohazd I wanted to check in with on the status of this issue. Were you able to get everything deployed and working? |
It does seem to deploy fine, but those errors are still there. |
I have the same problem. I have verified i have sufficient resources in my cluster. Release "es-exporter" does not exist. Installing it now.
INFO Loading Content into OpenSearch I see following resources in my name space NAME READY STATUS RESTARTS AGE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE NAME READY UP-TO-DATE AVAILABLE AGE NAME DESIRED CURRENT READY AGE NAME READY AGE |
I have deployed Sas Viya 4 (2023.05) on AWS using the aws-iac repo.
When deploying "components=cluster-logging" it fails.
The other steps work fine, and it even givs me the Opensearch user and password.
im running the playbook for "components=cluster-logging"
i.e. make sas-deployment prefix=viya4-01 namespace=sas-viya4 components=cluster-logging
TASK [monitoring : cluster-logging - deploy]
stderr_lines:
stdout_lines:
The text was updated successfully, but these errors were encountered: