-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubernetes deployment - opensearch #170
Comments
Here's the first issue as I expected we'd have:
|
I think now with a combination of |
Seems to be working, at least basically, closing for now. |
Malcolm v23.05.0 Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes. IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`. v23.04.0...v23.05.0 * New features - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (#190) - [End-to-end Malcolm and Hedgehog Linux ISO Installation]((https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (#181) - support Malcolm deployment with Kubernetes (#149) + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes) + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [[email protected]](mailto:[email protected]?subject=Malcolm). + contributing issues: * inotify issue (#168) * htadmin/nginx and htpasswd (#169) * opensearch (#170) * uploading large PCAP files (#171) * script consolidation (#172) * documentation (#173) * user-defined persistent volumes (#174) * opensearch keystore (#176) * expose other TCP services (#183) * provide with filebeat access to nginx access and error logs (#186) * use Secrets for some environment variables instead of ConfigMaps (#189) * Enhancements and fixes * remove name-map-ui container (#165) in favor of using NetBox for asset identification * Python script refactoring, consolidation and cleanup * standardization of Docker container entrypoints * create `./scripts/configure` alias for `./scripts/install.py --configure` * Component version updates - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72) - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0) - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/) - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0) - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22) - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24) - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/) - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0) - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
Malcolm v23.05.0 Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes. IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`. v23.04.0...v23.05.0 * New features - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (idaholab#190) - [End-to-end Malcolm and Hedgehog Linux ISO Installation((cisagov://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (idaholab#181) - support Malcolm deployment with Kubernetes (idaholab#149) + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes) + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [[email protected]](mailto:[email protected]?subject=Malcolm). + contributing issues: * inotify issue (idaholab#168) * htadmin/nginx and htpasswd (idaholab#169) * opensearch (idaholab#170) * uploading large PCAP files (idaholab#171) * script consolidation (idaholab#172) * documentation (idaholab#173) * user-defined persistent volumes (idaholab#174) * opensearch keystore (idaholab#176) * expose other TCP services (idaholab#183) * provide with filebeat access to nginx access and error logs (idaholab#186) * use Secrets for some environment variables instead of ConfigMaps (idaholab#189) * Enhancements and fixes * remove name-map-ui container (idaholab#165) in favor of using NetBox for asset identification * Python script refactoring, consolidation and cleanup * standardization of Docker container entrypoints * create `./scripts/configure` alias for `./scripts/install.py --configure` * Component version updates - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72) - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0) - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/) - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0) - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22) - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24) - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/) - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0) - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)
Sub-issue of #149.
Currently we're testing with an external OpenSearch cluster. We need to figure out what it'll take to get it running nicely in kubernetes. Getting it turned on should be easy, but getting the underlying system tuned is sort of a mystery for me with Kubernetes (see elasticsearch and opensearch docs). We need to figure that out.
For a very simply, non-scalable implementation we can mimic the docker container. For scalable instances, we'll let people manage their own clusters and interact with the same way we do today with remote instances support we added a few releases ago.
Scalable instances can be deployed on Kubernetes with the user of Helm charts and/or the OpenSearch Kubernetes Operator repo. See documentation here(https://opensearch.org/docs/2.0/clients/k8s-operator/)
The text was updated successfully, but these errors were encountered: