v3.0.1 development #161
Merged
v3.0.1 development #161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Change base for Elasticsearch and Kibana Docker images (version 7.6.2) from Elastic.co to Open Distro for Elastic (based on Elastic 7.10.0); see idaholab#15. This is a major change which **breaks backwards compatibility** for several features (listed below). If you are using these features, you will need to back up the data and/or configuration associated with them and migrate them manually to the new tools. No automatic migration or upgrade of these features is performed. It's recommended that you re-run `install.py --configure` (see [System configuration and tuning](https://github.com/cisagov/Malcolm#ConfigAndTuning)) prior to running Malcolm v3.0.0. - Kibana [comments](https://github.com/gwintzer/kibana-comments-app-plugin) replaced with [Notebooks](https://opendistro.github.io/for-elasticsearch-docs/docs/notebooks/) - Kibana [elastalert](https://github.com/nsano-rururu/elastalert-kibana-plugin) plugin replaced with [Alerting](https://opendistro.github.io/for-elasticsearch-docs/docs/alerting/) plugin - Elasticsearch [curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) replaced with [Index Management](https://opendistro.github.io/for-elasticsearch-docs/docs/ism/) plugin - The third-party [Sankey visualization plugin](https://github.com/mmguero-dev/kbn_sankey_vis] has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see uniberg/kbn_sankey_vis#15) - The third-party [Kibana drill-down plugin](https://github.com/mmguero-dev/kibana-plugin-drilldownmenu/) providing Kibana-to-Moloch pivoting has been temporarily removed due to compatibility issues, although it is planned to be reintegrated in a Malcolm point release in the near future (see goodlabs-studio/kibana-plugin-drilldownmenu#5) - In addition to those replacements, the Real Time Anomaly Detection feature is now available: - [Real Time Anomaly Detection in Open Distro for Elasticsearch](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/real-time-anomaly-detection-in-open-distro-for-elasticsearch/) blog announcement - [Anomaly Detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/) documentation and source code for [Elasticsearch](https://github.com/opendistro-for-elasticsearch/anomaly-detection) and [Kibana](https://github.com/opendistro-for-elasticsearch/anomaly-detection-kibana-plugin) components - [Random Cut Forests writeup](https://opendistro.github.io/for-elasticsearch/blog/odfe-updates/2019/11/random-cut-forests/) - Malcolm startup time (especially the Logstash container) has been reduced drastically - Improvements to Malcolm's prebuilt Kibana dashboards - Improvements to build scripts - Minor tweaks and bugfixes for ISO-installed environments for Malcolm and Hedgehog Linux - Minor other bug fixes and performance improvements - Version bump - Yara [v4.0.5](https://github.com/VirusTotal/yara/releases/tag/v4.0.5)
…gstash containers
…gstash containers
…index management template
…mmunity fixes the packages
…ullcontext not being available. Also, fix suggestion for yum-based distributions to install python 3 requests via pip.
- kernel to 5.10 - update /etc/os-release with Hedgehog Linux version information for Malcolm and Hedgehog ISOs - use TMPDIR as MalcolmDir/.tmp for docker-compose to avoid issues with /tmp having noexec bit set
… unless configured to do so
…omething that elasticsearch.js (used by arkime viewer) sets and I'm not sure it's worth the effort at the moment
mmguero
referenced
this pull request
in idaholab/Malcolm
Mar 3, 2021
Malcolm v3.0.1 - Version bumps - Open Distro for Elastic ([v1.13.0](https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/main/release-notes/opendistro-for-elasticsearch-release-notes-1.13.0.md)), which adds the following functionality over the previous release - [Reporting](https://opendistro.github.io/for-elasticsearch-docs/docs/kibana/reporting/) - [Historical data anomaly detection](https://opendistro.github.io/for-elasticsearch-docs/docs/ad/#step-6-analyze-historical-data) - ODFE v1.13.0 is based on the Elastic components 7.10.2 ([elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-7.10.2.html), [kibana](https://www.elastic.co/guide/en/kibana/current/release-notes-7.10.2.html), [logstash](https://www.elastic.co/guide/en/logstash/current/logstash-7-10-2.html), [beats](https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-7.10.2.html)) - Zeek [3.0.13](https://github.com/zeek/zeek/releases/tag/v3.0.13) - NGINX [1.19.7](https://nginx.org/en/CHANGES) - Alpine Linux [3.13](https://alpinelinux.org/posts/Alpine-3.13.0-released.html) Docker base layer - docker-compose [1.28.5](https://docs.docker.com/compose/release-notes/) in Malcolm installable ISO version - Restored the [sankey visualization](https://github.com/uniberg/kbn_sankey_vis) which was temporarily removed in Malcolm v3.0.0 (although there are still a few minor cosmetic [issues](uniberg/kbn_sankey_vis#15) with it) - Removed port 8443 for upload (now just use /upload over the regular HTTPS port) - Fixed issue with ODFE email alerts not being able to use self-signed SMTP certificates by importing CA certs in `nginx/ca-trust` into the JDK trust store for Elasticsearch and Logstash (see #37) - Don't expose the Elasticsearch 9200 by default, it must now be explicitly be enabled during `install.py -c` (see #38) - For ISO-installed versions of Malcolm and Hedgehog Linux, populate `/etc/os-release` with information about the build/release version - Populate user-agent for a few clients ([Arkime's moloch-capture](arkime/arkime#1615), some hedgehog test connection processes) so they're not just sent as blank when communicating with Malcolm - Added Arkime link to Kibana dashboards' navigation pane - Fix some issues in control script with older python3 versions (3.6.x) with `contextlib.nullcontext` not being available - Fix suggestion for yum-based distributions to install python 3 requests via pip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Malcolm v3.0.1
nginx/ca-trustinto the JDK trust store for Elasticsearch and Logstash (see add CA certificate to elasticsearch jdk trust store for using self-signed SMTP servers idaholab/Malcolm#37)install.py -c(see don't automatically expose elasticsearch port (9200) unless explicitly configured to do so idaholab/Malcolm#38)/etc/os-releasewith information about the build/release versioncontextlib.nullcontextnot being available