Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

configuration/documentation for using another OpenSearch cluster rather than local docker #10

Closed
mmguero opened this issue Sep 9, 2020 · 2 comments
Closed

configuration/documentation for using another OpenSearch cluster rather than local docker #10

mmguero opened this issue Sep 9, 2020 · 2 comments
Assignees
@mmguero
Labels
doc Relating to Malcolm documentation docker Relating to docker and docker-compose as used by Malcolm opensearch Relating to Malcolm's use of OpenSearch

Comments

@mmguero
Copy link
Collaborator

mmguero commented Sep 9, 2020 •
edited
Loading

From Malcolm created by mmguero: cisagov#16

In some cases it will make more sense for people to use their own elasticsearch OpenSearch deployment rather than Malcolm's dockerized one. For example, in order to do a larger scale-out implementation with multiple data notes, etc.

I'm going to trying things out for this in a personal branch dedicated to this topic. Specifying the connection parameters (IP/port) should be pretty easy once things are normalized into a single source of environment variables in the compose file. I think the trick will be how to specify authentication information for all of the clients. This will include:

  • arkime
  • logstash
  • dashboards
  • dashboards-helper
  • pcap-monitor
  • api
@mmguero mmguero added doc Relating to Malcolm documentation docker Relating to docker and docker-compose as used by Malcolm opensearch Relating to Malcolm's use of OpenSearch labels Sep 9, 2020
@mmguero mmguero mentioned this issue Sep 9, 2020
Closed
mmguero pushed a commit to mmguero-dev/Malcolm that referenced this issue Apr 12, 2022
@n8hacks
Merge pull request idaholab#10 from aglad-eng/suricata_integration
a670e92 
Integrate Suricata into Main
@mmguero mmguero added this to Malcolm May 10, 2022
@mmguero mmguero moved this to Todo (spike) in Malcolm May 10, 2022
@mmguero mmguero changed the title Improve documentation on how to use another ES cluster rather than local docker configuration/documentation for using another OpenSearch cluster rather than local docker Aug 16, 2022
@mmguero
Copy link
Collaborator Author

mmguero commented Aug 16, 2022

While I'm experimenting with this, I've got a little test example that brings up a 3 node cluster (one cluster manager, two data nodes) with SSL enabled with an LDAP backend. Not anything you'd want to use for production, but will be useful for testing.

mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Aug 22, 2022
@mmguero
configuration/documentation for using another OpenSearch cluster rath…
7e41632 
…er than local docker (idaholab#10)

Squashed commit of the following:

commit 6f0cce6
Author: Seth Grover <[email protected]>
Date:   Mon Aug 22 14:33:19 2022 -0600

    README updates

commit b70154b
Author: Seth Grover <[email protected]>
Date:   Mon Aug 22 13:40:20 2022 -0600

    tweaks to install.py for opensearch URL stuff

commit 9674260
Author: Seth Grover <[email protected]>
Date:   Mon Aug 22 11:49:41 2022 -0600

    Quiet message from arkime

commit a67b1bd
Author: SG <[email protected]>
Date:   Fri Aug 19 13:44:14 2022 -0600

    proxy opensearch creds through dashboards for importing

commit 051ec1a
Author: SG <[email protected]>
Date:   Fri Aug 19 13:27:01 2022 -0600

    import requests.auth

commit 3c65c1a
Author: SG <[email protected]>
Date:   Fri Aug 19 13:25:49 2022 -0600

    bump to v6.3.0

commit 4248e40
Author: SG <[email protected]>
Date:   Fri Aug 19 13:22:39 2022 -0600

    proxy opensearch creds through dashboards for importing

commit e7677dc
Author: SG <[email protected]>
Date:   Fri Aug 19 13:17:01 2022 -0600

    don't fail on some dashboards setup

commit b9dd113
Author: SG <[email protected]>
Date:   Fri Aug 19 13:01:06 2022 -0600

    set permissions on files containing secrets

commit 9fe37b1
Author: SG <[email protected]>
Date:   Fri Aug 19 12:43:58 2022 -0600

    fix logstash startup

commit 018a146
Author: SG <[email protected]>
Date:   Fri Aug 19 12:42:56 2022 -0600

    fix arkime and dashbaords startup

commit e679a31
Author: SG <[email protected]>
Date:   Fri Aug 19 12:23:13 2022 -0600

    work in progress specifying the opensearch connection info

commit 4808183
Author: SG <[email protected]>
Date:   Fri Aug 19 12:22:57 2022 -0600

    work in progress specifying the opensearch connection info

commit 21efd67
Author: SG <[email protected]>
Date:   Fri Aug 19 08:11:58 2022 -0600

    fix arkime startup

commit 58d07b5
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 16:54:16 2022 -0600

    fix build

commit 2c08399
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 16:13:02 2022 -0600

    do primary and secondary servers the same way

commit 2a40400
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 15:37:33 2022 -0600

    comments

commit 9b45c5b
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 15:34:47 2022 -0600

    work in progress on generalizing opensearch connection params

commit 303979c
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 15:27:20 2022 -0600

    work in progress on generalizing opensearch connection params

commit 83ff378
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 14:31:23 2022 -0600

    work in progress on generalizing opensearch connection params

commit e7f2227
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 14:04:38 2022 -0600

    work in progress on generalizing opensearch connection params

commit fe962ca
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 12:41:31 2022 -0600

    work in progress on generalizing opensearch connection params

commit 175163d
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 11:48:22 2022 -0600

    work in progress

commit 7d6973d
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 11:43:41 2022 -0600

    work in progress

commit 3dfcfb2
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 10:48:54 2022 -0600

    added a whole bunch of TODOs (search for TODO: handle)

commit df238a5
Author: Seth Grover <[email protected]>
Date:   Thu Aug 18 10:41:28 2022 -0600

    working on specifying creds for remote opensearch differently
@mmguero mmguero self-assigned this Aug 22, 2022
@mmguero mmguero moved this from Todo (spike) to Todo (develop) in Malcolm Aug 22, 2022
@mmguero mmguero moved this from Todo (develop) to Done in Malcolm Aug 22, 2022
@mmguero
Copy link
Collaborator Author

mmguero commented Aug 22, 2022

Should be working now for v6.3.0, including documentation. More testing to come before release.

@mmguero mmguero closed this as completed Aug 22, 2022
@mmguero mmguero moved this from Done to Released in Malcolm Sep 7, 2022
@mmguero mmguero mentioned this issue Jan 15, 2023
Closed
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 17, 2024
@mmguero
bump jinja2 from v3.1.2 to v3.1.3 (dependabot alert: Jinja vulnerable…
b6a3511 
… to HTML attribute injection when passing user input as keys to xmlattr filter idaholab#10)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
doc Relating to Malcolm documentation docker Relating to docker and docker-compose as used by Malcolm opensearch Relating to Malcolm's use of OpenSearch
Projects
Malcolm
Status: Released
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mmguero