nixos/druid: init module and package #219942

vsharathchandra · 2023-03-07T07:28:18Z

Description of changes

init apache druid at 29.0.0
init druid module
init druid nixos tests

Druid is a high performance, real-time analytics database that delivers sub-second queries on streaming and batch data at scale and under load.

Things done

illustris · 2023-03-10T08:36:18Z

editorconfig:

nix run nixpkgs#nixpkgs-fmt nixos/tests/druid/with_hdfs.nix

nixos manual:

All options need to be updated to markdown. For example,

diff --git a/nixos/modules/services/cluster/druid/default.nix b/nixos/modules/services/cluster/druid/default.nix
index 62c100ad8a4..a756c6edd07 100644
--- a/nixos/modules/services/cluster/druid/default.nix
+++ b/nixos/modules/services/cluster/druid/default.nix
@@ -4,10 +4,10 @@ let
   cfg = config.services.druid;

   druidServiceOption = serviceName: {
-    enable = mkEnableOption serviceName;
+    enable = mkEnableOption (mdDoc serviceName);
     restartIfChanged = mkOption {
       type = types.bool;
-      description = ''
+      description = mdDoc ''
         Automatically restart the service on config change.
         This can be set to false to defer restarts on clusters running critical applications.
         Please consider the security implications of inadvertently running an older version,
@@ -18,26 +18,26 @@ let
     config = mkOption {
       default = { };
       type = types.attrsOf types.anything;
-      description = "(key=value) Configuration to be written to runtime.properties of the druid ${serviceName}";
+      description = mdDoc "(key=value) Configuration to be written to runtime.properties of the druid ${serviceName}";
     };
-    jdk = mkPackageOption pkgs "JDK" {
+    jdk = mkPackageOptionMD pkgs "JDK" {
       default = [ "jdk8_headless" ];
     };

restricted eval:

replace fetchTarball with fetchurl

I have a few more suggestions, but can you push these changes to see if CI passes first?

nixos/modules/services/cluster/druid/default.nix

nixos/tests/all-tests.nix

nixos/tests/druid/with_hdfs.nix

nixos/modules/services/cluster/druid/default.nix

pkgs/applications/networking/cluster/druid/default.nix

nixos/modules/services/cluster/druid/default.nix

pkgs/applications/networking/cluster/druid/default.nix

nixos-discourse · 2024-03-06T17:10:27Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/3589

nixos/modules/services/cluster/druid/default.nix

pkgs/by-name/dr/druid/package.nix

illustris

Hey, sorry for coming back with more suggestions. This is a big PR, and I keep missing small details. Functionally the PR works fine the way it is now. All remaining suggestions are just conforming to conventions.

pkgs/by-name/dr/druid/package.nix

illustris · 2024-03-22T16:10:17Z

nixos/modules/services/cluster/druid/default.nix

+        "druid.service" = "servicename";
+      };
+    };
+    jdk = mkPackageOptionMD pkgs "JDK" {


Suggested change

jdk = mkPackageOptionMD pkgs "JDK" {

jdk = mkPackageOption pkgs "JDK" {

nixos/modules/services/cluster/druid/default.nix

pkgs/applications/networking/cluster/druid/default.nix

pkgs/by-name/dr/druid/package.nix

illustris · 2024-03-22T16:49:10Z

nixos/modules/services/cluster/druid/default.nix

+      allowedTCPPorts = [ (attrByPath [ "druid.plaintextPort" ] 8083 cfg."${name}".config) ];
+      extraConfig.services.druid.historical.internalConfig."druid.segmentCache.locations" = ''
+        [${(concatMapStringsSep "," (
+          x: "{\"path\":\"" + x.path + "\",\"maxSize\":" + x.maxSize + "}"


Suggested change

x: "{\"path\":\"" + x.path + "\",\"maxSize\":" + x.maxSize + "}"

x: ''{"path":"${x.path}","maxSize":"${x.maxSize}"}''

[optional] get rid of all the escapes

illustris

Thanks!

nixos-discourse · 2024-03-30T09:43:20Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1565

nixos-discourse · 2024-06-15T07:40:51Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1748

nixos-discourse · 2024-07-09T13:16:49Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1806

eclairevoyant

Thanks for the contribution! I've not looked atthe module yet, but here's some feedback for the package itself:

pkgs/by-name/dr/druid/package.nix

changes addressed

pkgs/by-name/dr/druid/package.nix

nixos/modules/services/cluster/druid/default.nix

nixos-discourse · 2024-07-26T17:36:00Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-already-reviewed/2617/1860

pkgs/by-name/dr/druid/package.nix

nixos/modules/services/cluster/druid/default.nix

Aleksanaa · 2024-07-27T17:21:37Z

You can ignore the "escaping with" part for now, it will be temporarily disabled

Edited: now force push to generate a new error report

gador · 2024-07-27T17:32:22Z

Could you format the files with nixfmt?

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/modules/services/cluster/druid/default.nix

+            (cfg.commonTmpDirs ++ tmpDirs);
+        };
+        networking.firewall.allowedTCPPorts =
+          mkIf ((attrByPath [ "openFirewall" ] false serviceOptions))


redundant parentheses

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+            enable = true;
+            package = pkgs.mariadb;
+            initialDatabases = [{ name = "druid"; }];
+            initialScript = pkgs.writeText "mysql-init.sql" (''


redundant parentheses

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+      passthru.override = args': testsForPackage (args // args');
+    };
+  testDruidCluster = { druidPackage, hadoopPackage, ... }:
+    pkgs.testers.nixosTest ({


redundant parentheses

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+            inherit coreSite;
+          };
+        };
+        broker = { pkgs, ... }: {


definition pkgs is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+          };
+
+        };
+        coordinator = { pkgs, ... }: {


definition pkgs is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+            inherit coreSite;
+          };
+        };
+        mm = { pkgs, ... }: {


definition pkgs is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+          services.zookeeper.enable = true;
+          networking.firewall.allowedTCPPorts = [ 2181 ];
+        };
+        namenode = { pkgs, ... }: {


definition pkgs is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+            inherit coreSite;
+          };
+        };
+        overlord = { pkgs, ... }: {


definition pkgs is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+    </Configuration>
+  '';
+  log4j = pkgs.writeText "log4j2.xml" log4jConfig;
+  name = "hadoop-hdfs";


definition name is not used

nixf-tidy-review · 2024-07-28T06:25:34Z

nixos/tests/druid/default.nix

+  log4j = pkgs.writeText "log4j2.xml" log4jConfig;
+  name = "hadoop-hdfs";
+  coreSite = { "fs.defaultFS" = "hdfs://namenode:8020"; };
+  package = pkgs.hadoop_3_2;


definition package is not used

nixf-tidy-review

✅ Looks good!

gador · 2024-07-28T08:30:32Z

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).

Result of nixpkgs-review pr 219942 run on x86_64-linux 1

2 packages blacklisted:

nixos-install-tools
tests.nixos-functions.nixos-test

2 packages built:

druid
nixpkgs-manual

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

nixpkgs-manual:

Package is missing maintainers.
If the package is using runCommand please make sure to inherit or list one or more maintainers.

gador · 2024-07-28T09:01:50Z

nixosTests pass as well, all LGTM. Thanks for all the work you put in!

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Mar 7, 2023

ofborg bot added 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Mar 17, 2023

illustris suggested changes Mar 17, 2023

View reviewed changes

illustris reviewed Mar 21, 2023

View reviewed changes

pkgs/applications/networking/cluster/druid/default.nix Outdated Show resolved Hide resolved

Janik-Haag added the 12. first-time contribution This PR is the author's first one; please be gentle! label Jun 13, 2023

h7x4 added 8.has: module (new) This PR adds a module in `nixos/` 8.has: tests This PR has tests labels Oct 22, 2023

vsharathchandra marked this pull request as draft January 21, 2024 11:36

fsnkty suggested changes Feb 11, 2024

View reviewed changes

nixos/modules/services/cluster/druid/default.nix Outdated Show resolved Hide resolved

nixos/modules/services/cluster/druid/default.nix Outdated Show resolved Hide resolved

pkgs/applications/networking/cluster/druid/default.nix Outdated Show resolved Hide resolved

vsharathchandra force-pushed the druid_nix_module branch from 73dfbf5 to 7ef389e Compare February 29, 2024 17:24

vsharathchandra marked this pull request as ready for review March 6, 2024 17:02

illustris suggested changes Mar 11, 2024

View reviewed changes

vsharathchandra force-pushed the druid_nix_module branch from 9ea5e06 to 7308bf8 Compare March 14, 2024 03:57

mantainers: add vsharathchandra

0d41cda

vsharathchandra force-pushed the druid_nix_module branch from 7308bf8 to 4de6ff1 Compare March 14, 2024 04:07

illustris approved these changes Mar 17, 2024

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Mar 20, 2024

illustris reviewed Mar 22, 2024

View reviewed changes

vsharathchandra force-pushed the druid_nix_module branch 2 times, most recently from 41b5cb3 to 270f37a Compare March 23, 2024 08:07

wegank removed the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Mar 25, 2024

illustris approved these changes Mar 25, 2024

View reviewed changes

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Mar 28, 2024

eclairevoyant previously requested changes Jul 10, 2024

View reviewed changes

vsharathchandra force-pushed the druid_nix_module branch from 270f37a to a716833 Compare July 10, 2024 19:01

github-actions bot added the 8.has: maintainer-list (update) This PR changes `maintainers/maintainer-list.nix` label Jul 10, 2024

vsharathchandra force-pushed the druid_nix_module branch from a716833 to bd9a517 Compare July 10, 2024 19:09

SuperSandro2000 reviewed Jul 16, 2024

View reviewed changes

vsharathchandra force-pushed the druid_nix_module branch from bd9a517 to eac60ad Compare July 18, 2024 07:22

ofborg bot added the 11.by: package-maintainer This PR was created by the maintainer of the package it changes label Jul 18, 2024

vsharathchandra requested a review from SuperSandro2000 July 18, 2024 11:36

vsharathchandra mentioned this pull request Jul 21, 2024

Packaging request: Apache Druid #192752

Closed

gador reviewed Jul 26, 2024

View reviewed changes

vsharathchandra force-pushed the druid_nix_module branch from eac60ad to 43fd5ad Compare July 27, 2024 16:47

vsharathchandra force-pushed the druid_nix_module branch 3 times, most recently from e77cdab to c363092 Compare July 28, 2024 06:17

nixf-tidy-review reviewed Jul 28, 2024

View reviewed changes

sharath.v added 2 commits July 28, 2024 06:51

druid: init at 30.0.0

52eab37

nixos/druid: init module

4bfc173

vsharathchandra force-pushed the druid_nix_module branch from c363092 to 888656f Compare July 28, 2024 07:08

nixf-tidy-review approved these changes Jul 28, 2024

View reviewed changes

druid: add passthru test

9943804

vsharathchandra force-pushed the druid_nix_module branch from 888656f to 9943804 Compare July 28, 2024 07:20

gador merged commit 2c5b304 into NixOS:master Jul 28, 2024
27 of 29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/druid: init module and package #219942

nixos/druid: init module and package #219942

vsharathchandra commented Mar 7, 2023 •

edited

Loading

illustris commented Mar 10, 2023 •

edited

Loading

nixos-discourse commented Mar 6, 2024

illustris left a comment

illustris Mar 22, 2024

illustris Mar 22, 2024

illustris left a comment

nixos-discourse commented Mar 30, 2024

nixos-discourse commented Jun 15, 2024

nixos-discourse commented Jul 9, 2024

eclairevoyant left a comment

nixos-discourse commented Jul 26, 2024

Aleksanaa commented Jul 27, 2024 •

edited

Loading

gador commented Jul 27, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review Jul 28, 2024

nixf-tidy-review left a comment

gador commented Jul 28, 2024

gador commented Jul 28, 2024

	jdk = mkPackageOptionMD pkgs "JDK" {
	jdk = mkPackageOption pkgs "JDK" {

	x: "{\"path\":\"" + x.path + "\",\"maxSize\":" + x.maxSize + "}"
	x: ''{"path":"${x.path}","maxSize":"${x.maxSize}"}''

nixos/druid: init module and package #219942

nixos/druid: init module and package #219942

Conversation

vsharathchandra commented Mar 7, 2023 • edited Loading

Description of changes

Things done

illustris commented Mar 10, 2023 • edited Loading

editorconfig:

nixos manual:

restricted eval:

nixos-discourse commented Mar 6, 2024

illustris left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illustris left a comment

Choose a reason for hiding this comment

nixos-discourse commented Mar 30, 2024

nixos-discourse commented Jun 15, 2024

nixos-discourse commented Jul 9, 2024

eclairevoyant left a comment

Choose a reason for hiding this comment

nixos-discourse commented Jul 26, 2024

Aleksanaa commented Jul 27, 2024 • edited Loading

gador commented Jul 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixf-tidy-review left a comment

Choose a reason for hiding this comment

gador commented Jul 28, 2024

gador commented Jul 28, 2024

vsharathchandra commented Mar 7, 2023 •

edited

Loading

illustris commented Mar 10, 2023 •

edited

Loading

Aleksanaa commented Jul 27, 2024 •

edited

Loading