Subsequent puppet runs fail on unauthenticated replicasets

[stevenpost]

Module version: current master
MongoDB version: 6.0.12

The current provider implementation fails on subsequent runs with the message:

Notice: /Stage[main]/Mongodb::Replset/Mongodb_replset[mongo06rs0]/ensure: created (corrective)
Debug: Checking for dead and alive members
Debug: Checking replicaset member llb-mongo06n01.sfpd.fgov.be:27017 ...
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Host llb-mongo06n01.sfpd.fgov.be:27017 is available for replset mongo06rs0
Debug: Alive members: [{"host"=>"llb-mongo06n01.sfpd.fgov.be:27017"}]
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(db.isMaster())'
Debug: Checking for replset mongo06rs0 changes
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(db.isMaster())'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("81dc9fe4-d9a4-4157-92ae-9cbe82cad214") }, $clusterTime: { clusterTime: Timestamp(1712233467, 1), signature: { hash: BinData(0, E7676032F00C6C300EFBA5CF676601ECB1EFC994), keyId: 7353983806124064775 } }, $db: "admin" }' Retry: '1'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("e368fa68-75e0-456a-ae23-da700fd8c1ec") }, $clusterTime: { clusterTime: Timestamp(1712233467, 1), signature: { hash: BinData(0, E7676032F00C6C300EFBA5CF676601ECB1EFC994), keyId: 7353983806124064775 } }, $db: "admin" }' Retry: '2'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("706cb4b7-1a37-4e23-a124-0289969ee6ea") }, $clusterTime: { clusterTime: Timestamp(1712233477, 1), signature: { hash: BinData(0, 04FBDEAE05E8167D5D9C12B663A8D4626D812823), keyId: 7353983806124064775 } }, $db: "admin" }' Retry: '3'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())'
Debug: Got an exception: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.config()), with: Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("479b85e5-db6c-43bd-bba2-87210bf71e00") }, $clusterTime: { clusterTime: Timestamp(1712233477, 1), signature: { hash: BinData(0, 04FBDEAE05E8167D5D9C12B663A8D4626D812823), keyId: 7353983806124064775 } }, $db: "admin" }
Error: /Stage[main]/Mongodb::Replset/Mongodb_replset[mongo06rs0]: Could not evaluate: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.config()), with: Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.config())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("479b85e5-db6c-43bd-bba2-87210bf71e00") }, $clusterTime: { clusterTime: Timestamp(1712233477, 1), signature: { hash: BinData(0, 04FBDEAE05E8167D5D9C12B663A8D4626D812823), keyId: 7353983806124064775 } }, $db: "admin" }

The self.instances method fails to correctly fill up the current resources:

Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("7d0583e4-f4cb-43c7-a9ca-4929369fcf42") }, $clusterTime: { clusterTime: Timestamp(1712233447, 1), signature: { hash: BinData(0, 35D87CBEF2591035C0CD26C3B01A47C5B7450E83), keyId: 7353983806124064775 } }, $db: "admin" }' Retry: '1'

[stevenpost]

I observe the same failure using MongoDB 4.4.27.

[stevenpost]

Running on 3 nodes sequentially results in 3 different replicasets, each with the same name and a single member.

Instead of doing a single run on every node, expanding the set with every run, I now tried setting up the nodes first , then creating the set with all 3 nodes at once. This results in only the one where the set is initiated as the single member of the set.

Detection of the current set is clearly not working as intended:

Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())'
Debug: Got an exception: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf()), with: Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.conf())' returned 1: MongoServerError: not authorized on admin to execute command { replSetGetConfig: 1, lsid: { id: UUID("2974a240-7c36-4cf9-a28d-fd164e6e1184") }, $db: "admin" }
Notice: /Stage[main]/Mongodb::Replset/Mongodb_replset[mongo06rs0]/ensure: created (corrective)

Since no set actually exists, this results in the provider accidentally doing the right thing and initiates the set. However the current detection of dead and alive members is also wrong.

Debug: Checking for dead and alive members
Debug: Checking replicaset member mongo06n01:27017 ...
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '1'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '2'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received' Retry: '3'
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Got an exception: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.status()), with: Execution of '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: no replset config has been received
Debug: Mongo v4 rs.status() RS not initialized output
Debug: Checking replicaset member mongo06n02:27017 ...
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '1'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '2'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '3'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Got an exception: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.status()), with: Execution of '/bin/mongosh admin --quiet --host mongo06n02:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication
Warning: Can't connect to replicaset member mongo06n02:27017.
Debug: Checking replicaset member mongo06n03:27017 ...
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '1'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '2'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Request failed: 'Execution of '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication' Retry: '3'
Debug: Executing: '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())'
Debug: Got an exception: Could not evaluate MongoDB shell command: load('/root/.mongoshrc.js'); EJSON.stringify(rs.status()), with: Execution of '/bin/mongosh admin --quiet --host mongo06n03:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(rs.status())' returned 1: MongoServerError: command replSetGetStatus requires authentication
Warning: Can't connect to replicaset member mongo06n03:27017.
Debug: Alive members: [{"host"=>"mongo06n01:27017"}]
Debug: Dead members: [{"host"=>"mongo06n02:27017"}, {"host"=>"mongo06n03:27017"}]
Debug: Executing: '/bin/mongosh admin --quiet --host 127.0.0.1:27017 --tls --tlsCertificateKeyFile /etc/pki/tls/private/mongodb.pem --tlsCAFile /etc/pki/tls/certs/ca-mongodb.pem --eval load('/root/.mongoshrc.js'); EJSON.stringify(db.isMaster())'
Debug: Initializing the replset mongo06rs0
Debug: Starting replset config is "{\"_id\":\"mongo06rs0\",\"members\":[{\"host\":\"mongo06n01:27017\",\"_id\":0}],\"settings\":{}}"

This in turn again creates a set with a single member.

[stevenpost]

This seems to be caused by custom configuration:

class { 'mongodb::server':
  auth => true,
  [...]
  config_data => {
    security.clusterAuthMode => 'x509',
  },
}

The resulting config file:

#mongodb.conf - generated from Puppet

systemLog:
  path: /var/log/mongodb/mongod.log
  destination: file
  logAppend: true
  logRotate: reopen
  quiet: false


storage:
  dbPath: /var/lib/mongodb


security:
  authorization: disabled

net:
  bindIp:  127.0.0.1,10.201.208.140
  port: 27017
  tls:
    mode: preferTLS
    certificateKeyFile: /etc/pki/tls/private/mongodb.pem
    CAFile: /etc/pki/tls/certs/ca-mongodb.pem

replication:
    replSetName: mongo06rs0




security.clusterAuthMode: x509
net.tls.allowConnectionsWithoutCertificates: true

Problem was entirely on my side.