v0.14.4 armv7 - step-ca - Error opening database of Type badger with source /root/.step/db: error opening Badger database: Unable to mmap RDWR log file: invalid argument

[tuxnice]

Subject of the issue

The command "step-ca $(step path)/config/ca.json" runs into following error:
Error opening database of Type badger with source /root/.step/db: error opening Badger database: Unable to mmap RDWR log file: invalid argument

Your environment

• OS - Raspbian Buster 10.4

• Version - step/step-ca v0.14.4 armv7

• Installed packages
  gccgo/stable 4:8.3.0-1+rpi2 armhf
  golang-go/stable,now 2:1.11~1+b6 armhf

Steps to reproduce

~ # step ca init (runs without issues)
~ # step-ca $(step path)/config/ca.json
Error opening database of Type badger with source /root/.step/db: error opening Badger database: Unable to mmap RDWR log file: invalid argument

Expected behaviour

Serving HTTPS on 127.0.0.1:8443 ...

Actual behaviour

Can not run step-ca

Additional context

Maybe could be fixed with:
https://github.com/dgraph-io/badger#memory-usage

[dopey]

Hey @tuxnice, thanks for opening the issue.

This is sort of hidden in the documentation, but have you tried setting "badgerValueLogLoadingMode": "FileIO" (https://github.com/smallstep/certificates/blob/master/docs/database.md#badger) in the ca.json as part of the db config?

[tuxnice]

Hello dopey,

yes I tried it, but I got the same error. Here is the section of my ca.json:

   "db": {
      "type": "badger",
      "dataSource": "/root/.step/db",
      "badgerFileLoadingMode": "",
      "badgerValueLogLoadingMode": "FileIO"
   }

thanks in advance,
tuxnice

[alexpilotti]

I'm hitting the same issue running step-ca in an Alpine container on a Raspberry Pi 4

[dopey]

Can you try setting type to badgerV2 please?

Do an rm -rf /root/.step/db, and then restart the CA using badgerV2. Let me know if that still doesn't help.

[alexpilotti]

There's a Badger issue for this: ipfs/kubo#5305

The suggested solution is to reduce the vLogFileSize, e.g: "vlogFileSize": "100 M", so this needs to be available similarly in ca.json

[alexpilotti]

@dopey badgerV2 works for me, thanks!

[dopey]

Awesome! I'll update the documentation to let users know to use badgerV2 along with valueLogLoadingMode:FileIO. Thanks for confirming!

[tuxnice]

@dopey
This does not helped for me, here is the error:

~ # step-ca $(step path)/config/ca.json
badger 2020/05/28 19:59:29 INFO: All 0 tables opened in 0s
badger 2020/05/28 19:59:29 INFO: Replaying file id: 0 at offset: 0
badger 2020/05/28 19:59:29 INFO: Replay took: 10.208µs
Error opening database of Type badgerV2 with source /root/.step/db: error opening Badger database: During db.vlog.open: Map log file. Path=/root/.step/db/000000.vlog. Error=cannot allocate memory

I also added "vlogFileSize": "100 M" without success with the same error.

[dopey]

@tuxnice would you mind posting the entire db section of your ca.json here please? And just to verify, you did rm -rf your db directory?

[tuxnice]

Sure, I did. Here it is the db section:

   "db": {
      "type": "badgerV2",
      "dataSource": "/root/.step/db",
      "badgerFileLoadingMode": "",
      "badgerValueLogLoadingMode": "FileIO",
      "vlogFileSize": "100 M"
   },

Error with the same settings above, with "type": "badgerV2":

root@raspberrypi:~ # rm -rf .step/db/
root@raspberrypi:~ # step-ca $(step path)/config/ca.json
badger 2020/05/28 20:12:05 INFO: All 0 tables opened in 0s
Error opening database of Type badgerV2 with source /root/.step/db: error opening Badger database: During db.vlog.open: Error while creating log file in valueLog.open: Mmap value log file. Path=/root/.step/db/000000.vlog. Error=cannot allocate memory

With "type": "badger":

root@raspberrypi:~ # rm -rf .step/db/
root@raspberrypi:~ # step-ca $(step path)/config/ca.json
Error opening database of Type badger with source /root/.step/db: error opening Badger database: Unable to mmap RDWR log file: invalid argument

[dopey]

@tuxnice I'll cut a release later this afternoon with logFileSize hardcoded to "100 M". If you could test that out and lemme know if it works, that would be great. Assuming it works, we'll have to surface that option in the ca.json. Currently the only option we surface is the valueLogLoadingMode.

[tuxnice]

Yes, I'll do. But right now it looks like I have time to test it on the weekend at the earliest.

[dopey]

@tuxnice Now that I'm looking at the actual code vs. the documentation I think the documentation might be wrong. The name of the attribute should be badgerFileLoadingMode (not badgerValueLogLoadingMode).

When you have the chance can you try setting your database configuration to:

"db": {
      "type": "badgerV2",
      "dataSource": "/root/.step/db",
      "badgerFileLoadingMode": "FileIO",
   },

If that doesn't fix the issue, then I did cut a release (https://github.com/smallstep/certificates/releases/tag/v0.14.5-rc.2.100MB.badgerV2) that hardcodes the vlogFileSize to 100MB. But I would try just changing the db config to the one I pasted before trying the release candidate. Let me know.

I just updated the documentation.

[tuxnice]

Thank you, "badgerFileLoadingMode": "FileIO" was fixing the problem. Also with both versions: badger and badgerV2. Is there a reason why not using badgerV2 as default?

[dopey]

Awesome! Glad that worked. My bad on the documentation snafu.

The reason is that most users are using badgerV1 at this time (we assume, since we don't really have a good way of knowing) and badgerV2 is not backwards compatible. The plan is to switch to badgerV2 as the default with the next "major" release - v0.15.0. Users on badgerV1 will need to manually update or explicitly set badgerV1 as their db type.

We could do a soft switch to writing badgerV2 in the ca.json. I'm just weary of breaking things for folks who run step ca init multiple times to get a clean config (pretty niche probably).

Let me know if you have any questions or suggestions about the switch from v1 to v2. We're still figuring out the process and we want it to be as seamless as possible for users on v1.

I'm going to close this issue since I think we got to the bottom of it (bad documentation). But feel free to re-open if you run into any more problems.

[rmelilloii]

Ladies and Gents o/
Good morning and happy Friday!

Is this issue still ongoing? I suppose that the latest version does not need me to apply the "./config/ca.json" anymore?

Install: helm install step-certificates smallstep/step-certificates
I run "step ca init" - all good.
I can generate Certificates.

But when I follow the instructions here : https://smallstep.com/docs/step-ca/getting-started/#run-your-certificate-authority
step-ca $(step path)/config/ca.json
I only get: Error opening database of Type badgerv2: error opening Badger database: Cannot acquire directory lock on "/home/step/db". Another process is using this Badger database.: resource temporarily unavailable

I can only change the ca.json from applying a new configmap (with the changes mentioned here).
Tried:
"db": { "type": "badgerv2", "dataSource": "/home/step/db", "badgerFileLoadingMode": "FileIO", "vlogFileSize": "100 M" },

and

"db": { "type": "badgerv2", "dataSource": "/home/step/db", "badgerFileLoadingMode": "FileIO" },

After that I re-create the pod, try again, same error.

Am I misunderstanding how is it supposed to work? Thanks!

[shugyosha89]

I was experiencing this error. It wasn't fixed by using the above guidance.

I solved it by:

1. Change database provider from Badger to MySQL (see docs)
2. Restart step ca (I'm using docker so I restarted the container) → it created a new provisioner
3. Take the new password from the secrets/password file and use it when making new certificate requests