Enable stateful functions – stream/metadata missing in Zookeeper

[HeikoMueller]

Hi. I have a running Pulsar cluster on bare metal with brokers, bookies, function_workers and zookeeper. In the initial setup, I did not enable Stream Storage in bookie config. Now I added this with

extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
storageserver.grpc.port=4181
dlog.bkcEnsembleSize=3
dlog.bkcWriteQuorumSize=2
dlog.bkcAckQuorumSize=2
storage.range.store.dirs=data/bookkeeper/ranges
storage.serve.readonly.tables=false
storage.cluster.controller.schedule.interval.ms=30000

The bookie does not start now with an error complaining about missing metadata for Stream Storage. In my zookeeper, I see these directories:
[admin, counters, ledgers, loadbalance, managed-ledgers, pulsar, schemas, stream, zookeeper]
but in /stream, I only find:
[storage]

I tried to re-initialize the cluster as mentioned here
apache/bookkeeper#2084 (comment)

... but I am unable to re-initialize the cluster with ./bin/pulsar initialize-cluster-metadata , seeing this error:

2024-06-17T13:12:40,546+0200 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100c32d40e90060
Unexpected error occured.
org.apache.bookkeeper.stream.storage.exceptions.StorageRuntimeException: Failed to fetch cluster metadata from zookeeper @/stream/metadata
at org.apache.bookkeeper.stream.storage.impl.cluster.ZkClusterMetadataStore.getClusterMetadata(ZkClusterMetadataStore.java:201)
at org.apache.bookkeeper.stream.storage.impl.cluster.ZkClusterInitializer.initializeCluster(ZkClusterInitializer.java:90)
at org.apache.pulsar.PulsarClusterMetadataSetup.initializeCluster(PulsarClusterMetadataSetup.java:308)
at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:258)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /stream/metadata
at org.apache.zookeeper.KeeperException.create(KeeperException.java:117)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:53)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1972)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:327)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:316)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:313)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:304)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:35)
at org.apache.bookkeeper.stream.storage.impl.cluster.ZkClusterMetadataStore.getClusterMetadata(ZkClusterMetadataStore.java:195)
... 3 more
Terminating JVM...
2024-06-17T13:12:40,550+0200 [main] WARN org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1

What am I missing here? Can somebody help in telling how to enable stateful functions on an already running Pulsar cluster (or what I did wrong here)?