Skip to content

Latest commit

 

History

History
175 lines (126 loc) · 5.44 KB

pmxcfs.adoc

File metadata and controls

175 lines (126 loc) · 5.44 KB

Proxmox Cluster file system (pmxcfs)

The Proxmox Cluster file system (pmxcfs) is a database-driven file system for storing configuration files, replicated in real time to all cluster nodes using corosync. We use this to store all PVE related configuration files.

Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. That imposes restriction on the maximal size, which is currently 30MB. This is still enough to store the configuration of several thousand virtual machines.

Advantages

  • seamless replication of all configuration to all nodes in real time

  • provides strong consistency checks to avoid duplicate VM IDs

  • read-only when a node loses quorum

  • automatic updates of the corosync cluster configuration to all nodes

  • includes a distributed locking mechanism

POSIX Compatibility

The file system is based on FUSE, so the behavior is POSIX like. But some feature are simply not implemented, because we do not need them:

  • you can just generate normal files and directories, but no symbolic links, …​

  • you can’t rename non-empty directories (because this makes it easier to guarantee that VMIDs are unique).

  • you can’t change file permissions (permissions are based on path)

  • O_EXCL creates were not atomic (like old NFS)

  • O_TRUNC creates are not atomic (FUSE restriction)

File access rights

All files and directories are owned by user root and have group www-data. Only root has write permissions, but group www-data can read most files. Files below the following paths:

/etc/pve/priv/
/etc/pve/nodes/${NAME}/priv/

are only accessible by root.

Technology

We use the Corosync Cluster Engine for cluster communication, and SQlite for the database file. The filesystem is implemented in user space using FUSE.

File system layout

The file system is mounted at:

/etc/pve

Files

corosync.conf

corosync cluster configuration file (previous to {pve} 4.x this file was called cluster.conf)

storage.cfg

{pve} storage configuration

datacenter.cfg

{pve} datacenter wide configuration (keyboard layout, proxy, …​)

user.cfg

{pve} access control configuration (users/groups/…​)

domains.cfg

{pve} Authentication domains

authkey.pub

public key used by ticket system

pve-root-ca.pem

public certificate of cluster CA

priv/shadow.cfg

shadow password file

priv/authkey.key

private key used by ticket system

priv/pve-root-ca.key

private key of cluster CA

nodes/<NAME>/pve-ssl.pem

public ssl certificate for web server (signed by cluster CA)

nodes/<NAME>/pve-ssl.key

private ssl key for pve-ssl.pem

nodes/<NAME>/pveproxy-ssl.pem

public ssl certificate (chain) for web server (optional override for pve-ssl.pem)

nodes/<NAME>/pveproxy-ssl.key

private ssl key for pveproxy-ssl.pem (optional)

nodes/<NAME>/qemu-server/<VMID>.conf

VM configuration data for KVM VMs

nodes/<NAME>/lxc/<VMID>.conf

VM configuration data for LXC containers

firewall/cluster.fw

Firewall config applied to all nodes

firewall/<NAME>.fw

Firewall config for individual nodes

firewall/<VMID>.fw

Firewall config for VMs and Containers

Symbolic links

local

nodes/<LOCAL_HOST_NAME>

qemu-server

nodes/<LOCAL_HOST_NAME>/qemu-server/

lxc

nodes/<LOCAL_HOST_NAME>/lxc/

Special status files for debugging (JSON)

.version

file versions (to detect file modifications)

.members

Info about cluster members

.vmlist

List of all VMs

.clusterlog

Cluster log (last 50 entries)

.rrd

RRD data (most recent entries)

Enable/Disable debugging

You can enable verbose syslog messages with:

echo "1" >/etc/pve/.debug

And disable verbose syslog messages with:

echo "0" >/etc/pve/.debug

Recovery

If you have major problems with your Proxmox VE host, e.g. hardware issues, it could be helpful to just copy the pmxcfs database file /var/lib/pve-cluster/config.db and move it to a new Proxmox VE host. On the new host (with nothing running), you need to stop the pve-cluster service and replace the config.db file (needed permissions 0600). Second, adapt /etc/hostname and /etc/hosts according to the lost Proxmox VE host, then reboot and check. (And don´t forget your VM/CT data)

Remove Cluster configuration

The recommended way is to reinstall the node after you removed it from your cluster. This makes sure that all secret cluster/ssh keys and any shared configuration data is destroyed.

In some cases, you might prefer to put a node back to local mode without reinstall, which is described here:

  • stop the cluster file system in /etc/pve/

    # systemctl stop pve-cluster

  • start it again but forcing local mode

    # pmxcfs -l

  • remove the cluster config

    # rm /etc/pve/cluster.conf
# rm /etc/cluster/cluster.conf
# rm /var/lib/pve-cluster/corosync.authkey

  • stop the cluster file system again

    # service pve-cluster stop

  • restart pve services (or reboot)

    # service pve-cluster start
# service pvedaemon restart
# service pveproxy restart
# service pvestatd restart