Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[manila-csi-plugin] Cannot extend pvc #2619

Open
stevenyslins opened this issue Jun 14, 2024 · 7 comments
Open

[manila-csi-plugin] Cannot extend pvc #2619

stevenyslins opened this issue Jun 14, 2024 · 7 comments
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@stevenyslins
Copy link

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened:

When I extend the volume, my pvc get Extending Error status
image

What you expected to happen:
Resize volume succeeded

How to reproduce it:

  1. Create a share type
$ manila type-create test-share-type true

+----------------------+--------------------------------------+
| Property             | Value                                |
+----------------------+--------------------------------------+
| ID                   | 9aff85eb-dd63-4883-9a1d-a11e9de86943 |
| Name                 | test-share-type                      |
| Visibility           | public                               |
| is_default           | -                                    |
| required_extra_specs | driver_handles_share_servers : True  |
| optional_extra_specs |                                      |
| Description          | None                                 |
+----------------------+--------------------------------------+
  1. Create a share network
    Need to create a neutron-net and neutron-subnet.
$ manila share-network-create \
     --name test-share-net \
     --neutron-net-id 6e28ce79-6aff-42c1-8a9c-641fb8a5d370 \
     --neutron-subnet-id fbeb0e31-b50c-484a-8605-1ff3cdc92e2f

+-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                          | Value                                                                                                                                                                                                                                                                                                                                                                                             |
+-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                | 11832582-0c53-4575-a9ef-4073259f03c9                                                                                                                                                                                                                                                                                                                                                              |
| name                              | test-share-net                                                                                                                                                                                                                                                                                                                                                                                    |
| project_id                        | acb78797bb0040dbbb0ddcdcf3db8fef                                                                                                                                                                                                                                                                                                                                                                  |
| created_at                        | 2024-06-14T12:05:32.852194                                                                                                                                                                                                                                                                                                                                                                        |
| updated_at                        | None                                                                                                                                                                                                                                                                                                                                                                                              |
| description                       | None                                                                                                                                                                                                                                                                                                                                                                                              |
| status                            | active                                                                                                                                                                                                                                                                                                                                                                                            |
| security_service_update_support   | True                                                                                                                                                                                                                                                                                                                                                                                              |
| network_allocation_update_support | True                                                                                                                                                                                                                                                                                                                                                                                              |
| share_network_subnets             | [{'id': '9fdd2a8f-2a54-46be-86b5-0ab261c2d211', 'availability_zone': None, 'created_at': '2024-06-14T12:05:32.865126', 'updated_at': None, 'segmentation_id': None, 'neutron_net_id': '6e28ce79-6aff-42c1-8a9c-641fb8a5d370', 'neutron_subnet_id': 'fbeb0e31-b50c-484a-8605-1ff3cdc92e2f', 'ip_version': None, 'cidr': None, 'network_type': None, 'mtu': None, 'gateway': None, 'metadata': {}}] |
+-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  1. Create a storageclass

storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-manila-test-nfs
provisioner: nfs.manila.csi.openstack.org
allowVolumeExpansion: true
parameters:
  # Manila share type
  type: test-share-type
  shareNetworkID: 11832582-0c53-4575-a9ef-4073259f03c9

  csi.storage.k8s.io/provisioner-secret-name: csi-manila-secrets
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-expand-secret-name: csi-manila-secrets
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-manila-secrets
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-manila-secrets
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system

apply storageclass yaml

$ kubectl apply -f storageclass.yaml
  1. Create a persistentVolumeClaim:

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-share-pvc-test
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-test-nfs

apply pvc.yaml

$ kubectl apply -f pvc.yaml
image
  1. Extend the volume to 2Gi, the expansion will get error
$ kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
nfs-share-pvc-test   Bound    pvc-b9722890-a85a-4e0e-b967-f121cd3b6464   1Gi        RWX            csi-manila-test-nfs   4m57s


$ kubectl -n default patch pvc nfs-share-pvc-test -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}'
persistentvolumeclaim/nfs-share-pvc-test patched


$ kubectl describe pvc nfs-share-pvc-test
Name:          nfs-share-pvc-test
Namespace:     default
StorageClass:  csi-manila-test-nfs
Status:        Bound
Volume:        pvc-b9722890-a85a-4e0e-b967-f121cd3b6464
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nfs.manila.csi.openstack.org
               volume.kubernetes.io/storage-provisioner: nfs.manila.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       <none>
Conditions:
  Type       Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----       ------  -----------------                 ------------------                ------  -------
  Resizing   True    Mon, 01 Jan 0001 00:00:00 +0000   Fri, 14 Jun 2024 12:28:44 +0000
Events:
  Type     Reason                 Age                    From                                                                                                                  Message
  ----     ------                 ----                   ----                                                                                                                  -------
  Warning  ProvisioningFailed     7m28s                  nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc  failed to provision volume with StorageClass "csi-manila-test-nfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  ProvisioningFailed     6m41s (x6 over 7m27s)  nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc  failed to provision volume with StorageClass "csi-manila-test-nfs": rpc error: code = Aborted desc = volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 is already being processed
  Normal   ExternalProvisioning   5m41s (x9 over 7m38s)  persistentvolume-controller                                                                                           waiting for a volume to be created, either by external provisioner "nfs.manila.csi.openstack.org" or manually created by system administrator
  Normal   Provisioning           5m37s (x8 over 7m38s)  nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc  External provisioner is provisioning volume for claim "default/nfs-share-pvc-test"
  Normal   ProvisioningSucceeded  5m36s                  nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc  Successfully provisioned volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464
  Warning  ExternalExpanding      48s                    volume_expand                                                                                                         Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Warning  VolumeResizeFailed     43s                    external-resizer nfs.manila.csi.openstack.org                                                                         resize volume "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464" by resizer "nfs.manila.csi.openstack.org" failed: rpc error: code = Internal desc = failed to resize volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464: share 1357f637-c7f7-44cd-8919-498c7a14f574 is in error state "extending_error": extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure.
  Normal   Resizing               31s (x6 over 48s)      external-resizer nfs.manila.csi.openstack.org                                                                         External resizer is resizing volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464
  Warning  VolumeResizeFailed     30s (x5 over 41s)      external-resizer nfs.manila.csi.openstack.org                                                                         resize volume "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464" by resizer "nfs.manila.csi.openstack.org" failed: rpc error: code = Unknown desc = Bad request with: [POST http://10.78.19.155:8786/v2/shares/1357f637-c7f7-44cd-8919-498c7a14f574/action], error message: {"badRequest": {"code": 400, "message": "Invalid share: Share 1357f637-c7f7-44cd-8919-498c7a14f574 status must be 'available' to extend, but current status is: extending_error."}}

Anything else we need to know?:

Environment:

  • openstack-cloud-controller-manager(or other related binary) version: v1.25.3
  • OpenStack version:
$ nova --version
18.3.0
  • Others:
@gouthampacha
Copy link
Contributor

hey @stevenyslins

The request failed in manila; it appears you have access to the manila CLI; could you try this:

manila message-list

(or openstack share message list)

maybe there's some detail there around why the share wasn't extended as you expected.

If there's something failing on the backend, you might need to contract your cloud provider.

@stevenyslins
Copy link
Author

stevenyslins commented Jun 17, 2024 •
edited
Loading

Hi @gouthampacha

My instruction is as follows, and I am also the administrator of this OpenStack installation.

Thank you for your response.

$ manila message-list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share message list "
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID                                   | Resource Type | Resource ID                          | Action ID | User Message                                                                                                                                                                                                                              | Detail ID | Created At                 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| 8af43d1c-1050-4e57-8fda-763e894b2cf5 | SHARE         | 1357f637-c7f7-44cd-8919-498c7a14f574 | 008       | extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure. | 015       | 2024-06-14T12:28:31.276346 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+


$ manila list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share list "
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+
| ID                                   | Name                                     | Size | Share Proto | Status          | Is Public | Share Type Name | Host                 | Availability Zone |
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+
| 1357f637-c7f7-44cd-8919-498c7a14f574 | pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 | 1    | NFS         | extending_error | False     | test-share-type | test01@generic       | nova              |
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+

@gouthampacha
Copy link
Contributor

My instruction is as follows, and I am also the administrator of this OpenStack installation.

That's great; because from the error message below; it looks like an ir-recoverrable error occurred on the backend driver's attempt to extend the share.

$ manila message-list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share message list "
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID                                   | Resource Type | Resource ID                          | Action ID | User Message                                                                                                                                                                                                                              | Detail ID | Created At                 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| 8af43d1c-1050-4e57-8fda-763e894b2cf5 | SHARE         | 1357f637-c7f7-44cd-8919-498c7a14f574 | 008       | extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure. | 015       | 2024-06-14T12:28:31.276346 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+

Can you look at your manila-share log file and determine happened here? From the o/p it appears you are using the Generic driver - which creates cinder volumes and exports them as NFS/CIFS shares. Maybe the extension request failed to get communicated to cinder? Or maybe it failed during the interaction? Does this always occur?

@stevenyslins
Copy link
Author

At 2024-06-14 12:28, I grepped the manila-share.log as follows:

# cat /var/log/kolla/manila/manila-share.log |grep -i "2024-06-14 12:28"
2024-06-14 12:28:23.263 16 INFO manila.share.manager [None req-8caafbd0-45f7-4b72-b47c-cce6eb7695f0 - - - - - -] Updating share status
2024-06-14 12:28:29.707 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] [23ea5473-fbd5-4cb6-a2ba-c6db0d1ee280] AMQP server on 10.78.19.158:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed
2024-06-14 12:28:30.713 16 INFO oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] [23ea5473-fbd5-4cb6-a2ba-c6db0d1ee280] Reconnected to AMQP server on 10.78.19.158:5672 via [amqp] client with port 56084.
2024-06-14 12:28:30.716 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Failed to publish message to topic 'openstack': Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': 'd5a29e94-efb7-437c-8905-13f42e74b0c5', 'publisher_id': 'share.test@generic', 'event_type': 'share.extend.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'extending', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:28:29.706843', '_unique_id': 'da7580ad3b3a407da12b317398fb9f2f', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-d06ace44-3924-4a4a-85b8-2338ff4b06e3', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['heat_stack_owner', 'reader', 'member', 'admin', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:28:29.706802', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1005, in ensure
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     ret, channel = autoretry_method()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/connection.py", line 523, in _ensured
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return fun(*args, **kwargs)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/connection.py", line 599, in __call__
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return fun(*args, channel=channels[0], **kwargs), channels[0]
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 994, in execute_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     method()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1477, in _publish_and_creates_default_queue
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     queue.declare()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._create_queue(nowait=nowait, channel=channel)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 615, in _create_queue
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.queue_declare(nowait=nowait, passive=False, channel=channel)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 643, in queue_declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     ret = channel.queue_declare(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/channel.py", line 1162, in queue_declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return queue_declare_ok_t(*self.wait(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/abstract_channel.py", line 99, in wait
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.connection.drain_events(timeout=timeout)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 525, in drain_events
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     while not self.blocking_read(timeout):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 531, in blocking_read
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self.on_inbound_frame(frame)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/method_framing.py", line 53, in on_frame
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     callback(channel, method_sig, buf, None)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 537, in on_inbound_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self.channels[channel_id].dispatch_method(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     listener(*args)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/channel.py", line 293, in _on_close
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     raise error_for_code(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging During handling of the above exception, another exception occurred:
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/notify/messaging.py", line 78, in notify
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.transport._send_notification(target, ctxt, message,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 134, in _send_notification
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._driver.send_notification(target, ctxt, message, version,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 694, in send_notification
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self._send(target, ctxt, message,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 659, in _send
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     conn.notify_send(exchange, target.topic, msg, retry=retry)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1553, in notify_send
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._ensure_publishing(self._publish_and_creates_default_queue,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1383, in _ensure_publishing
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.ensure(method, retry=retry, error_callback=_error_callback)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1027, in ensure
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     raise exceptions.MessageDeliveryFailure(msg)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:31.272 16 ERROR manila.share.manager [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Extend share failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Traceback (most recent call last):
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4483, in extend_share
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     self.driver.extend_share(
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 115, in wrap
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(self, context, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 660, in extend_share
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     helper.disable_access_for_maintenance(server_details, share['name'])
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 178, in wrapped_func
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return source_func(self, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(*args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 176, in source_func
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(self, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 385, in disable_access_for_maintenance
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     self._ssh_exec(server,
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 168, in _ssh_exec
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return processutils.ssh_execute(ssh, ' '.join(command),
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py", line 583, in ssh_execute
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     raise ProcessExecutionError(exit_code=exit_status,
2024-06-14 12:28:31.272 16 ERROR manila.share.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Exit code: 1
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Stdout: ''
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.272 16 ERROR manila.share.manager
2024-06-14 12:28:31.273 16 INFO manila.message.api [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Creating message record for request_id = req-d06ace44-3924-4a4a-85b8-2338ff4b06e3
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Exception during message handling: manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4483, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     self.driver.extend_share(
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 115, in wrap
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 660, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     helper.disable_access_for_maintenance(server_details, share['name'])
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 178, in wrapped_func
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return source_func(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 176, in source_func
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 385, in disable_access_for_maintenance
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     self._ssh_exec(server,
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 168, in _ssh_exec
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return processutils.ssh_execute(ssh, ' '.join(command),
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py", line 583, in ssh_execute
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     raise ProcessExecutionError(exit_code=exit_status,
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Exit code: 1
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stdout: ''
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 236, in wrapped
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/utils.py", line 579, in wrapper
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4499, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     raise exception.ShareExtendingError(
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Exit code: 1
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stdout: ''
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server

# cat /var/log/kolla/manila/manila-share.log |grep -i "1357f637-c7f7-44cd-8919-498c7a14f574"
2024-06-14 12:21:40.049 16 ERROR oslo_messaging.notify.messaging [None req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': '75ca8778-fa63-48a3-9a06-10630f45ecb7', 'publisher_id': 'share.test@generic', 'event_type': 'share.create.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': None, 'host': 'test@generic', 'status': 'creating', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:21:39.038772', '_unique_id': '971cb0b2153c4e5e86aba4e7456af06b', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['admin', 'heat_stack_owner', 'member', 'reader', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:21:39.038699', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:23:12.179 16 ERROR oslo_messaging.notify.messaging [None req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': '2b94a873-1dcc-4295-af01-96cbb598b42a', 'publisher_id': 'share.test@generic', 'event_type': 'share.create.end', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'creating', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:23:11.170967', '_unique_id': 'cddf1670fe3340728d7e39af05a2e666', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['admin', 'heat_stack_owner', 'member', 'reader', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:23:11.170933', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:23:41.049 16 INFO manila.share.access [None req-30525a3b-ae4e-4d17-b82f-ada3aece2c10 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Access rules were successfully modified for share instance 6acebb21-b851-466b-affe-2fcfd85efde8 belonging to share 1357f637-c7f7-44cd-8919-498c7a14f574.
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': 'd5a29e94-efb7-437c-8905-13f42e74b0c5', 'publisher_id': 'share.test@generic', 'event_type': 'share.extend.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'extending', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:28:29.706843', '_unique_id': 'da7580ad3b3a407da12b317398fb9f2f', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-d06ace44-3924-4a4a-85b8-2338ff4b06e3', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['heat_stack_owner', 'reader', 'member', 'admin', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:28:29.706802', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Exception during message handling: manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.

And in the Cinder log, I cannot find any helpful information.

# cat /var/log/kolla/cinder/cinder-volume.log |grep -i "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464"
<no log>

# cat /var/log/kolla/cinder/cinder-volume.log |grep -i "2024-06-14 12:28"
<no log>

Additionally, when I use the Manila CLI to extend the volume, it works.

## Extend the volume to 2Gi
## command : manila extend <share> <new_size>
$ manila extend 1357f637-c7f7-44cd-8919-498c7a14f574 2

@gouthampacha
Copy link
Contributor

Thanks; from the error its apparent that exportfs gave manila an error:

2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"

when performing an unmount operation; which is strange because the list of exports to remove is compiled by executing exportfs: https://opendev.org/openstack/manila/src/commit/0999857c47fca78b0d0a2f8059cd5aa1527cb422/manila/share/drivers/helpers.py#L382

Does this always happen when the extension is requested via manila-csi? The reason I ask is because there is no difference in the API calls between manila-csi and the manila CLI you're using:

https://github.com/gophercloud/gophercloud/blob/3bb883c4ebe42c7eb9382406e4b3f9e1cc3743ca/openstack/sharedfilesystems/v2/shares/requests.go#L310
https://github.com/openstack/python-manilaclient/blob/85f21732176ed6218e4b8b64ac3261dd40f54c7a/manilaclient/v2/shares.py#L741-L744

To triage the exportfs behavior, I'm curious what's the operating system of the manila service image in your environment? and what version?

@stevenyslins
Copy link
Author

Does this always happen when the extension is requested via manila-csi?

Yes, when using the manila-csi-plugin, the status always results in Extending Error.

To triage the exportfs behavior, I'm curious what's the operating system of the manila service image in your environment? and what version?

Our environment runs the Manila service on the following operating system:

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

And manila image as follows:

# docker ps -a |grep -i manila
ff5a572dc04c   quay.io/openstack.kolla/manila-scheduler:2023.2-ubuntu-jammy                  "dumb-init --single-…"   2 weeks ago   Up 7 days (healthy)                 manila_scheduler
3a26a6b15310   quay.io/openstack.kolla/manila-data:2023.2-ubuntu-jammy                       "dumb-init --single-…"   2 weeks ago   Up 7 days (healthy)                 manila_data
ceaf386e0a11   quay.io/openstack.kolla/manila-share:2023.2-ubuntu-jammy                      "dumb-init --single-…"   3 weeks ago   Up 7 days (healthy)                 manila_share
086a858d35f3   quay.io/openstack.kolla/manila-api:2023.2-ubuntu-jammy                        "dumb-init --single-…"   3 weeks ago   Up 7 days (healthy)                 manila_api

Thank you for your response.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gouthampacha @k8s-ci-robot @stevenyslins @k8s-triage-robot