Want to understand more details about file in local temp path syncing to Storage server.

[jejass]

Hello Team,

We are currently assisting cx in their efforts to leverage the blob CSI driver to invoke blobfuse for accessing a Storage account from their AKS cluster. During our investigation, we have encountered the following questions and would appreciate your insights:

Question 1: Synchronization Mechanism and Error Handling in Blobfuse

We would like to confirm our understanding of the blobfuse mechanism, where data is initially written to a local temporary path before being synchronized with the Storage server. Could you please clarify the following:

Are there any scenarios that may prevent successful synchronization? In the event of a sync operation failure, how does blobfuse handle this situation? Does the eviction of local cache operations verify whether the data has been successfully uploaded to the Storage server? How does blobfuse ensure that data in the local temporary path is ultimately written to Storage without the risk of data loss?

Question 2: Blobfuse Disk Type Selection Logic

Additionally, we observed a performance discrepancy in the AKS cluster, specifically related to the choice of pod SKU and the corresponding disk type. We test the performance by running below command in the pod.
while true; do BT=$(date +'%Y/%m/%d %T.%9N');dd bs=1M count=1000 if=/dev/zero of=/mnt/blob/testfile oflag=direct >>/root/log.txt 2>&1 ;echo "Begin at $BT, Sucess at $(date +'%Y/%m/%d %T.%9N') ">> /root/log.txt 2>&1 ;sleep 0.5 ; done

Below testing result is using 'Standard_D8s_v5' SKU. The initial upload speed is significantly higher but decreases after subsequent attempts.

Our findings indicate a correlation with pod SKUs, with "Standard_D8ds_v5" showing better performance due to its ephemeral OS disk.

We are curious about the logic behind blobfuse's choice of disk type. Could you provide insights into how blobfuse determines which disk type to use? How does disk type affect the performance? Are there any considerations or configurations that can be adjusted to optimize performance, especially when working with different pod SKUs?

Your expertise in addressing these queries is crucial for cx's successful implementation, and we appreciate your time and assistance. Please feel free to reach out if you require any further details.

[vibhansa-msft]

Hi,

Thanks for reaching out. In case file-cache model (local caching) blobfuse will download the file on open to a local disk, in case of create it will be created on local disk. Post this all read/write operations will be served from local disk only. Only on a file-close, data will be uploaded to container. In any case if blobfuse was killed/terminated before it could complete the upload then data will be lost. As all the read/writes are served from local disk, based on the SKU/disk type performance will differ.
Blobfuse does not choose any disk type or location, it totally configuration driven. In your system/VM/Host if you have a better disk available then you need to configure that path as local cache path while mounting.

[vibhansa-msft]

This is correct understanding. However, you shall know that there are applications that fire close command and do not wait for it to complete and in such cases application firing close() call does not mean file is synced. I have observed such cases in "dd" command where it does not wait for close() call to complete. So you need to be careful on what your application does.

[waychow1986]

Hi @vibhansa-msft , i found this description at Azure offical document which is said 'The data is written to Blob Storage only after the file handle is closed.' https://learn.microsoft.com/en-us/azure/storage/blobs/blobfuse2-what-is#file-caching-on-disk
I am confused that this is different as you said that close call shall be blocked till the upload completes.

[vibhansa-msft]

Why do you think this one differs. This says data is synced when handle is closed and I am saying when you close the handle call will return back after the data upload completes. So, both statements are complementary to each-other. "Only after handle is closed" just means that data is not synced in between the file operations until you close the handle.

[jejass]

@vibhansa-msft , thanks for your response. as you mentioned before, 'dd' command will not wait for close() call to complete. does it mean the output speed does not take the time of uploading local cache to Storage into consideration when calculating the speed?

We attempted to assess performance using 'fio.' Could you please confirm whether 'fio' handles 'file-close' in the same manner as 'dd,' not waiting for the 'file-close' to finish? I acknowledge that this might not be directly related to blobfuse. Any information on this matter would be greatly appreciated.

currently, we are testing blobfuse to check if it meets requirement of cx's scenario. is there any recommended tool to test the read/write performance of blobfuse? is there any documentation that introduce any benchmark of blobfuse?

[vibhansa-msft]

Might be based on your system it differs. Generally we also do read/write perf testing using fio and dd only. There are cases where we write our own code in python/golang to read/write a file and the numbers reported by our own code is much higher then what dd/fio reports. If you are comparing the perf against some other tool then using dd/fio as a standard tool helps but if you are just interested in evaluating overall time taken by application to read/write using blobfuse then having your own tool will give you much accurate numbers.