Need ability to clear all storage

[brandonh-msft]

Which service(blob, file, queue, table) does this issue concern?

All

Which version of the Azurite was used?

3.8.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

npm

What's the Node.js version?

12.18.4

What problem was encountered?

I need to be able to clear storage like I could with Azure Storage Emulator using clear all, clear blob, etc.

Steps to reproduce the issue?

I'm working off the output from -h - it says nothing about the ability to do this.

Have you found a mitigation/solution?

No, I am new to Azurite so I'm unsure if there's some "hack" you can do to effectively do this.

Use case:
When developing with a technology like Durable Functions, which heavily utilizes queues, tables, and blob to maintain state of orchestrations it's incredibly useful to be able to clear backend storage contents as a way to "restart" an orchestration debugging session.

[blueww]

@brandonh-msft

Normally, You can just clear the files listed here , in the workspace .

If you use SQL to save metadata, you can just delete the related DB.

[brandonh-msft]

Of course; but that only works as long as y'all never change how you're storing stuff on the backend and/or always keep the docs in sync when you do.

Since this is a replacement for the old emulator, I think it should at least have parity with it. I want to focus on the task at hand, and having to context switch to poke around filesystems or databases isn't conducive to that.

[blueww]

@brandonh-msft

Thanks for the reply!
Would you please clarify what do you mean " it should at least have parity with it. "?

Open source contributions are very welcome. If you have a better solution for that, and would like to contribute to it, feel free to send PR.

[brandonh-msft]

Would you please clarify what do you mean " it should at least have parity with it. "?

Anything that I could do w/ the old Azure Storage Emulator I should be able to do in Azurite.

[blueww]

@brandonh-msft

Thanks for the reply!

Would you please share what's your preferred way for Azurite to clean all data?
Do you want a command line to clean up that?

For this request, we will evaluate per the customer vote. If there are many customer need it, we will high priority it.
And since Azurite is open source, we welcome contribution very much! If you would like to contribute to resolve it, feel free to send PR.

[brandonh-msft]

Do you want a command line to clean up that?

yes

[blueww]

Will follow up that with @brandonh-msft in mail.

[codermrrob]

It is correct that the ability to clear or clean the storage while developing durable functions is really necessary.
It is laborious to rely on deleting files - I must quit from Visual Studio before I can delete them, restart VS and run my durable functions and then repeat for every debug session. It's a poor workflow.

I have also tried using the VSCode durable function plugin to delete the hub (nothing else seemed to work) but it is somewhat slow compared to previous CLI command for the old storage emulator.

I appreciate this is an open source project and if I find time I will do something about it. For now I just want to draw attention to this area, that for developers working a lot on durable functions Azurite breaks our development workflow in comparison with the old storage emulator.

[jacobherrera]

After a new laptop with only VS2022, my workflow is hampered by the absence of a method equivalent to "AzureStorageEmulator.exe clear all". As more people migrate to VS2022, this will only become more urgent.

[blueww]

@jacobherrera
Thanks for the information!
Currently We can only cleanup Azurite storage data, when Azurite is not running. So we need VS team support to see how to apply this in VS.
I will contact VS team for this issue, and update later.

[blueww]

We have discussed with VS team in mail, and generally agree will have a fix for it in VS.
Will update if any progress from VS team.

[brandonh-msft]

I want to point out that what's been asked & advocated for isn't yet another UI solution, it's a command-line driven one that affords scriptability and no dependence on external/consuming product teams for implementation/support; parity with Azure Storage Emulator.

If this is to be implemented in each consumer of Azurite in whatever way the consumer so chooses, Azurite is forever beholden to those consumers to never break/change the way it stores things lest the consumer's functionality stop working. Or, at best, Azurite has now tied itself to the release cadence of whichever external consumers it deems "unbreakable" which, IMO, is against the very spirit of Azurite.
This, vs Azurite owning the way things are cleared/stored in and of itself and external consumers simply exercising exposed APIs to clear the storage in the way Azurite requires it done for whatever version being run.
Additionally, implementing it as an API (i.e. terminal command) would unblock all users everywhere now vs waiting for a release of their IDE of choice to get the functionality.

[blueww]

@brandonh-msft
Thanks for the clarify!
We will also leave this issue open to track the work of a command-line to clean up Azurite data.
However, per current Azurite framework, to clean up Azurite data, there must be no Azurite instance is running with the specific workspace (or the data might not be aligned and take issues): so we must:

1. stop Azurite,
2. clean up Azurite workspace folder (manually or by commandline),
3. start Azurite.

[blueww]

@brandonh-msft
I have written a batch file to Clean storage data of Azurite.
I save it to a bat file and tested it on my windows machine. It looks work well,
Would you please see if it can meet your requirement?

Please note:

1. input the workspace folder before run the batch file.
2. You can add more parameter in start Azurite, like "-d "

powershell -command "Get-WmiObject Win32_Process | where {$_.CommandLine -like '*azurite.js*' -and $_.Name -eq 'node.exe'} | ForEach-Object { Invoke-WmiMethod -Path $_.__Path -Name Terminate }"
rmdir <WorkspaceFolder>  /q /s
start /MIN Azurite -l <WorkspaceFolder>

[brandonh-msft]

Seems like that solves for clear all but not clear blob, clear table, or clear queue per initial ask.

[blueww]

@brandonh-msft
For clean blob:
You can just modify the 2nd line of the script a little: instead of removing the whole workspace folder, delete all sub file/directory whose name like *blob* in the workspace folder. like:

powershell -command "Get-ChildItem <WorkspaceFolder> | where {$_.Name -like '*blob*' }| Remove-Item -Recurse -Force"

clear table, or clear queue can also use similar way.

[brandonh-msft]

I'm fine if you want to propose that as a temporary workaround but not the permanent solution.

1. It's not affordable outside this work item - people would have to come find this to find that script line, nobody's going to discover that on their own
2. It's not obvious what's being done if you were to embed this in a script somewhere; you'd have to comment the line to point out what you're doing here and why
3. It's not cross-platform except and unless somebody installs Powershell Core on their box, which shouldn't be a prereq for using Azurite.

So what I'm saying is thanks, and I'll use that if/when I get back to having Azurite as part of my daily dev workflow (I've moved on from Functions/DF work) but would much rather see something I can discover & interpret from azurite -h than a separate script I pulled from the bowels of the internet ;)

[blueww]

@brandonh-msft
Thanks for the feedback!
Will keep this issue open to track it.

[Clockware]

+1 for this request. Something like "factory-reset" would be nice that resolves all the errors and clears data.
Azurite broke after power off (doesn't start anymore), no clue for easy fixes.

[zpertee]

I'm sorry to ask such an ignorant question, but can anyone point me to the default Azurite location for VS 2022 on Win 11? I desperately need this functionality, but can't seem to find it.

[brandonh-msft]

I'm sorry to ask such an ignorant question, but can anyone point me to the default Azurite location for VS 2022 on Win 11? I desperately need this functionality, but can't seem to find it.

When you fire up VS and target a Fxn app as your startup project, you'll get this kind of output in the 'Service Dependencies' area of the Output pane:

Ensuring Azure Functions Core Tools are up to date. This may take a few minutes...
<Project Name>: <THIS IS THE AZURITE INSTALL LOCATION>\azurite.exe --location "<azurite data location>" --debug "<azurite data location>\debug.log" --skipApiVersionCheck

That'll tell you where it is. It's likely going to be something like c:\program files\microsoft visual studio\2022\common7\ide\extensions\microsoft\Azure Storage Emulator\azurite.exe

[zpertee]

@brandonh-msft Thank you!!

[bmode]

Another +1 for this request. We are using the old Azure Storage Emulator for some of our automated end-to-end tests. Those tests run the "clear all" command at startup to ensure a consistent environment between test runs. The lack of similar functionality has so far prevented us from migrating those tests to use Azurite.

[blueww]

@bmode

Thanks for raise the request!
Before we add this function, could you please replace the "clear all" command to a command to delete all files under the workspace folder of Azurite, it will have same effect. (When start Azurite you can specify workspace folder with "-l", see details.)

[codermrrob]

@blueww When I previously tried to delete all files the problem I had, at least when running with Visual Studio, the files were locked and I had to close VS in order to delete the files. This is very disruptive to a development workflow.

[blueww]

@codermrrob

The file is locked should because the Azurite instance started by VS is still running, so the file is locked by the running Azurite instance.

Currently to clean up Azurite data, there must be no Azurite instance is running with the specific workspace (or the data might not be aligned and take issues) We still don't find a good way to cleanup Azurite data when the Azurite instance is running, there might need big framework change to get this.

As you said "Those tests run the "clear all" command at startup", so could you cleanup the workspace folder before start running Azurite in the test?

[codermrrob]

Actually for durable function development all I really need to do is clean all the data within the task hub that the durable function uses. If I can clear all task hub data easily that is probably good enough most of the time.

Currently I use the VSCode Durable Function extension to achieve this, and it works well as workaround.

[blueww]

Thanks @codermrrob for the update!
Good to know you have work around it.

[brandonh-msft]

Thanks @codermrrob for the update! Good to know you have work around it.

Just chiming in to make sure we're not saying that every user who has need to "clear all" for azurite should "just" install VS Code + DF Extension in order to do it.

[zakkra]

The above .bat deleted the Azurite.exe too. Serves me right for not taking a better look before running it

[blueww]

@zakkra

Do you put the Azurite.exe in same folder as the WorkspaceFolder? （this command will remove all files under workspacefolder）
If so, could you set a separate folder for WorkspaceFolder?
Or user following command to clear up only blob data (and table/queue in similar way)

powershell -command "Get-ChildItem <WorkspaceFolder> | where {$_.Name -like '*blob*' }| Remove-Item -Recurse -Force"

[rruenroeng]

Hi there!

I'm a bit uncomfortable with the above proposed solutions. I feel like I'm begging to leave my compy in a sad state. Is it simpler to uninstall Azurite and reinstall?

[blueww]

@rruenroeng
Where do you get Azurite? Do you run it from npm, VSCode Extension, docker, or Visual studio?
Uninstall Azurite and reinstall normally won't work to clean the data, since it won't clean the file location (specified with "-l" parameter).

[KSmanis]

This feature would make a lot of sense with the recently introduced --inMemoryPersistence flag, e.g., a signal (think SIGHUP) that soft resets azurite without spawning a new process.

[rruenroeng]

Sorry for responding a lifetime later @blueww ,
I was running it from Visual Studio. That's also where I got it.

[blueww]

@rruenroeng
The latest Azurite version (3.28.0) support a new parameter '--inMemoryPersistence', it will cache data in memory instead of disk, so with this parameter all data will be cleaned up by restart Azurite. (There is also some limitation, see details.)
However, it might still take time for Visual Studio upgrading Azurite in it to 3.28.0. Before that, you can also consider get Azurite from npm, VS code extension, or docker image.

[brandonh-msft]

@rruenroeng The latest Azurite version (3.28.0) support a new parameter '--inMemoryPersistence', it will cache data in memory instead of disk, so with this parameter all data will be cleaned up by restart Azurite. (There is also some limitation, see details.) However, it might still take time for Visual Studio upgrading Azurite in it to 3.28.0. Before that, you can also consider get Azurite from npm, VS code extension, or docker image.

This is good but still not what's needed. I do want persistence between runs of Azurite (e.g. a reboot of my system, etc.) while I'm developing, but need the ability to clear this on demand when necessary. And to be super-candid, the fact we're adding features like in-memory persistence (not sure how much of an ask this was from users) instead of something that's been requested for over 3 years now is a bit frustrating.

[MMThornberg]

@rruenroeng The latest Azurite version (3.28.0) support a new parameter '--inMemoryPersistence', it will cache data in memory instead of disk, so with this parameter all data will be cleaned up by restart Azurite. (There is also some limitation, see details.) However, it might still take time for Visual Studio upgrading Azurite in it to 3.28.0. Before that, you can also consider get Azurite from npm, VS code extension, or docker image.

Sounds good - however where can this parameter be set in Visual Studio?

[blueww]

@rruenroeng The latest Azurite version (3.28.0) support a new parameter '--inMemoryPersistence', it will cache data in memory instead of disk, so with this parameter all data will be cleaned up by restart Azurite. (There is also some limitation, see details.) However, it might still take time for Visual Studio upgrading Azurite in it to 3.28.0. Before that, you can also consider get Azurite from npm, VS code extension, or docker image.

Sounds good - however where can this parameter be set in Visual Studio?

Would you like to contact VS team to ask them to open this configuration in the VS started Azurite.
This doc has the details on how to report a VS feedback issue : https://docs.microsoft.com/en-us/visualstudio/ide/how-to-report-a-problem-with-visual-studio?view=vs-2022

[MMThornberg]

@rruenroeng The latest Azurite version (3.28.0) support a new parameter '--inMemoryPersistence', it will cache data in memory instead of disk, so with this parameter all data will be cleaned up by restart Azurite. (There is also some limitation, see details.) However, it might still take time for Visual Studio upgrading Azurite in it to 3.28.0. Before that, you can also consider get Azurite from npm, VS code extension, or docker image.

This is good but still not what's needed. I do want persistence between runs of Azurite (e.g. a reboot of my system, etc.) while I'm developing, but need the ability to clear this on demand when necessary. And to be super-candid, the fact we're adding features like in-memory persistence (not sure how much of an ask this was from users) instead of something that's been requested for over 3 years now is a bit frustrating.

Workspace is located here: C:\Users\USERNAME\AppData\Local.vstools\azurite
Delete the files and folders mentioned here (except actual names are pre- and postfixed "__": https://github.com/Azure/azurite?tab=readme-ov-file#workspace-structure
Then restart VS.

[tomkerkhove]

Any update on this?

[blueww]

@tomkerkhove

We are working on some other higher priority features. So I am afraid this won't be in our recent priority list.
You can find some workaround in the above comments.