Indexing/building performance issues

[Wezz19]

Describe the bug

• High CPU and RAM usage for the IDE process, when working on a sketch for longer than 30mins.
• Indexing and building never stops after 60+ mins
• Scrolling the editor is extremely slow and choppy after 60+ mins
• Hovering the pointer over variables never brings up additional info about them, after 30+ mins
• (The IDE has never crashed for me though, even after 4+ h usage)

(all the CPU and RAM usages are for the IDE 2.0 process)

1. (Time since IDE launch: 0mins) CPU 0% and RAM about 400MB. Indexing completes in less than 2s.
2. (Time since IDE launch: 30mins) CPU 47% and RAM constantly jumping between 1100->500MB, indexing and building sketch takes longer and longer after even the tiniest edit.
3. (Time since IDE launch: 60mins) CPU jumping between 57-49% and RAM jumping between 2300->1200MB. Memory graph in Task manager looks like a saw blade. Indexing and building never finishes.
4. (Time since IDE launch: 0mins) Saving and restarting the IDE brings the CPU down to 2% and RAM to 500MB.

To Reproduce

1. Load up a sketch
2. Edit, verify and upload for at least 30 to 60mins

Workaround
Restart the IDE every 30min

Hardware/Software

• OS: Windows 10
• IDE: 2.0 nightly 20211219
• CPU: i5-6200U
• Board: Nano clone

Additional context
Possibly related issues:
#431
#709

[IanBUK]

I'm getting performance issues with RC3 on MacOS that are resolved by restarting the IDE. I'll check to see if memory/cpu usage trends upwards.

[Ali229]

Getting similar behavior, the IDE is extremely slow, and keeps indexing/building and take system resources while doing it, during this time the IDE stops error checking or keeps showing error on code that has no syntax errors. For now it's so slow and unusable that I have to use another IDE to write code, when I'm done I copy it to Arduino and compile!

[IanBUK]

I’ve gone back to version 1.8.3 of the ide. Yes, I’ve lost all the new features, but at least it doesn’t slow down so much.

This is a real barrier to release.

[IrregularShed]

I'm in the exact same situation unfortunately. I don't like using the old IDE because it's not got many of the niceties that I've grown to expect over the past decade with other IDEs and editors (VS Code really spoiled us) so it slows me down; sadly the new one slows me down in another way.

[peter76]

Same problem here on Fedora linux; editor turns unbearably slow after 50 lines of code or so; just keeps indexing and intellisense stops working. VS Code works like a charm with the arduino extension installed.

Just to sort out some misconfiguration, I added a new user on my system and ran a freshly downloaded ide from there, same problem.

Let me know if I can do something to look into this, the ide looks promising, but is unusable as it is:-(

[xinkiknix]

For me the issue got solved by renaming all directories (also hidden directories) containing any reference to arduino installed items. Uninstalling Arduino still left some directories (for future reuse?).
I am running version 2.0 rc6, and it seems to behave ever since.

[JiriBilek]

I always do a clean install (delete the Arduino IDE folder and create a new one) and the problem still persists.
Using this IDE with ESP32 toolchain is quite annoying. On the other hand, the VSCode runs like a charm, so the problem is in the Arduino IDE and not in the toolchain, I think.

[IrregularShed]

I got a brand new laptop a couple of months ago and installed this on it as one of the first pieces of software - there was no difference for me either.

[serpinio]

The same for me, but with ESP32 it takes much less time for the editor to become "stuck" due to the fact that the chip supports WiFi and Bluetooth. Because of that even a blank sketch with WiFi enabled takes 630KB on flash after compilation. So a lot of code is recompiled compiled every time I make the slightest change in my .ino file.

I have a proposal: please add an option to only execute language server after quicksave/manual save. And/or the option could also be "Maximum frequency of Language Server updates", which would allow the user to limit the language server to re-index/re-build only once every 30 seconds (depending on how complex the sketch is, that is up to the user, hence the flexible duration). There is a plethora more ways to make it more controllable by the user, but one or both of these should at least make it much more bearable to work comfortably with the Language Server running in the background.

The biggest issue is that the editor rushes to recompile everything on each character type in the editor. I tried disabling auto save, but nothing changed. I believe this is simply wasteful to trigger recompilation on the tiniest change in code. Sure, if the code you changed wasn't verified yet, you won't be able to see your changes in tool-tips and suggestions, but it makes much more sense to give the programmer the choice when to re-build the code. Considering that Arduino IDE 2.0 compiles in single-core, seems like it places those indexing/building tasks in queue. So if you type, then pause for a couple seconds, then type again and do so for like 10-20 characters, that's it. The language server queue is overwhelmed and you're never getting your suggestions and have to work with a super-sluggish editor. Having a powerful 16-core CPU AMD 5950X and 64GB RAM makes no difference compared to my i7 2600K/16GB due to this queuing behavior of a single-core editor.

Adding a switch to only re-index/re-build the sketch on save will also help with editor completely freezing when it's in such an overwhelmed state and you open another project window. For me in 95% of cases this guarantees complete IDE freeze in 5-20 seconds for both windows. Sometimes it's so bad, the OS becomes unresponsive. Not a single other app I use does this.

tl;dr;

Please add an option to re-index/re-build the sketch only on save (manual or auto). This way one could change auto-save to happen only once a minute and the editor would have plenty of time to finish the re-index/re-build process even for monstrously huge tool-chains like the one for the ESP32 chips. And/or the setting to limit how often (in seconds) the language server can re-index/re-build the sketch in the background with only one process per sketch active at a time.

And thank you for a very useful editor which is available for Linux unlike VCode. OS Linux Mint 19.1.

In the mean-time: a workaround for those who want to be able to comfortably work in the editor:

1. Press CTRL+SHIFT+P
2. Type "server" in the search field
3. Click "Arduino: Stop Language Server"
4. Enjoy working in a super-responsive IDE with lightning-fast suggestions
5. When ready to re-index, select the "Arduino: Start Language Server" command and wait for a re-index

Boy oh boy does disabling the language server make a difference when you have to type in a whole new method or a class without having to worry about slowing down the IDE more and more with each key press! I would also recommend to set up keyboard shortcuts for this. Mine are CTRL+SHIFT+0 to stop and CTRL+SHIFT+1 to restart the Language Server (File > Advanced > Keyboard Shortcuts). In fact, the first thing I do when opening any project in Arduino IDE is disable Language Server. I only enable it when I'm lost and need suggestions and automatic reference discovery, and then disable it as soon as I get my answer.

[Oscarm99]

I'm also facing this issue on a Ryzen 5 2500u laptop with 8GB of ram with over 30% CPU continuous usage and between 1 and 1.5 gigs of ram for the "Arduino IDE" process on Windows 10 using IDE 2.0.0-rc6. This happens with Nano, Uno and Mega despite the sketch size. Having multiple instances of the IDE open at the same time do not degrade performance by a lot more.
My current workaround is to work on IDE v1.8.19. Hope this will get fixed soon!

[StefanL38]

ptillisch asked If I can provide detailed information about my system

I'm using a 16GB RAM 2TB-SSD Lenovo Thinkpad Tablet Gen.2
not an uptodate HIGH-End gaming machine. But pretty fast.
I have no performance problems with anything else

Example: I use the software mailstore Home
https://www.mailstore.com/de/produkte/mailstore-home/
Which has indexed 8,2 GB of email-content
If I let do mailstore home a search inside the 8,2 GB for a keyword the results appear within 2 seconds.

Gerätename X1-Tablet-16-2
Prozessor Intel(R) Core(TM) i7-7Y75 CPU @ 1.30GHz 1.61 GHz
Installierter RAM 16,0 GB (15,8 GB verwendbar)
Geräte-ID CDE53F97-A7E0-43E3-A7EE-C3FD33BA0B61
Produkt-ID 00330-80000-00000-AA986
Systemtyp 64-Bit-Betriebssystem, x64-basierter Prozessor
Stift- und Toucheingabe Unterstützung der Stift- und Toucheingabe mit 10 Touchpunkten

Edition Windows 10 Pro
Version 21H2
Installiert am ‎27.‎03.‎2021
Betriebssystembuild 19044.1706
Leistung Windows Feature Experience Pack 120.2212.4170.0

I have installed additional boards
ESP8266, ESP32, Seeeduino SAMD, PJCR Teensy
I also attach the filelist of Arduino15 and Arduino mainfolder

Arduino15-Dirlist.zip
Arduino-main-Dirlist.zip

Both have about 40.000 lines = files

I'm using F-Secure 2022 as the internetsecurity suite.
I have quite some basic knowledge about Windows so if you can give me some hints what kind of information would be helpful to reproduce the issue then post this information

You should make Stop Language Server the default

Press CTRL+SHIFT+P
Type "server" in the search field
Click "Arduino: Stop Language Server"
Enjoy working in a super-responsive IDE with lightning-fast suggestions
When ready to re-index, select the "Arduino: Start Language Server" command and wait for a re-index

quote of user ptillisch at Arduino.cc

Even though I think there is surely some room for improvement in the results we are experiencing, none of the members of the team working on the Arduino IDE 2.x are able to reproduce the exceptionally long processing times reported by those users. This makes the problem difficult to investigate, and validation of any potential improvements or fixes difficult to validate.

This makes me think:
maybe you are doing the same bad habits as microsoft-developpers do
Using superhigh-end computers with 64GB of RAM 32 GB-RAM-disk and top-speed SSD-Raids
for developping. This is OK to speed up the rebuilding process
but
when it comes to test the application-performance you should use a 3third generation processor in a 4GB-RAM system using a classcial HDD with a lot of additional boards installed.

[xinkiknix]

I did this as requested:
Besides that the sketch (any sketch) can not find required libraries (e.g. Arduino.h) the CPU utilisation is very low
result:

[kittaakos]

I appreciate your help, @xinkiknix ❤️

Could you please tell me how did you try to compile the sketch? What did you do?

I pushed two minor fixes for the VS Code extension (see the commit ref here). Could you please uninstall the test VS Code extension (vscode-arduino-tools-test) and use the same download link to re-download and re-install it again? The link did not change: https://github.com/arduino/vscode-arduino-tools/raw/run-arduino-ls-in-code/build-artifacts/vscode-arduino-tools-test-0.0.2-beta.4.vsix

Thank you!

[xinkiknix]

I assumed it tried to compile while running the Arduino:verify?.

I installed the latest as requested and ran some test, changing code, adding comments, ... and running start language server and verify.

While running start language server I notice that endpoint security starts eating all CPU

Stopping anti-virus reduces the time required to run the process.

Same I observed in latest nightly build, any code change results in endpoint going to +80% for a very long time.

while running verify clangd.exe barely goes beyond 8% short peak of 28%. Endpoint does not show up as using much CPU.

Is there a way to make the AV software skip checking while updating local files from within Arduino?

I am using Avira AV software

"Building sketch" keeps running in nightly build while CPU usage of Arduino.exe hovers around 28%

[InstantMuffin]

Stopping anti-virus reduces the time required to run the process.

Yes. It is unrelated to the issue at hand, but the language server does write the compiled data to the hard drive which on-access AV software generally checks.

Is there a way to make the AV software skip checking while updating local files from within Arduino?

If you like to use AV and its on-access scanning feature, you can exempt specific directories from the scan. Unfortunately I can't tell you where the cache is built on windows. I would suspect either in the temp folder or somewhere in the appdata directory structure.
On XUbuntu the cache looks like this, feel free to use filenames and foldernames (exempting the number in the root folder's name) to do a filesystem search on your system. Keep in mind to have the Arduino IDE running in the background (and ideally you have typed a few characters into its textfield to make sure the cache is being built):

[xinkiknix]

This has no effect on AV utilisation, there must be other folders where data is written. I excluded the complete temp directory...
In the nightly build 0729 I see little CPU utilisation after a while, <3% while Building Sketch is still running...

and this runs for a very long time, eventually stops. After restarting the IDE CPU utilisation goes to 0.1%

[InstantMuffin]

So as an affected user running linux, apart from the vscode test which I have done (I could not reproduce the issue there so far), what can I do?

-I tried running the Arduino IDE's profiler. It doesn't work on linux and indefinitely hangs on "processing" after stopping the recording (I waited for the "processing" to finish for a couple of hours to no avail)
-@per1234 mentioned there is a debugger as part of Arduino IDE's featureset in the developer tools, I assume it is under "sources"?
-Reproduce this in a VM and send that VM+snapshot to a dev?
-Let a dev use TeamViewer etc. when the IDE encounters the bug? Any tools you would like me to install beforehand?

@kittaakos as a fellow German wouldn't have any timezone issues with me.

[xinkiknix]

I did run the test on Friday, and did some testing during the weekend and today (running nightly build).
The VS Code "Arduino: Verify" runs fine now, no errors and no extensive CPU utilisation, only short bursts when running "Arduino: Verify". When switching boards ( "Arduino: start language server" I see a significant difference between boards. And I think I found the reason why tasks run slow and high CPU utilisation:
CC1plus.exe and xtensa-esp32<xx>-elf-g++.exe for the respective boards are not signed, this triggers AV and runs CPU to 100% on all cores. I excluded these files in the AV for the boards I use (ESP32 & ESP32-S3) for these boards both "Arduino: Verify" and "Arduino: start language server" run very fast and with little CPU utilisation.
In Arduino nightly build the exclusion of these files in AV make the IDE run way much faster for verify and compile/Upload. the difference is from compile time of over 5 min. to compile < 1min, upload similar.
None of the Arduino components run with high CPU utilisation most of the time. There was one instance where 1 process ran for a few minutes with high CPU, but showing no activity in the IDE. (no Build, no Indexing,...). This process had as commandline:

> 11164 "D:\NightlybuildArduino\Arduino IDE.exe" --type=renderer --user-data-dir="C:\Users\Johan\AppData\Roaming\arduino-ide" --app-path="D:\NightlybuildArduino\resources\app" --no-sandbox --no-zygote --field-trial-handle=1588,2066934688397159316,1515606472287176484,131072 --disable-features=SpareRendererForSitePerProcess,WinRetrieveSuggestionsOnlyOnDemand --lang=en-GB --device-scale-factor=1 --num-raster-threads=4 --enable-main-frame-before-activation --renderer-client-id=4 --no-v8-untrusted-code-mitigations --mojo-platform-channel-handle=2516 /prefetch:1

That is it for today...

[InstantMuffin]

I think this may be helpful.
I just booted up the IDE and in the first few seconds autosuggestions worked fine, and there was no load.
I could see "indexing" in the bottom left corner. I'm not sure if this happened at one of the I assume files it was indexing (at a specific count) but as soon as it was done the load went up to 150% CPU.
It worked fine throughout at least part of the indexing process, then suddenly nothing and high load.
This all happened in less than 10 seconds of starting.

[xinkiknix]

I have been testing the latest nightly build (02/08) and am satisfied with the behaviour, Things run smoothly with very low CPU usage. I still have CC1plus.exe and xtensa-esp32-elf-g++.exe for the respective boards (as they are not digitally signed) as exceptions in my AV.

For me this looks it behaves as one would expect.

[InstantMuffin]

I have been testing rc9.1 for a short while now. I usually expect slowdowns after less than 10 minutes of use.
It has been more than half an hour, with much more time running the IDE in the background.

I will be using Arduino IDE during the next days and will report back if any slowdowns occur.
But honestly as far as this looks and feels to me (so odd to see how many features the IDE has now that slowdowns don't cut them out) this is fixed.

If you don't hear back from me until Friday, this is completely gone on my end.

I am very happy with this. Thank you for your continued efforts.

[kittaakos]

I would like to say thank you to all who helped us with the verification and testing of the Arduino language features. You're great. 🙏

I will keep this issue open for some time and will close later when no critical issues pop up.

---

language server does write the compiled data to the hard drive which on-access AV software generally checks.

This is done by the Arduino CLI.

Unfortunately I can't tell you where the cache is built on windows. I would suspect either in the temp folder or somewhere in the appdata directory structure.

Here is the code.

An example URI of a temp folder

• on macOS:

  file:///private/var/folders/z1/xkw1yh5n7rz4n8djprp1mdn80000gn/T/arduino-sketch-7D2D66808EB316067632A2C48E34D662
  

• on Windows:

  file:///c%3A/Users/kittaakos/AppData/Local/Temp/arduino-sketch-7320986B49B567949951B77342A53F22
  

⚠️ Please perform any antivirus adjustments at your own risk.

[Wezz19]

Indeed, the latest nightly (20220805) runs very well for me now also. I've been testing it for a few hours today, on the same weak laptop as in December.

Memory usage for the IDE always stays below 500MB and CPU under 5%. A huge improvement!

@kittaakos your latest fixes seems to have been a success! Many thanks! The IDE 2.0 is good for daily use now =D.
May I ask what exactly in this PR was the problem? Did some task queue up for each keypress?

By the way, is it possible to manually trigger an indexing without doing a verify? Sometimes they don't seem to trigger. Not a big issue though, since it happens very rarely.

[aliphys]

I have been using the nightly version, and it has been stable when it comes to CPU/RAM usage. Thanks! cc @per1234
#1240

[Micahfriesen]

Everything is very fast now, great work!

[InstantMuffin]

After almost a week with hours of use I can now definitely confirm that any performance issue is gone for me!

[DaleSchultz]

well, I just installed IDE 2 official release

Editor is unusable, just clicking somewhere takes about 1 second for the cursor to appear! Typing some text (even comment text) has a terrible lag and some characters even get lost.
Ran IDE 1.8.16 and it works fine, responsive.
I have not yet done any folder exclusion in the AV or turned language handling off.

If I create a new sketch and type junk into a comment I see Indexing n/85 alternating with "Building sketch" and this is with essentially an empty sketch.

Task manager shows IDE using 660Mb of memory with an empty sketch.
System:
Device name -------
Processor Intel(R) Core(TM) i3-10105 CPU @ 3.70GHz 3.70 GHz
Installed RAM 8.00 GB (7.74 GB usable)
Device ID ----------------------------
Product ID ------------
System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display

Edition Windows 11 Home
Version 21H2
Installed on ‎12/‎2/‎2021
OS build 22000.978
Experience Windows Feature Experience Pack 1000.22000.978.0

[DaleSchultz]

Today I excluded the temp file folder where the temp files are written and that made the IDE responsive again. We absolutely need an option to reindex on demand rather than on every keystroke. Most users are not going to be able to find this thread and parse out what has to be done to make the IDE usable.

[StefanL38]

I'm guessing all the fixes made here in the pre-release-version did
NOT find their way into the "official release"
@arduino development-team: did you ever notice that you are no longer in a small village and private garage-team just annoying some students of the same class with amateur errors? You are in a worldwide public.

Yes all this is voluntary and for free. This means there is absolutely no pressure for a certain release-date. Go on testing un-released beta-testing until the more special systems like DaleSchultz ones reports "runs flawlessly"

[per1234]

@StefanL38 I already explained to you that this sort of evidence free wild speculation is not appropriate here.

The code is open for anyone to see exactly what is in the release. If you have clear evidence that a fix somehow went missing, then you can provide that evidence. If you don't have that evidence, then you would be better to keep quiet because you only waste all of our time and make yourself look the fool.

[JiriBilek]

For me the Arduino IDE 2.0 works fine. No lagging. I am using it with ESP32 toolchain.

Edit: Windows 10 Pro 21H2, Intel i5-8265U CPU, 24 GB RAM

[FastThenLeft]

As a data point, I had the original problem with performance issues the more I used it. But, since rc9.1, every release has been running fast using minimal system resources on my 11-year-old laptop. I also have an entire college class using V2.0.0 with no performance issues.

[xinkiknix]

I am happy with version 2.0.0 no (major) issues, excellent performance. Using multiple instances with loads of tabs and no performance impact. Fit for purpose!!!