Integration CI Testing Failing on Multiple Branches

[dadams39]

Receiving Timeout Errors during one of the operation tests:

• Adjust CI Integration local testing documentation in Notion so that developer is able to do the following:
  • create a local `$HOME/.corso_test.toml" file
  • source the assumeRole.sh and be able to run all the CI_Tests locally
  • CLI Documentation Clean-UP #357
• Document/Adjust long-running processes of CI that are timing out
• Merge fixes into failing PRs.
  • Restore Policy options added #350
  • GC async population of m365 objects mail #332
  • [Minor] Rename cli flag folder to email-folder #343
  • ...and whatever additional branches that are affected

[ryanfkeepers]

Fwiw, the creation of a test file isn't necessary to run local integration tests. But I can see that the document lacks details either way. Adding some of that now.

[dadams39]

I get that it is not. I would imagine users with less experience with the project would have similar experiences.

[dadams39]

Findings

Fundamental failure was caused by the following:

• collectionChannelBufferSize = 1000
• Outlook not displaying all of the files in the web browser

During the CI tests, these invisible messages were being added to the collection and after reaching the 999th item, the process would hang and the tests would eventually timeout. Investigation is ongoing to understand why the BackupWriter didn't reduce the number of messages in the channel.

[ashmrtn]

Investigation is ongoing to understand why the BackupWriter didn't reduce the number of messages in the channel.

It's possible that (async) GC and BackupWriter were working on disjoint sets of folders and that caused a deadlock when the channel for ExchangeDataCollection got full. Kopia uses some (configurable) number of goroutines to upload items. Each goroutine works on a single file or directory at a time and processes it to completion before moving to another folder or directory. However, if the number of folders that need uploaded is larger than the number of goroutines it's possible that the GC goroutine(s) are adding items to folders that kopia is not currently uploading.

As an example, kopia may have 1 goroutine trying to upload items in the inbox directory. GC may have one goroutine adding items to the DataCollection for archive. However, because neither component will examine other directories before it's done with the current one, the system will deadlock if GC tries to fetch more messages than the channel size for the ExchangeDataCollection backing the archive directory

Increasing the number of goroutines kopia uses may help, but cannot guarantee a deadlock will not occur. This is because there can be an arbitrary number of folders in a backup

[ryanfkeepers]

Synchronous deadlock explanation:

SerializeMessages() loaded messages according to the following algorithm:

1. Retrieve all user's folders+messageIDs.
2. Aggregate messages by their folder id.
3. Iterating through each folder, download all messages in the folder, feeding each into a DataCollection channel.
4. Return the loaded set of folders as a slice of DataCollections for downstream consumers.

DataCollection channels used a buffer limit of 1000 entries. If any folder exceeded 1000 messages, the channel buffer would fill, blocking the function from continuing to load any further messages or folders, effectively locking the entire system.

[vkamra]

Addressing #360 will fix this issue. In addition to that fix - we should also reduce the scope of the Integration test(backup a specific folder only, restore as COPY in a restore- folder). The larger scoped, long-running tests should not run in CI.

[dadams39]

Explanation StrikeThrough of Document/Adjust long-running processes...

The major changes were #360 to deal with deadlock and a large refactor along the lines #361 that was implemented in stages. Both of those code changes are present in main as of today.

The final cause of tests timing out are to be addressed in PR #479

[dadams39]

CI documentation is a constantly improving process. Steps have been made to improve the initial setup of developer environments. While the process is not complete, the main objective of this issue was to address the cascading failures that were experienced several days ago. Those CI failures have been addressed or have a separate issue in the repository. The issue is to be closed.