Best Tag Parser configuration for multiple workspaces #6801

boocs · 2021-01-16T10:37:45Z

boocs
Jan 16, 2021

(Updated with new info)

@sean-mcmanus I tried to explain it better.

What is the expected way to setup browse.path includes for projects with multiple workspaces? I keep getting the feeling this is a bug so wanted to make sure.

Let's say we have 2 workspaces, in our project, Foo and Bar.

Foo is considered our main workspace and has a Foo.code-workspace:
It's setup like so:

"folders": [
		{
			"name": "Foo",
			"path": "."
		},
		{
			"name": "Bar",
			"path": "C:/Bar"
		}
	],
	"settings": {
		"typescript.tsc.autoDetect": "off"
	}

We have no files.exclude settings set anywhere.

First Configuration, we try, is named Separate and is configured like so:

Separate

Foo workspace browse.path setting:

"C_Cpp.default.browse.path": [
    "C:/Foo/src"
 ]

Bar workspace browse.path setting:

"C_Cpp.default.browse.path": [
    "C:/Bar/src"
 ]

Second Configuration is named Empty Array on Secondary Workspace and is configured like so:

Empty Array on Secondary Workspace

Foo workspace browse.path setting:

"C_Cpp.default.browse.path": [
    "C:/Foo/src",
    "C:/Bar/src"
 ]

Bar workspace browse.path setting:

"C_Cpp.default.browse.path": [ ]

Questions:

Which configuration are you suppose to be using? Separate or Empty Array on Secondary Workspace?
Should these configurations be equal?
Would it be considered a bug if Empty Array on Secondary Workspace is the better, by far, configuration?

Stats Using a Unreal Engine 4.26 Project that is not using Compile Commands:

Separate Configuration

Browse cache size ~1.6 GB
Find Symbols too slow to be usable
The Tag Parser statusBarItem icon appears often and for prolong periods when switching files/typing
Overall feels slow using anything Tag Parser related

Empty Array on Secondary Workspace Configuration

Seemingly no loss of functionality
Browse cache size ~0.9 GB
Find Symbols usable
Tag Parser statusBarItem doesn't appear or for short periods when switching files/typing
Tag Parser related operations feel a lot faster

This also holds true for UE 4.25 projects.

Answered by sean-mcmanus

Jan 19, 2021

The configurations should be identical if the files.exclude and other settings that affect the parsing are identical. I tried the repro with a simple project and my database sizes are identical. It's possible your 1st/FooIncludes workspace folder may have some files.exclude that the 2nd/BarIncludes project doesn't have. Can you open each folder as a single workspace folder with the combined browse path and see if they have the same size?

View full answer

sean-mcmanus · 2021-01-19T22:34:28Z

sean-mcmanus
Jan 19, 2021
Maintainer

The configurations should be identical if the files.exclude and other settings that affect the parsing are identical. I tried the repro with a simple project and my database sizes are identical. It's possible your 1st/FooIncludes workspace folder may have some files.exclude that the 2nd/BarIncludes project doesn't have. Can you open each folder as a single workspace folder with the combined browse path and see if they have the same size?

3 replies

boocs Jan 20, 2021
Author

I updated my post to reflect new info.

There are no files.exclude settings set.
I added info about Foo being main workspace with a Foo.code-workspace file.
I changed the name of Empty Array to Empty Array on Secondary Workspace Configuration

I tried another config based on your suggestion:

Third Configuration is named Empty Array on Primary Workspace and is configured like so:

Empty Array on Primary Workspace

Foo workspace browse.path setting:

"C_Cpp.default.browse.path": [ ]

Bar workspace browse.path setting:

"C_Cpp.default.browse.path": [ 
    "C:/Foo/src",
    "C:/Bar/src"
]

Using this on UE4 results in:
Tag Parser cache of ~1.6 GB

So it has to do with the *.code-workspace file/workspace or how it's configured?

I'll post some log info to see if there's anything. I'm waiting on the tag parser to finish.

boocs Jan 20, 2021
Author

Logs:

They do show the discrepancy.

This is for Unreal Engine 4.26:

Empty Array on Secondary Workspace

Discovering files: 77659 file(s) processed
  1 file(s) removed from database

Parsing: 31803 files(s) processed
Done parsing remaining files.

~0.9 GB Tag Parser Cache

Empty Array on Primary Workspace

Discovering files: 77659 file(s) processed
  0 file(s) removed from database

Parsing: 101238 files(s) processed
Done parsing remaining files.

~1.6 GB Tag Parser Cache

Info about settings:

Not much here:

Empty .vscode/settings.json in both workspaces

Both c_cpp_properties.json have these additional settings:

 name,
 intelliSenseMode,
 one forcedInclude with just #defines inside,
 compilerPath

*.code-workspace is shown in primary post
Global user setting doesn't have much: here's the part that may matter:

    "C_Cpp.intelliSenseCachePath": "K:\\.vscode\\cache\\default\\user",
    "C_Cpp.default.browse.databaseFilename": "K:\\.vscode\\cache\\browse\\${workspaceFolderBasename}\\user.vs.db",
    "C_Cpp.intelliSenseMemoryLimit": 6144,
    "C_Cpp.intelliSenseUpdateDelay": 1000,
    
    "C_Cpp.loggingLevel": "Debug"

I'll try what you said before and do only certain workspace includes and compare sizes.

boocs Jan 20, 2021
Author

Main Project Files Only

Basically the same configuration we have in the root post but we remove C:/Bar/src from browse.path

This shows the logs to a UE 4.26 project without UE4 source files in browse.path. Just our main project paths are there.

Logs

Main Project workspace (has browse.path) , UE4 workspace (empty array)

Discovering files: 9584 file(s) processed
  0 file(s) removed from database

Parsing: 77 files(s) processed
Done parsing remaining files.

2,384 KB File Size

Main Project workspace (Empty array) , UE4 workspace (has browse.path)

Discovering files: 9584 file(s) processed
  1 file(s) removed from database

Parsing: 51 files(s) processed
Done parsing remaining files.

2,388 KB File Size

tag parsing file: (logs)

Even though the first config shows 77 files(s) processed, only 52 show up in the 'tag parsing file:' log.
51 file path entries show up for the 2nd config's log matching the 51 files(s) processed.
The 1 file difference was a header file with just a comment and a #pragma once

Not enough files to tell but strange they're showing different logs with this amount of files.

Next up I'll do UE4 Source Files only in browse.path

boocs · 2021-01-20T11:50:35Z

boocs
Jan 20, 2021
Author

UE4 Project Files Only

The same configuration we have in the root post but we remove C:/Foo/src from browse.path

This shows the logs to a UE 4.26 project without Main Project source files in browse.path. Just our UE4 source paths are there.
Like the other examples, the UE 4.26 project doesn't use Compile Commands.

Logs

Main Project workspace (has browse.path) , UE4 workspace (empty array)

Discovering files: 77600 file(s) processed
  3 file(s) removed from database

Parsing: 31728 files(s) processed
Done parsing remaining files.

0.97 GB Cache

tag parsing file: has 31838 lines in the log

Main Project workspace (Empty array) , UE4 workspace (has browse.path)

Discovering files: 77600 file(s) processed
  0 file(s) removed from database

Parsing: 100427 files(s) processed
Done parsing remaining files.

1.59 GB Cache

tag parsing file: has 67488 lines in the log

More Settings Info

I used my extension to gather all the workspace configs for the UE 4.26 project.
I used JSON.stringify and created json files of all the setting objects.
I made sure to change one setting in the UE4 workspace to make sure everything was working.

I did a diff and all settings were the same except for that one setting.

Here are some of the settings I found:

"workspaceSymbols": "Just My Code",
"exclusionPolicy": "checkFolders",
..."limitSymbolsToIncludedHeaders": true,
"experimentalFeatures": "Disabled",
"intelliSenseEngine": "Default",
"intelliSenseEngineFallback": "Disabled",
"autocomplete": "Default"

(Update)
I just realized only diffed the "C_Cpp" section. Redoing diff.

Updated

Diffing all settings showed Launch and Tasks we're the only difference between Main Workspace settings and UE4 Workspace Settings.

files.exclude was set to:

"exclude": {
            "**/.git": true,
            "**/.svn": true,
            "**/.hg": true,
            "**/CVS": true,
            "**/.DS_Store": true
     }

0 replies

boocs · 2021-01-20T12:11:25Z

boocs
Jan 20, 2021
Author

@sean-mcmanus

I combined both workspace into one workspace and unfortunately it shows this:

Combine Workspace Main + UE4 source (Opened as a folder)

Discovering files: 72892 file(s) processed
  0 file(s) removed from database

Parsing: 100444 files(s) processed
Done parsing remaining files.

~1.6 GB cache size

67540 'tag parsing file:' lines in log

So if we go back and look at the Empty Array on Secondary Workspace Configuration, it seems like, for whatever reason, it's the best and only way to get better tag parser performance on large projects.

Maybe it's bugged somehow I'm not seeing. But the stats are so extreme that it needs to be looked at:

Empty Array on Secondary Workspace

Discovering files: 77659 file(s) processed
  1 file(s) removed from database

Parsing: 31803 files(s) processed
Done parsing remaining files.

~0.9 GB Tag Parser Cache

I also posted a bunch more stats about different configs and settings above.

0 replies

sean-mcmanus · 2021-01-21T01:19:22Z

sean-mcmanus
Jan 21, 2021
Maintainer

Ah yeah, the behavior you're seeing is because of the the "limitSymbolsToIncludedHeaders" setting, which is default to true, so in that case when it iterates over the browse.path for a particular workspace folder, the code that is external to the workspace folder will not get parsed unless it matches the name of a header that is included by another file in that workspace. So in your case, when C:/Bar/src is referenced in the 1st workspace folder, not all the files will get parsed, but when it's used by the 2nd folder it which it is no longer external, then all the files will get parsed. For the 3rd case, it seems like the same thing is happening (C:/Bar/src files are all getting parsed), and it looks like making C:/Foo/src external doesn't have much affect, maybe cause it doesn't include many headers? I didn't follow 100% of what your wrote, so let me know if I missed something.

4 replies

boocs Jan 21, 2021
Author

Yeah, that's about right. I confirmed it by using process monitor. Header files that weren't included in anything in the 'empty array config' workspace weren't even touched by cpptools. Header files that we're included in source files were tag parsed though.

So basically it's an obscure 'include what you use' tag parser configuration for a particular workspace?
Is there a way to get this as a global setting for every workspace?
a. It seems really useful with the reduction of cache size and performance improvements.
b. There's also no way for compile commands to make use of this setting.
c. Another reason would be if you have two big workspaces you would have to make a third dummy workspace to make it so both projects have of empty array in their browse.path.
Do you see any problems with this type of configuration except for not finding symbols in headers you don't include in source files?

sean-mcmanus Jan 21, 2021
Maintainer

Yeah.

a. It's already globally set to true, but you can change "C_Cpp.default.browse.limitSymbolsToIncludedHeaders", but there's no way currently to make a workspace not parse headers that are under the workspace folder (short of adding some files.exclude or moving the folders out of browse.path into a different folder). I've filed a feature request at #6827 .
b. Compile commands should still use the setting. Browse.path for compile commands is constructed by including all the folders for source files and headers referenced, so limitSymbolsToIncludedHeaders would still apply to folders outside of the workspace folder that is configured to use that compile_commands.json.
c. Instead of using a 3rd dummy workspace, you could just swap the browse.paths so that they both reference only external paths.
3. I don't see any problems.

boocs Jan 22, 2021
Author

I've filed a feature request at #6827 .
b. Compile commands should still use the setting.

Cool, thanks for the request!

I retested Compile Commands and you're right it does work. I had thought I tested this correctly when Compile Commands first came out. I probably didn't understand everything in the config and how it worked so I tested it wrong the first time.

Thanks for the help!

boocs Jan 22, 2021
Author

b. Compile commands should still use the setting.

Just for more thorough info:

You can only have one compile commands file if you don't want to tag parse every header in external workspaces.

If you use a compile command's file in any secondary workspaces, all headers in the paths(specified in the file) will get tag parsed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Best Tag Parser configuration for multiple workspaces #6801

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Best Tag Parser configuration for multiple workspaces #6801

boocs Jan 16, 2021

Separate

Empty Array on Secondary Workspace

Questions:

Separate Configuration

Empty Array on Secondary Workspace Configuration

Replies: 4 comments · 7 replies

sean-mcmanus Jan 19, 2021 Maintainer

boocs Jan 20, 2021 Author

Empty Array on Primary Workspace

So it has to do with the *.code-workspace file/workspace or how it's configured?

boocs Jan 20, 2021 Author

Logs:

Empty Array on Secondary Workspace

Empty Array on Primary Workspace

Info about settings:

boocs Jan 20, 2021 Author

Main Project Files Only

Logs

Main Project workspace (has browse.path) , UE4 workspace (empty array)

Main Project workspace (Empty array) , UE4 workspace (has browse.path)

tag parsing file: (logs)

boocs Jan 20, 2021 Author

UE4 Project Files Only

Logs

Main Project workspace (has browse.path) , UE4 workspace (empty array)

Main Project workspace (Empty array) , UE4 workspace (has browse.path)

More Settings Info

I did a diff and all settings were the same except for that one setting.

Updated

Diffing all settings showed Launch and Tasks we're the only difference between Main Workspace settings and UE4 Workspace Settings.

boocs Jan 20, 2021 Author

Combine Workspace Main + UE4 source (Opened as a folder)

Empty Array on Secondary Workspace

sean-mcmanus Jan 21, 2021 Maintainer

boocs Jan 21, 2021 Author

sean-mcmanus Jan 21, 2021 Maintainer

boocs Jan 22, 2021 Author

boocs Jan 22, 2021 Author

boocs
Jan 16, 2021

Replies: 4 comments 7 replies

sean-mcmanus
Jan 19, 2021
Maintainer

boocs Jan 20, 2021
Author

boocs Jan 20, 2021
Author

boocs Jan 20, 2021
Author

boocs
Jan 20, 2021
Author

boocs
Jan 20, 2021
Author

sean-mcmanus
Jan 21, 2021
Maintainer

boocs Jan 21, 2021
Author

sean-mcmanus Jan 21, 2021
Maintainer

boocs Jan 22, 2021
Author

boocs Jan 22, 2021
Author