Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover #67217

Closed
VincentBu opened this issue Mar 28, 2022 · 7 comments
Closed

Test failure BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover #67217

VincentBu opened this issue Mar 28, 2022 · 7 comments
Labels
arch-x64 area-System.Diagnostics.Tracing JitStress CLR JIT issues involving JIT internal stress modes os-windows

Comments

@VincentBu
Copy link
Contributor

Run: runtime-coreclr libraries-jitstress 20220327.1

Failed test:

net7.0-windows-Release-x64-CoreCLR_checked-jitstress1-Windows.10.Amd64.Open

- BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover

Error message:

Half-way through waiting for remote process.
Memory load: 14

Image Name                     PID Services

========================= ======== ============================================
svchost.exe                    764 BrokerInfrastructure, DcomLaunch, LSM,

PlugPlay, Power, SystemEventsBroker

svchost.exe                    824 RpcEptMapper, RpcSs

svchost.exe                    940 Appinfo, CertPropSvc, DsmSvc, gpsvc,

iphlpsvc, lfsvc, ProfSvc, sacsvr, Schedule,
SENS, SessionEnv, ShellHWDetection, Themes,
UserManager, UsoSvc, Winmgmt, wlidsvc,

WpnService, wuauserv

svchost.exe                    952 TermService

svchost.exe                    336 Dhcp, EventLog, lmhosts, NgcCtnrSvc,

TimeBrokerSvc, vmictimesync

svchost.exe                    424 NcbService, PcaSvc, TrkWks, UALSVC,

UmRdpService, vmickvpexchange,

vmicshutdown, WdiSystemHost

svchost.exe                   1084 CDPSvc, EventSystem, FontCache, netprofm,

nsi, RemoteRegistry, W32Time,

WinHttpAutoProxySvc

svchost.exe                   1100 vmicheartbeat

svchost.exe                   1140 BFE, CoreMessagingRegistrar, DPS, MpsSvc,

pla

svchost.exe                   1348 CryptSvc, Dnscache, LanmanWorkstation,

NlaSvc, WinRM

svchost.exe                   1416 Wcmsvc

svchost.exe                   2028 AppHostSvc

svchost.exe                   1408 StateRepository, tiledatamodelsvc

svchost.exe                   1892 W3SVC, WAS

svchost.exe                   1932 DiagTrack

svchost.exe                   2108 LanmanServer

svchost.exe                   3636 CDPUserSvc_4c08c, OneSyncSvc_4c08c

svchost.exe                   4036 AppReadiness

svchost.exe                   4092 ClipSVC

Timed out at 3/27/2022 8:18:29 AM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\B37C0968\w\A069088A\uploads\3040.vq1og4mr.ajy.dmp
Process ID: 3040
Handle: 1092
Name: dotnet
MainModule: C:\h\w\B37C0968\p\dotnet.exe
StartTime: 3/27/2022 8:17:02 AM
TotalProcessorTime: 00:00:02.3750000
Threads:
Thread #1 (OS 0xBDC)   [MTA]
[HelperMethodFrame] (System.Threading.Thread.SleepInternal)
System.Threading.Thread.Sleep(Int32)
BasicEventSourceTests.TestsManifestGeneration.<Test_EventSource_EtwManifestGenerationRollover>b__6_0()
[DebuggerU2MCatchHandlerFrame]
[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
Thread #2 (OS 0x158C) [Finalizer] [Background]
[DebuggerU2MCatchHandlerFrame]


Stack trace
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 224
@VincentBu VincentBu added os-windows JitStress CLR JIT issues involving JIT internal stress modes arch-x64 labels Mar 28, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 28, 2022
@ghost
Copy link

ghost commented Mar 28, 2022

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-coreclr libraries-jitstress 20220327.1

Failed test:

net7.0-windows-Release-x64-CoreCLR_checked-jitstress1-Windows.10.Amd64.Open

- BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover

Error message:

Half-way through waiting for remote process.
Memory load: 14

Image Name                     PID Services

========================= ======== ============================================
svchost.exe                    764 BrokerInfrastructure, DcomLaunch, LSM,

PlugPlay, Power, SystemEventsBroker

svchost.exe                    824 RpcEptMapper, RpcSs

svchost.exe                    940 Appinfo, CertPropSvc, DsmSvc, gpsvc,

iphlpsvc, lfsvc, ProfSvc, sacsvr, Schedule,
SENS, SessionEnv, ShellHWDetection, Themes,
UserManager, UsoSvc, Winmgmt, wlidsvc,

WpnService, wuauserv

svchost.exe                    952 TermService

svchost.exe                    336 Dhcp, EventLog, lmhosts, NgcCtnrSvc,

TimeBrokerSvc, vmictimesync

svchost.exe                    424 NcbService, PcaSvc, TrkWks, UALSVC,

UmRdpService, vmickvpexchange,

vmicshutdown, WdiSystemHost

svchost.exe                   1084 CDPSvc, EventSystem, FontCache, netprofm,

nsi, RemoteRegistry, W32Time,

WinHttpAutoProxySvc

svchost.exe                   1100 vmicheartbeat

svchost.exe                   1140 BFE, CoreMessagingRegistrar, DPS, MpsSvc,

pla

svchost.exe                   1348 CryptSvc, Dnscache, LanmanWorkstation,

NlaSvc, WinRM

svchost.exe                   1416 Wcmsvc

svchost.exe                   2028 AppHostSvc

svchost.exe                   1408 StateRepository, tiledatamodelsvc

svchost.exe                   1892 W3SVC, WAS

svchost.exe                   1932 DiagTrack

svchost.exe                   2108 LanmanServer

svchost.exe                   3636 CDPUserSvc_4c08c, OneSyncSvc_4c08c

svchost.exe                   4036 AppReadiness

svchost.exe                   4092 ClipSVC

Timed out at 3/27/2022 8:18:29 AM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\B37C0968\w\A069088A\uploads\3040.vq1og4mr.ajy.dmp
Process ID: 3040
Handle: 1092
Name: dotnet
MainModule: C:\h\w\B37C0968\p\dotnet.exe
StartTime: 3/27/2022 8:17:02 AM
TotalProcessorTime: 00:00:02.3750000
Threads:
Thread #1 (OS 0xBDC)   [MTA]
[HelperMethodFrame] (System.Threading.Thread.SleepInternal)
System.Threading.Thread.Sleep(Int32)
BasicEventSourceTests.TestsManifestGeneration.<Test_EventSource_EtwManifestGenerationRollover>b__6_0()
[DebuggerU2MCatchHandlerFrame]
[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
Thread #2 (OS 0x158C) [Finalizer] [Background]
[DebuggerU2MCatchHandlerFrame]


Stack trace
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 224
Author: VincentBu
Assignees: -
Labels:

area-System.Diagnostics.Tracing, os-windows, JitStress, arch-x64, untriaged
Milestone: -

@tommcdon
Copy link
Member

@BruceForstall this seems to be an issue caused by JIT stress, could you do a first-level triage of the test failure?

@tommcdon tommcdon added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 29, 2022
@ghost
Copy link

ghost commented Mar 29, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-coreclr libraries-jitstress 20220327.1

Failed test:

net7.0-windows-Release-x64-CoreCLR_checked-jitstress1-Windows.10.Amd64.Open

- BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover

Error message:

Half-way through waiting for remote process.
Memory load: 14

Image Name                     PID Services

========================= ======== ============================================
svchost.exe                    764 BrokerInfrastructure, DcomLaunch, LSM,

PlugPlay, Power, SystemEventsBroker

svchost.exe                    824 RpcEptMapper, RpcSs

svchost.exe                    940 Appinfo, CertPropSvc, DsmSvc, gpsvc,

iphlpsvc, lfsvc, ProfSvc, sacsvr, Schedule,
SENS, SessionEnv, ShellHWDetection, Themes,
UserManager, UsoSvc, Winmgmt, wlidsvc,

WpnService, wuauserv

svchost.exe                    952 TermService

svchost.exe                    336 Dhcp, EventLog, lmhosts, NgcCtnrSvc,

TimeBrokerSvc, vmictimesync

svchost.exe                    424 NcbService, PcaSvc, TrkWks, UALSVC,

UmRdpService, vmickvpexchange,

vmicshutdown, WdiSystemHost

svchost.exe                   1084 CDPSvc, EventSystem, FontCache, netprofm,

nsi, RemoteRegistry, W32Time,

WinHttpAutoProxySvc

svchost.exe                   1100 vmicheartbeat

svchost.exe                   1140 BFE, CoreMessagingRegistrar, DPS, MpsSvc,

pla

svchost.exe                   1348 CryptSvc, Dnscache, LanmanWorkstation,

NlaSvc, WinRM

svchost.exe                   1416 Wcmsvc

svchost.exe                   2028 AppHostSvc

svchost.exe                   1408 StateRepository, tiledatamodelsvc

svchost.exe                   1892 W3SVC, WAS

svchost.exe                   1932 DiagTrack

svchost.exe                   2108 LanmanServer

svchost.exe                   3636 CDPUserSvc_4c08c, OneSyncSvc_4c08c

svchost.exe                   4036 AppReadiness

svchost.exe                   4092 ClipSVC

Timed out at 3/27/2022 8:18:29 AM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\B37C0968\w\A069088A\uploads\3040.vq1og4mr.ajy.dmp
Process ID: 3040
Handle: 1092
Name: dotnet
MainModule: C:\h\w\B37C0968\p\dotnet.exe
StartTime: 3/27/2022 8:17:02 AM
TotalProcessorTime: 00:00:02.3750000
Threads:
Thread #1 (OS 0xBDC)   [MTA]
[HelperMethodFrame] (System.Threading.Thread.SleepInternal)
System.Threading.Thread.Sleep(Int32)
BasicEventSourceTests.TestsManifestGeneration.<Test_EventSource_EtwManifestGenerationRollover>b__6_0()
[DebuggerU2MCatchHandlerFrame]
[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
Thread #2 (OS 0x158C) [Finalizer] [Background]
[DebuggerU2MCatchHandlerFrame]


Stack trace
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 224
Author: VincentBu
Assignees: -
Labels:

area-System.Diagnostics.Tracing, os-windows, JitStress, arch-x64, area-CodeGen-coreclr, untriaged
Milestone: -

@BruceForstall
Copy link
Member

@tommcdon I couldn't repro locally with or without stress. And the lab job didn't repro on the next two runs.

My Kusto queries aren't working to find the full test history for this, unfortunately.

Most likely, it is not JitStress related, but because JitStress runs cause tests to run much more frequently (due to a large number of JitStress variations), any existing timing or other "flakiness" in the test or system is made more visible in JitStress jobs.

@BruceForstall BruceForstall added area-System.Diagnostics.Tracing and removed area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI labels Mar 30, 2022
@ghost
Copy link

ghost commented Mar 31, 2022

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-coreclr libraries-jitstress 20220327.1

Failed test:

net7.0-windows-Release-x64-CoreCLR_checked-jitstress1-Windows.10.Amd64.Open

- BasicEventSourceTests.TestsManifestGeneration.Test_EventSource_EtwManifestGenerationRollover

Error message:

Half-way through waiting for remote process.
Memory load: 14

Image Name                     PID Services

========================= ======== ============================================
svchost.exe                    764 BrokerInfrastructure, DcomLaunch, LSM,

PlugPlay, Power, SystemEventsBroker

svchost.exe                    824 RpcEptMapper, RpcSs

svchost.exe                    940 Appinfo, CertPropSvc, DsmSvc, gpsvc,

iphlpsvc, lfsvc, ProfSvc, sacsvr, Schedule,
SENS, SessionEnv, ShellHWDetection, Themes,
UserManager, UsoSvc, Winmgmt, wlidsvc,

WpnService, wuauserv

svchost.exe                    952 TermService

svchost.exe                    336 Dhcp, EventLog, lmhosts, NgcCtnrSvc,

TimeBrokerSvc, vmictimesync

svchost.exe                    424 NcbService, PcaSvc, TrkWks, UALSVC,

UmRdpService, vmickvpexchange,

vmicshutdown, WdiSystemHost

svchost.exe                   1084 CDPSvc, EventSystem, FontCache, netprofm,

nsi, RemoteRegistry, W32Time,

WinHttpAutoProxySvc

svchost.exe                   1100 vmicheartbeat

svchost.exe                   1140 BFE, CoreMessagingRegistrar, DPS, MpsSvc,

pla

svchost.exe                   1348 CryptSvc, Dnscache, LanmanWorkstation,

NlaSvc, WinRM

svchost.exe                   1416 Wcmsvc

svchost.exe                   2028 AppHostSvc

svchost.exe                   1408 StateRepository, tiledatamodelsvc

svchost.exe                   1892 W3SVC, WAS

svchost.exe                   1932 DiagTrack

svchost.exe                   2108 LanmanServer

svchost.exe                   3636 CDPUserSvc_4c08c, OneSyncSvc_4c08c

svchost.exe                   4036 AppReadiness

svchost.exe                   4092 ClipSVC

Timed out at 3/27/2022 8:18:29 AM after 60000ms waiting for remote process.
Wrote mini dump to: C:\h\w\B37C0968\w\A069088A\uploads\3040.vq1og4mr.ajy.dmp
Process ID: 3040
Handle: 1092
Name: dotnet
MainModule: C:\h\w\B37C0968\p\dotnet.exe
StartTime: 3/27/2022 8:17:02 AM
TotalProcessorTime: 00:00:02.3750000
Threads:
Thread #1 (OS 0xBDC)   [MTA]
[HelperMethodFrame] (System.Threading.Thread.SleepInternal)
System.Threading.Thread.Sleep(Int32)
BasicEventSourceTests.TestsManifestGeneration.<Test_EventSource_EtwManifestGenerationRollover>b__6_0()
[DebuggerU2MCatchHandlerFrame]
[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
Thread #2 (OS 0x158C) [Finalizer] [Background]
[DebuggerU2MCatchHandlerFrame]


Stack trace
   at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 224
Author: VincentBu
Assignees: -
Labels:

area-System.Diagnostics.Tracing, os-windows, JitStress, arch-x64, untriaged
Milestone: -

@tommcdon
Copy link
Member

tommcdon commented Apr 3, 2022

Closing as we were not able to repro the issue

@tommcdon tommcdon closed this as completed Apr 3, 2022
@ghost ghost locked as resolved and limited conversation to collaborators May 3, 2022
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label May 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-System.Diagnostics.Tracing JitStress CLR JIT issues involving JIT internal stress modes os-windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jeffhandley @danmoseley @BruceForstall @tommcdon @VincentBu