Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EBPF trace sometimes fail to record packet type #1020

Open
wks opened this issue Nov 10, 2023 · 2 comments
Open

EBPF trace sometimes fail to record packet type #1020

wks opened this issue Nov 10, 2023 · 2 comments
Labels
P-low Priority: Low. A low-priority issue won't be scheduled and assigned. Any help is welcome.

Comments

@wks
Copy link
Collaborator

wks commented Nov 10, 2023

When using bpftrace, it sometimes cannot record the packet type of the trace point probe!(mmtk, work, typename.as_ptr(), typename.len()); The log output looks like this:

@type_name[140584722155381]: mmtk_openjdk::gc_work::ScanWeakProcessorRoots<mmtk_openjdk::Ope
@type_name[140584722159582]: mmtk_openjdk::gc_work::ScanManagementRoots<mmtk_openjdk::OpenJD
@type_name[140584722163486]: 
@type_name[140584722164295]: mmtk_openjdk::gc_work::ScanAOTLoaderRoots<mmtk_openjdk::OpenJDK
@type_name[140584722181767]: mmtk_openjdk::gc_work::ScanCodeCacheRoots<false, mmtk::schedule
@type_name[140584722182951]: mmtk_openjdk::gc_work::ScanManagementRoots<mmtk_openjdk::OpenJD

In the following bpftrace script, the output of str(arg0, arg1) is already an empty string.

usdt:/path/to/xxxxxx.so:mmtk:work {
    $work_name = str(arg0, arg1);
    ...
}

I even tried to print each individual character, but all characters are zero.

unroll(10) {
    printf("Char %d: %s\n", $i, str(arg0 + $i, 1));
    $i = $i + 1;
}

But if we print the string at address 140584722163486 in mmtk-core, it can properly print the string.

Rust:

            probe!(mmtk, work, typename.as_ptr(), typename.len());
            eprintln!("TYPENAME[{}]: {}", typename.as_ptr() as usize, typename);

Output:

TYPENAME[140584722163486]: mmtk_openjdk::gc_work::ScanUniverseRoots<mmtk_openjdk::OpenJDK<false>, mmtk::scheduler::gc_work::ProcessEdgesWorkRootsWorkFactory<mmtk_openjdk::OpenJDK<false>, mmtk::plan::generational::gc_work::GenNurseryProcessEdges<mmtk_openjdk::OpenJDK<false>, mmtk::plan::generational::immix::global::GenImmix<mmtk_openjdk::OpenJDK<false>>>, mmtk::scheduler::gc_work::UnsupportedProcessEdges<mmtk_openjdk::OpenJDK<false>>>>

The only thing special about this string, as I can observe, is that this type name is longer than others.

I can ignore this packet for now because it is not my concern at this moment. But it is interesting why we can't capture the packet name in bpftrace.

@udesou udesou added the P-low Priority: Low. A low-priority issue won't be scheduled and assigned. Any help is welcome. label Nov 13, 2023
@wks
Copy link
Collaborator Author

wks commented Nov 13, 2023

With bpftrace -kk, it prints out the following error message:

WARNING: Failed to probe_read_user_str: Bad address (-14)

That means the memory is likely not mapped.

One way to work it around is force reading from the typename:

            probe!(mmtk, work, typename.as_ptr(), typename.len(), *(typename.as_ptr()) as usize);

And the problem disappears.

According to Rust's API doc, std::any::type_name<T>() is currently implemented with "compiler diagnostics and debuginfo". I suspect the memory is mapped on an on-demand basis (but how?) and will not be available unless we load from it.

@wks
Copy link
Collaborator Author

wks commented Jan 10, 2024

PR #1022 uses the workaround mentioned in #1020 (comment) and it satisfies our current need. We may further investigate the root cause of this and find better solutions in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P-low Priority: Low. A low-priority issue won't be scheduled and assigned. Any help is welcome.
Projects
None yet
Development

No branches or pull requests

2 participants