Can't access user address

[hao-lee]

Here is a vma extracted from the smaps file. It is evident that this particular VMA contains 1942012kB of anonymous memory.

700000000-8003b7000 rw-p 00000000 00:00 0
Size:            4198108 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:             1942012 kB
Pss:             1942012 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1942012 kB
Referenced:      1942012 kB
Anonymous:       1942012 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
ProtectionKey:         0
VmFlags: rd wr mr mw me ac sd

Now, using drgn 0.0.24, we can scan each address within this VMA and count the number of addresses that can be successfully accessed.

task = find_task(prog, 2023533)
addr = 0x700000000
count = 0
while addr < 0x8003b7000:
	try:
		data = access_process_vm(task, addr, 4096)
		count += 1
	except:
		#print(hex(addr))
		pass
	addr += 4096

print(count)
print(count * 4096 / 1024/1024)

The result indicates a count of 20436, and the total accessible size is 79.828125MB, which is significantly smaller than the actual size.

I don't know how to solve this...

[osandov]

access_process_vm can only read pages that are actually resident in memory. The reads that fail are likely for pages that have either been swapped out or haven't even been populated yet.

If you want to read the memory of a process on a live system, try reading from /proc/$pid/mem. I believe that will fault in pages as needed.

[hao-lee]

@osandov

access_process_vm can only read pages that are actually resident in memory. The reads that fail are likely for pages that have either been swapped out or haven't even been populated yet.

If you want to read the memory of a process on a live system, try reading from /proc/$pid/mem. I believe that will fault in pages as needed.

Thanks for reply. But my system doesn't have swap and the kernel source says Rss: is the size of resident memory.

fs/proc/task_mmu.c:819:         SEQ_PUT_DEC("Rss:            ", mss->resident);

I think 1942012 kB are all resident pages. I also use /proc/pid/pagemap to verify the count of present pages in this vma range. It seems drgn indeed can't access many memory addresses even the page is present and can be access by crash command.

[osandov]

Huh, that does sound like a bug. I wasn't able to reproduce this on x86-64. What architecture and kernel version are you running?

[hao-lee]

Huh, that does sound like a bug. I wasn't able to reproduce this on x86-64. What architecture and kernel version are you running?

kernel version: 4.18.0-193.6.3.el8_2
arch: x86_64
CPU: Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz

I've tested multiple machines, and they all seem to have this issue, particularly when the process involves heavy memory usage. For example, I'm using drgn to scan a process with an Rss of 90G, but it can only access a small fraction of the memory pages. How can I assist in analyzing this problem?

[osandov]

Could you please run this script on one of these processes and share the output: https://gist.github.com/osandov/12d524c7c5cebfda2593b29e0b564234

Pass it the PID and optionally which address range to check (or leave off the addresses to check the entire process address space). So for your initial report, it'd look like:

sudo drgn ./test_access_remote_vm.py 2023533 0x700000000 0x8003b7000