Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImportError – name clash when debugging MacOS Kernel with LLDB #295

Open
golddranks opened this issue Apr 10, 2022 · 11 comments
Open

ImportError – name clash when debugging MacOS Kernel with LLDB #295

golddranks opened this issue Apr 10, 2022 · 11 comments

Comments

@golddranks
Copy link

golddranks commented Apr 10, 2022 •
edited
Loading

I'm trying to remote debug MacOS kernel with LLDB. I have Voltron and Apple's Kernel Debug Kit installed.

In LLDB, first I create the target:

target create /Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel

After that, once I set it to load scripts:

settings set target.load-script-from-symbol-file true

Then, it crashes with

error: module importing failed: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/xnu.py", line 1286, in <module>
    from misc import *
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/misc.py", line 8, in <module>
    from scheduler import *
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/scheduler.py", line 1035, in <module>
    from kevent import GetKnoteKqueue
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/kevent.py", line 3, in <module>
    from memory import vm_unpack_pointer
ImportError: cannot import name 'vm_unpack_pointer' from 'memory' (/Users/kon/Library/Python/3.8/lib/python/site-packages/voltron/plugins/api/memory.py)

The problem seems to be that a file
/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/kevent.py tries to import stuff from memory, where the intended file for that is /Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/memory.py

However, for some reason, it actually accesses Voltron's memory.py: /Users/me/Library/Python/3.8/lib/python/site-packages/voltron/plugins/api/memory.py

I am not very well-versed in Pythons modules and import rules, and especially I don't know how LLDB loads scripts, but to me, this seems like something that is not normally supposed to happen. Is this just unfortunate name clash, or is something broken in my system?
@akx
Copy link

akx commented Apr 11, 2022

This seems like 'voltron/plugins/api' has ended up in the Python interpreter's sys.path, so all absolute imports end up being imported from there.

However, looking at the other imports in the traceback, it seems like lldbmacros expects itself to be the first in path anyway (otherwise the other imports would be from lldbmacros.scheduler ... or from .scheduler ... (or it's Python 2, which evidently it isn't).

IOW, print(sys.path) – what does it look like?

@golddranks
Copy link
Author

So I snuck a print statement in the start of xnu.py which was at the bottom of the callstack mentioned above. It says:

['/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A',
'/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros',
'/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python',
'/Users/kon/Library/Python/3.8/lib/python/site-packages/voltron',
'/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/kon/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
'.',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']

For reference, here's sys.path printed from running python3 normally (I have only the Python installed by XCode CLI tools currently linked in my system; LLDB uses that.):

['',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/kon/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages']

So, LLDB (or some of the scripts?) seems to tweak sys.path?

Also, it should be noted that the KDK lldbmacros are converted from Python 2 to Python 3 by myself:

sudo 2to3 -w /Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/

...this is because LLDB doesn't seem to support Python 2 anymore, but the KDKs for older kernels are just that.

It's slightly cursed, but it used to somehow work before I installed Voltron. Checking the code, it seems that 2to3 doesn't convert the relative imports like from .scheduler import *, they are like from scheduler import *.

@akx
Copy link

akx commented Apr 11, 2022 •
edited
Loading

Yeah,

/Users/kon/Library/Python/3.8/lib/python/site-packages/voltron'

should absolutely not be on sys.path by itself.

It's enough that site-packages is, since that will allow you to import voltron... or from voltron... import ..., but having the voltron package's directory there too will (and does) cause chaos.

If you can find out why it's there, that'd be the next step.

  • probably an invocation related to sys.path somewhere, likely in Voltron itself
  • Another option is that there's a .pth file that refers the voltron package directory; the site machinery in Python scans those and puts them on sys.path.

  • voltron/install.sh

    Lines 271 to 274 in d9fef0b

    if [ -n "${VENV}" ]; then
    echo "script import sys;sys.path.append('${LLDB_SITE_PACKAGES}')" >> ${LLDB_INIT_FILE}
    fi
    echo "command script import $LLDB_ENTRY_FILE" >> ${LLDB_INIT_FILE}
    seems interesting too; maybe look at your lldb init file to see if there's the append invocation there

@golddranks
Copy link
Author

My $HOME/.lldbinit contains just

command script import /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron/entry.py

and manually tweaking /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron/entry.py to print sys.path at start yields:

['/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A',
'/Users/kon/Library/Python/3.8/lib/python/site-packages/voltron',
'/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/kon/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
'.']

so the voltron path is already there before the voltron entry script is called.

Also, no .pth files in site-packages or the voltron subdir. And quick grepping the package doesn't reveal anything else related to sys.path but entry.py adding some stuff here: https://github.com/snare/voltron/blob/master/voltron/entry.py#L31 , but that's after I printed the sys.path that already contained the voltron path.

@akx
Copy link

akx commented Apr 11, 2022

For one, I don't think that linked segment in voltron/entry.py should be there at all in 2022, since it's all just Python 2.7 fluff.

What if you do command script import sys;print("Honk!", sys.path) in .lldbinit?

@golddranks
Copy link
Author

golddranks commented Apr 11, 2022 •
edited
Loading

We're getting closer! /Users/me/entry.py is an one-liner that prints the current sys.path. .lldbinit:

command script import sys;print(sys.path)
command script import /Users/me/entry.py

['/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A',
'/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/me/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
'.']
my entry:  ['/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A',
'/Users/me',
'/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/me/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
'.']

So, it seems like the culprit is LLDB...?

@akx
Copy link

akx commented Apr 11, 2022

Ohhhhh, now I get it! Yep - so clearly command script import foo/baz.py adds foo to sys.path, then imports baz. (Makes sense.)

command script import /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron/entry.py
command script 'import sys; sys.path[:] = [p for p in sys.path if not p.endswith("/voltron")]'

or similar should fix that up... 😁

@golddranks
Copy link
Author

golddranks commented Apr 12, 2022 •
edited
Loading

That would work, if it would accept any kind of python statements, but seems the expression you can use there has some rather haphazard limitations. First of all, import is part of the LLDB command, and it is required. After that, it seems to allow only one space directly after, and no whitespace after that. Clearly the user is meant just to pass in a path there, but it doesn't bother validating the input properly. I was able to delete the path with

command script import sys;sys.path=sys.path[:1]+sys.path[2:]

But that depends on the order of the path and feels like it's from a code golf competition. I also tried calling a script that deletes the path, but that doesn't seems to affect the path outside of the said script.

It would help if voltron could be imported not as a script file entry.py, but as a package. I tried to fix it by creating a new wrapper:

mkdir /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron_lldb
echo "from voltron import entry" > /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron_lldb/__init__.py
echo "command script import /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron_lldb" > $HOME/.lldbinit

And it ALMOST works, no /Users/kon/Library/Python/3.8/lib/python/site-packages/voltron to be seen!

command script import sys;print(sys.path)
['/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A',
'/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros',
'/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python',
'/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python38.zip',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/lib-dynload',
'/Users/kon/Library/Python/3.8/lib/python/site-packages',
'/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages',
'.',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload']

But no cigar:

(lldb) target create /Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel
(lldb) settings set target.load-script-from-symbol-file true

error: module importing failed: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/xnu.py", line 1287, in <module>
    from misc import *
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/misc.py", line 8, in <module>
    from scheduler import *
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/scheduler.py", line 1035, in <module>
    from kevent import GetKnoteKqueue
  File "/Library/Developer/KDKs/KDK_11.6.5_20G527.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/Python/lldbmacros/kevent.py", line 3, in <module>
    from memory import vm_unpack_pointer
ImportError: cannot import name 'vm_unpack_pointer' from 'memory' (/Users/kon/Library/Python/3.8/lib/python/site-packages/voltron/plugins/api/memory.py)

WHY? \:D/ I have no idea. (The same thing happens both with the wrapper, and the path deletion method.)

@golddranks
Copy link
Author

Could it be that it caches something? And now that you mention it, how does it even find memory.py from a subdirectory of voltron, plugins/api/memory.py? Btw. I renamed that file to voltron_memory.py, and it found another one from plugins/view/memory.py. KDK started working only after renaming those two...

@akx
Copy link

akx commented Apr 12, 2022 •
edited
Loading

Oh, right, yeah, you'd maybe also need del sys.modules['memory'] to clear Python's memory of having imported that module as memory...

@day0n
Copy link

day0n commented Apr 30, 2022

There is someone upadate?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@akx @golddranks @day0n