-
Kirill Smelkov authored
Because of the way wendelin.core organizes its in-tree python importing redirector (see wendelin.py) it is possible to import the same module twice with python thinking it is importing two different modules. For example when installed in develop mode python resolves the following imports to the same bigfile/__init__.py import wendelin.bigfile import bigfile but tries to load that module twice and independently. Which leads to virtmem DSO, linked to from under bigfile/_bigfile extension, being initialized twice and complaining about that because only single gil hook should be requested to be installed: (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ python Python 3.9.19+ (heads/3.9:40d77b93672, Apr 12 2024, 06:40:05) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import wendelin.bigfile >>> import bigfile python: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed. Аварийный останов This problem was there from day 1, but it was not creating issues in practice because wendelin.core users do `import wendelin...` and there was also no problem with running pytest in the source tree. However with py39 and pytest8 we see that running pytest somehow started to unconditionally import things from under two namespaces which leads to inability to run tests even when instructing pytest to collect them via python-modules namespace instead of filesystem: (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic ======================== test session starts ======================== platform linux -- Python 3.9.19+, pytest-8.2.2, pluggy-1.5.0 -- /home/kirr/src/wendelin/venv/py39.venv/bin/python3.9 cachedir: .pytest_cache rootdir: /home/kirr/src/wendelin/wendelin.core configfile: pyproject.toml collecting ... python3.9: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed. Fatal Python error: Aborted Current thread 0x00007fa172a60740 (most recent call first): File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed File "<frozen importlib._bootstrap_external>", line 1173 in create_module File "<frozen importlib._bootstrap>", line 565 in module_from_spec File "<frozen importlib._bootstrap>", line 666 in _load_unlocked File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1007 in _find_and_load File "/home/kirr/src/wendelin/wendelin.core/bigfile/__init__.py", line 31 in <module> File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed File "<frozen importlib._bootstrap_external>", line 850 in exec_module File "<frozen importlib._bootstrap>", line 680 in _load_unlocked File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1007 in _find_and_load File "<frozen importlib._bootstrap>", line 1030 in _gcd_import File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1007 in _find_and_load File "<frozen importlib._bootstrap>", line 1030 in _gcd_import File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1007 in _find_and_load File "<frozen importlib._bootstrap>", line 1030 in _gcd_import File "/home/kirr/local/py3.9/lib/python3.9/importlib/__init__.py", line 127 in import_module File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/pathlib.py", line 591 in import_path File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/python.py", line 492 in importtestmodule ... File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/runner.py", line 567 in collect_one_node File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 837 in _collect_one_node File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 974 in genitems File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 811 in perform_collect File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_collection ... File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 178 in main File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 206 in console_main File "/home/kirr/src/wendelin/venv/py39.venv/bin/pytest", line 8 in <module> Аварийный останов (образ памяти сброшен на диск) This happens because wendelin.bigfile is importing wendelin.bigfile._bigfile as `from ._bigfile import ...` which under pytest leads to importing both wendelin.bigfile._bigfile and bigfile._bigfile and further conflicting when setting up GIL hooks. -> Fix this issue by avoiding relative imports and always referring to wendelin.core modules with `wendelin.` prefix. The list of places where relative imports were used was small and found via $ git grep -w import |grep '\s\.' bigfile/__init__.py:from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim wcfs/__init__.py:from .client._wcfs import \ Everywhere else we were already importing things from under wendelin namespace via fully specified module path. After the fix both $ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic and $ pytest -vsx bigfile/tests/test_basic.py start to work ok from inside the worktree. /reported-and-tested-by @vnmabus /reviewed-by @levin.zimmermann /reviewed-on nexedi/wendelin.core!26
3846997b