GitLab

Because of the way wendelin.core organizes its in-tree python importing redirector (see wendelin.py) it is possible to import the same module twice with python thinking it is importing two different modules. For example when installed in develop mode python resolves the following imports to the same bigfile/init.py

import wendelin.bigfile
import bigfile

but tries to load that module twice and independently. Which leads to virtmem DSO, linked to from under bigfile/_bigfile extension, being initialized twice and complaining about that because only single gil hook should be requested to be installed:

(py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ python
Python 3.9.19+ (heads/3.9:40d77b93672, Apr 12 2024, 06:40:05)
[GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wendelin.bigfile
>>> import bigfile
python: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
Аварийный останов

This problem was there from day 1, but it was not creating issues in practice because wendelin.core users do import wendelin... and there was also no problem with running pytest in the source tree.

However with py39 and pytest8 we see that running pytest somehow started to unconditionally import things from under two namespaces which leads to inability to run tests even when instructing pytest to collect them via python-modules namespace instead of filesystem:

(py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic
======================== test session starts ========================
platform linux -- Python 3.9.19+, pytest-8.2.2, pluggy-1.5.0 -- /home/kirr/src/wendelin/venv/py39.venv/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/kirr/src/wendelin/wendelin.core
configfile: pyproject.toml
collecting ... python3.9: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
Fatal Python error: Aborted

Current thread 0x00007fa172a60740 (most recent call first):
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1173 in create_module
  File "<frozen importlib._bootstrap>", line 565 in module_from_spec
  File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "/home/kirr/src/wendelin/wendelin.core/bigfile/__init__.py", line 31 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 850 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
  File "/home/kirr/local/py3.9/lib/python3.9/importlib/__init__.py", line 127 in import_module
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/pathlib.py", line 591 in import_path
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/python.py", line 492 in importtestmodule
  ...
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 837 in _collect_one_node
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 974 in genitems
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 811 in perform_collect
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_collection
  ...
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/home/kirr/src/wendelin/venv/py39.venv/bin/pytest", line 8 in <module>
Аварийный останов (образ памяти сброшен на диск)

This happens because wendelin.bigfile is importing wendelin.bigfile._bigfile as from ._bigfile import ... which under pytest leads to importing both wendelin.bigfile._bigfile and bigfile._bigfile and further conflicting when setting up GIL hooks.

-> Fix this issue by avoiding relative imports and always referring to wendelin.core modules with wendelin. prefix.

The list of places where relative imports were used was small and found via

$ git grep -w import |grep '\s\.'
bigfile/__init__.py:from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim
wcfs/__init__.py:from .client._wcfs import \

Everywhere else we were already importing things from under wendelin namespace via fully specified module path.

After the fix both

$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic

and

$ pytest -vsx bigfile/tests/test_basic.py

start to work ok from inside the worktree.

/reported-by @vnmabus
/cc @levin.zimmermann