Commit 3846997b authored by Kirill Smelkov's avatar Kirill Smelkov

*: Do not use relative imports

Because of the way wendelin.core organizes its in-tree python importing
redirector (see wendelin.py) it is possible to import the same module
twice with python thinking it is importing two different modules. For
example when installed in develop mode python resolves the following
imports to the same bigfile/__init__.py

    import wendelin.bigfile
    import bigfile

but tries to load that module twice and independently. Which leads to
virtmem DSO, linked to from under bigfile/_bigfile extension, being
initialized twice and complaining about that because only single gil
hook should be requested to be installed:

    (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ python
    Python 3.9.19+ (heads/3.9:40d77b93672, Apr 12 2024, 06:40:05)
    [GCC 12.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import wendelin.bigfile
    >>> import bigfile
    python: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
    Аварийный останов

This problem was there from day 1, but it was not creating issues in
practice because wendelin.core users do `import wendelin...` and there
was also no problem with running pytest in the source tree.

However with py39 and pytest8 we see that running pytest somehow started
to unconditionally import things from under two namespaces which leads
to inability to run tests even when instructing pytest to collect them
via python-modules namespace instead of filesystem:

    (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic
    ======================== test session starts ========================
    platform linux -- Python 3.9.19+, pytest-8.2.2, pluggy-1.5.0 -- /home/kirr/src/wendelin/venv/py39.venv/bin/python3.9
    cachedir: .pytest_cache
    rootdir: /home/kirr/src/wendelin/wendelin.core
    configfile: pyproject.toml
    collecting ... python3.9: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.
    Fatal Python error: Aborted

    Current thread 0x00007fa172a60740 (most recent call first):
      File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
      File "<frozen importlib._bootstrap_external>", line 1173 in create_module
      File "<frozen importlib._bootstrap>", line 565 in module_from_spec
      File "<frozen importlib._bootstrap>", line 666 in _load_unlocked
      File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
      File "/home/kirr/src/wendelin/wendelin.core/bigfile/__init__.py", line 31 in <module>
      File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
      File "<frozen importlib._bootstrap_external>", line 850 in exec_module
      File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
      File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
      File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
      File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
      File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
      File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 972 in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
      File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
      File "/home/kirr/local/py3.9/lib/python3.9/importlib/__init__.py", line 127 in import_module
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/pathlib.py", line 591 in import_path
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/python.py", line 492 in importtestmodule
      ...
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/runner.py", line 567 in collect_one_node
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 837 in _collect_one_node
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 974 in genitems
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 811 in perform_collect
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_collection
      ...
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 178 in main
      File "/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 206 in console_main
      File "/home/kirr/src/wendelin/venv/py39.venv/bin/pytest", line 8 in <module>
    Аварийный останов (образ памяти сброшен на диск)

This happens because wendelin.bigfile is importing
wendelin.bigfile._bigfile as `from ._bigfile import ...` which under
pytest leads to importing both wendelin.bigfile._bigfile and
bigfile._bigfile and further conflicting when setting up GIL hooks.

-> Fix this issue by avoiding relative imports and always referring to
   wendelin.core modules with `wendelin.` prefix.

The list of places where relative imports were used was small and found via

    $ git grep -w import |grep '\s\.'
    bigfile/__init__.py:from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim
    wcfs/__init__.py:from .client._wcfs import \

Everywhere else we were already importing things from under wendelin
namespace via fully specified module path.

After the fix both

    $ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic

and

    $ pytest -vsx bigfile/tests/test_basic.py

start to work ok from inside the worktree.

/reported-and-tested-by @vnmabus
/reviewed-by @levin.zimmermann
/reviewed-on nexedi/wendelin.core!26
parent 37cf1383
# -*- coding: utf-8 -*-
# BigFile submodule for Wendelin
# Copyright (C) 2014-2021 Nexedi SA and Contributors.
# Copyright (C) 2014-2024 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
......@@ -28,4 +28,4 @@
# https://github.com/mdavidsaver/setuptools_dso/issues/11#issuecomment-808258994
import golang
from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim
from wendelin.bigfile._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim
# -*- coding: utf-8 -*-
# Copyright (C) 2018-2022 Nexedi SA and Contributors.
# Copyright (C) 2018-2024 Nexedi SA and Contributors.
# Kirill Smelkov <kirr@nexedi.com>
#
# This program is free software: you can Use, Study, Modify and Redistribute
......@@ -81,7 +81,7 @@ from persistent import Persistent
from zodbtools.util import ashex as h
from wendelin.lib.zodb import zurl_normalize_main
from .client._wcfs import \
from wendelin.wcfs.client._wcfs import \
PyWCFS as _WCFS, \
PyWatchLink as WatchLink \
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment