1. 15 Apr, 2020 1 commit
    • Kirill Smelkov's avatar
      setup: Fix hooking of git_lsfiles in PEP517 mode · bd1fb19e
      Kirill Smelkov authored
      In PEP517 mode setup.py is sourced - not executed - and the build fails
      with ImportError like this:
      
          Preparing wheel metadata ... error
          ERROR: Command errored out with exit status 1:
           command: /home/kirr/src/wendelin/venv/z-dev/bin/python2 /home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp2F3aEs
               cwd: /home/kirr/src/wendelin/wendelin.core
          Complete output (53 lines):
          running dist_info
          creating /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info
          writing requirements to /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/requires.txt
          writing /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/PKG-INFO
          writing top-level names to /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/top_level.txt
          writing dependency_links to /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/dependency_links.txt
          writing entry points to /tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/entry_points.txt
          writing manifest file '/tmp/pip-modern-metadata-sPiqUt/wendelin.core.egg-info/SOURCES.txt'
          package init file '__init__.py' not found (or not a regular file)
          Traceback (most recent call last):
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
              main()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
              json_out['return_val'] = hook(**hook_input['kwargs'])
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
              return hook(metadata_directory, config_settings)
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/build_meta.py", line 155, in prepare_metadata_for_build_wheel
              self.run_setup()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/build_meta.py", line 234, in run_setup
              self).run_setup(setup_script=setup_script)
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/build_meta.py", line 141, in run_setup
              exec(compile(code, __file__, 'exec'), locals())
            File "setup.py", line 374, in <module>
              """.splitlines()]
            File "/home/kirr/src/tools/go/pygolang/golang/pyx/build.py", line 118, in setup
              setuptools_dso.setup(**kw)
            File "/home/kirr/src/tools/py/pypa/setuptools_dso/src/setuptools_dso/__init__.py", line 37, in setup
              _setup(**kws)
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
              return distutils.core.setup(**attrs)
            File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
              dist.run_commands()
            File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
              self.run_command(cmd)
            File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
              cmd_obj.run()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/dist_info.py", line 31, in run
              egg_info.run()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run
              self.find_sources()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
              mm.run()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 534, in run
              self.add_defaults()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 574, in add_defaults
              rcfiles = list(walk_revctrl())
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line 20, in walk_revctrl
              for item in ep.load()(dirname):
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2434, in load
              return self.resolve()
            File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2444, in resolve
              raise ImportError(str(exc))
          ImportError: 'module' object has no attribute 'git_lsfiles'
      
      See comments added to register_as_entrypoint for explanation of what
      happens.
      
      Wendelin.core will soon switch to PEP517 mode (by adding pyproject.toml)
      to build-require Cython, Pygolang and friends.
      bd1fb19e
  2. 18 Dec, 2019 1 commit
    • Kirill Smelkov's avatar
      bigfile/py: Move data structures to public .h file · 907bd9d4
      Kirill Smelkov authored
      This is needed so that e.g. a Python class implemented in C or Cython
      (cdef class) could inherit from PyBigFile.
      
      Don't put _bigfile.h into separate include/ directory, and keep it along
      main .c file, similarly to how pygolang is organized.
      907bd9d4
  3. 12 Jul, 2019 1 commit
    • Kirill Smelkov's avatar
      *: Use defer for dbclose & friends · 5c8340d2
      Kirill Smelkov authored
      For tests this makes sure that if one test fails, it won't make following
      tests fail just because the next test will fail trying to lock test database.
      
      For regular code (demo_zbigarray.py) this is also a good thing to do -
      to always close the database irregardless of whether an exception was
      raised before program reached end of main.
      
      Pygolang becomes regular - not test only - dependency. Being regular
      dependency is currently required only by demo_zbigarray.py, but it will
      be also used in upcoming wcfs, so adding pygolang into wendelin.core
      dependencies aligns with the plan.
      
      dbclose now uses defer almost everywhere - there are still few places in
      tests, where one test function is opening/closing test database multiple
      times - those were not (yet ?) converted.
      5c8340d2
  4. 18 Jun, 2019 3 commits
  5. 04 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      py.bench: Move it to -> pygolang · 318efce0
      Kirill Smelkov authored
      So that it can be available to everyone and in particular B & friends to
      be available from introduced importable golang.testing package. The move
      itself:
      
      	kirr/pygolang@9bf03d9c
      
      While moving the code was restructured / improved a bit and py.bench
      interface reworked to mimic `go test -bench` in defaults.
      318efce0
  6. 17 Apr, 2018 1 commit
  7. 16 Apr, 2018 2 commits
  8. 24 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source... · f11386a4
      Kirill Smelkov authored
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
      
      Nexedi stack is licensed under Free Software licenses with various exceptions
      that cover three business cases:
      
      - Free Software
      - Proprietary Software
      - Rebranding
      
      As long as one intends to develop Free Software based on Nexedi stack, no
      license cost is involved. Developing proprietary software based on Nexedi stack
      may require a proprietary exception license. Rebranding Nexedi stack is
      prohibited unless rebranding license is acquired.
      
      Through this licensing approach, Nexedi expects to encourage Free Software
      development without restrictions and at the same time create a framework for
      proprietary software to contribute to the long term sustainability of the
      Nexedi stack.
      
      Please see https://www.nexedi.com/licensing for details, rationale and options.
      f11386a4
  9. 28 Mar, 2017 1 commit
  10. 16 Mar, 2017 1 commit
  11. 17 Jan, 2017 1 commit
  12. 28 Sep, 2016 1 commit
  13. 14 Jul, 2016 3 commits
    • Kirill Smelkov's avatar
      wendelin.core v0.7 · f7803634
      Kirill Smelkov authored
      f7803634
    • Kirill Smelkov's avatar
      setup: Propagate only setuptools, not whole sys.path · 8f624957
      Kirill Smelkov authored
      This updates and fixes 487e5226 (setup: specify setuptools location
      explicitly when calling make.) to use @kazuhiko original idea to
      propagate only setuptools location. The reason is - when propagating
      whole sys.path things break under tox tests:
      
      ---- 8< ----
      ========================================================================== test session starts ===========================================================================
      platform linux2 -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
      rootdir: /home/kirr/src/wendelin/release/wendelin.core, inifile:
      collected 33 items
      
      bigarray/tests/test_arrayzodb.py .......
      bigarray/tests/test_basic.py ........
      bigfile/tests/test_basic.py ....
      bigfile/tests/test_filefile.py .
      bigfile/tests/test_filezodb.py ........
      bigfile/tests/test_thread.py ....
      lib/tests/test_calc.py .
      
      ======================================================================= 33 passed in 14.14 seconds =======================================================================
      x86_64-linux-gnu-gcc -pthread -g -Wall -D_GNU_SOURCE -std=gnu99 -fplan9-extensions -Wno-declaration-after-statement -Wno-error=declaration-after-statement  -Iinclude -I3rdparty/ccan -I3rdparty/include   bigfile/tests/tfault.c lib/bug.c lib/utils.c 3rdparty/ccan/ccan/tap/tap.c  -o bigfile/tests/tfault.t
      t/tfault-run bigfile/tests/tfault.t faultr on_pagefault
      ok 1 - !pagefault_init()
      Fatal Python error: Py_Initialize: Unable to get the locale encoding
        File ".../wendelin.core/.tox/py27-ZODB3-zblk0-fs-numpy110/lib/python2.7/encodings/__init__.py", line 123
          raise CodecRegistryError,\
                                  ^
      SyntaxError: invalid syntax
      
      Current thread 0x00007f9b80024780 (most recent call first):
      t/tfault-run: line 28: 21521 Аварийный останов         (core dumped) gdb -q -batch $tfault core > core.info
      E: can't gdb(core)
      Makefile:189: ошибка выполнения рецепта для цели «faultr.tfault»
      make: *** [faultr.tfault] Ошибка 1
      rm bigfile/tests/test_virtmem.t bigfile/tests/test_ram.t bigfile/tests/tfault.t bigfile/tests/test_pagemap.t
      error: Failed to execute `make test`
      ERROR: InvocationError: '.../wendelin.core/.tox/py27-ZODB3-zblk0-fs-numpy110/bin/python setup.py test'
      ________________________________________________________________________________ summary _________________________________________________________________________________
      ERROR:   py27-ZODB3-zblk0-fs-numpy110: commands failed
      ---- 8< ----
      
      What happens here is:
      
      - gdb is used in automated tests
      - gdb is linked with libpython3.5
      - tox is currently running tests with python27
      - setup.py sets PYTHONPATH to whole path from python27
      - gdb, upon starting, initializes python runtime, which tries to load
        py27 modules under py35 -> oops.
      
      So propagating only setuptools location practically solves the issue for
      now, but still there is potential risk that in future, there will be
      other modules put in the same location as setuptools (location is parent
      dir of a module/package) which python tries to load on startup and which
      might be incompatible between 2 & 3.
      
      Thus not setting PYTHONPATH at all - e.g. creating environments
      virtualenv way - would be better, but buildout fundamentally works the
      other way. We can not also use -c 'buildout sys.path hack' in $PYTHON
      itself, as $PYTHON is used in general way inside Makefile.
      
      So let the hack with setuptools location in PYTHONPATH stay there until
      it practically works.
      8f624957
    • Kazuhiko's avatar
      setup: specify setuptools location explicitly when calling make. · 487e5226
      Kazuhiko authored
      If wendelin.core is built under buildout and setuptools exists only in
      buildout environment, 'python setup.py' called inside make will fail
      without this change.
      
          Building 'wendelin.core'
          Running easy_install:
          "/path/to/python2.7" "-c" "import sys; sys.path[0:0] = ['/path/to/setuptools-19.6.2-py2.7.egg']; from setuptools.command.easy_install import main; main()"...
          path_list=['/path/to/setuptools-19.6.2-py2.7.egg']
          ...
          <<< setup.py: os.system('make %s PYTHON="%s"' % (target, sys.executable))
          ...
          make[1]: Leaving directory '/tmp/xxx/wendelin.core-0.6/3rdparty/ccan'
          /path/to/python2.7 setup.py ll_build_ext --inplace
          Traceback (most recent call last):
            File "setup.py", line 18, in <module>
              from setuptools import setup, Extension, Command, find_packages
          ImportError: No module named setuptools
          Makefile:40: recipe for target 'bigfile/_bigfile.so' failed
          make: *** [bigfile/_bigfile.so] Error 1
          error: Setup script exited with error: Failed to execute `make all`
      
      [ @kirr: as a solution we are propagating whole python path via make.
      
        This should cover both setuptools case, and any other future potential
        "preloaded-by-buildout" egg. ]
      
      /reviewed-on !1
      487e5226
  14. 06 Jul, 2016 1 commit
    • Kirill Smelkov's avatar
      Fix build for Python 3.5 · e6beab19
      Kirill Smelkov authored
      @kazuhiko reports that wendelin.core build is currently broken on Python 3.5.
      Indeed it was:
      
          In file included from bigfile/_bigfile.c:37:0:
          ./include/wendelin/compat_py2.h: In function ‘_PyThreadState_UncheckedGetx’:
          ./include/wendelin/compat_py2.h:66:28: warning: implicit declaration of function ‘_Py_atomic_load_relaxed’ [-Wimplicit-function-declaration]
               return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current);
                                      ^
          ./include/wendelin/compat_py2.h:66:53: error: ‘_PyThreadState_Current’ undeclared (first use in this function)
               return (PyThreadState*)_Py_atomic_load_relaxed(&_PyThreadState_Current);
                                                               ^
          ./include/wendelin/compat_py2.h:66:53: note: each undeclared identifier is reported only once for each function it appears in
          ./include/wendelin/compat_py2.h:67:1: warning: control reaches end of non-void function [-Wreturn-type]
           }
           ^
      
      The story here is that in 3.5 they decided to remove direct access to
      _PyThreadState_Current and atomic implementations - because that might
      semantically conflict with other headers implementing atomics - and
      provide only access by function.
      
      Starting from Python 3.5.2rc1 the function to get current thread state
      without asserting it is !NULL - _PyThreadState_UncheckedGet() - was added:
      
          https://github.com/python/cpython/commit/df858591
      
      so for those python versions we can directly use it.
      
      After the fix wendelin.core tox tests pass under all python2.7, python3.4 and python3.5.
      
      More context here:
      
          https://bugs.python.org/issue26154
          https://bugs.python.org/issue25150
      
      Fixes: #1
      e6beab19
  15. 13 Jun, 2016 2 commits
  16. 02 Oct, 2015 3 commits
  17. 19 Aug, 2015 1 commit
  18. 17 Aug, 2015 1 commit
    • Kirill Smelkov's avatar
      Drop support for ZODB < 3.10 · 105ab1c7
      Kirill Smelkov authored
      ZODB 3.10.4 was released almost 4 years ago, and contains significant
      change how ghost objects coming from DB are initially setup.
      105ab1c7
  19. 12 Jun, 2015 1 commit
  20. 02 Jun, 2015 2 commits
  21. 01 Jun, 2015 2 commits
  22. 28 May, 2015 1 commit
  23. 25 May, 2015 2 commits
  24. 14 May, 2015 1 commit
    • Kirill Smelkov's avatar
      Fix build with non-std "python" · a280b53d
      Kirill Smelkov authored
      In 5755a6b3 (Basic setup.py / Makefile to build/install/sdist stuff +
      bigfile.so skeleton) we overlooked one thing:
      
          when tailing from setup.py to make level, we forgot to setup the
          PYTHON env variable properly, and that leads to changing building
          with std python.
      
      Look e.g.
      
          $ python3 setup.py build_ext -i
          running build_ext
          python setup.py ll_build_ext --inplace
          ^^^
      
      which is obviously not correct.
      
      Fix it.
      
      The tox setup we have setup for testing with multiple interpreters
      (7af2b2d7 "Tox setup to test things with py27 & py34 and several
      versions of ZODB") did not caught this, because for every python and
      environment versions tox initializes separate virtualenv, in which
      .../bin/python is a link to chosen python, and this way both python and
      my-selected-for-build-python are the same.
      
      Cc: @Tyagov
      a280b53d
  25. 15 Apr, 2015 1 commit
    • Kirill Smelkov's avatar
      Fix build with Python that enables -Werror=declaration-after-statement · e0b25398
      Kirill Smelkov authored
      In C99 declaration after statement is ok, and we explicitly compile with -std=gnu99.
      
      Python >= 3.4 however adds -Werror=declaration-after-statement even for extension
      modules irregardless of their compilation flags:
      
        https://bugs.python.org/issue21121
      
      and the build fails this way:
      
          building 'wendelin.bigfile._bigfile' extension
          creating build/temp.linux-x86_64-3.4
          creating build/temp.linux-x86_64-3.4/bigfile
          creating build/temp.linux-x86_64-3.4/lib
          gcc -pthread -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall
      -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -D_GNU_SOURCE
      -I./include -I./3rdparty/ccan -I./3rdparty/include -I/usr/include/python3.4m -c bigfile/_bigfile.c -o build/temp.linux-x86_64-3.4/bigfile/_bigfile.o
      -std=gnu99 -fplan9-extensions -fvisibility=hidden
          bigfile/_bigfile.c: In function ‘pyfile_new’:
          bigfile/_bigfile.c:679:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
               static char *kw_list[] = {"blksize", NULL};
               ^
          cc1: some warnings being treated as errors
          error: command 'gcc' failed with exit status 1
      
      Ensure we turn off this warning and error because we already rely on compiling in C99 mode.
      Reported-and-tested-by: 's avatarIvan Tyagov <ivan@tyagov.com>
      e0b25398
  26. 03 Apr, 2015 4 commits
    • Kirill Smelkov's avatar
      wendelin.core v0.1 · 4b6821f4
      Kirill Smelkov authored
      4b6821f4
    • Kirill Smelkov's avatar
      Demo program that shows how to work with ZBigArrays bigger than RAM in size · 1ee72371
      Kirill Smelkov authored
      This shows how to first generate such arrays (in steps, as every
      transaction change should fit in memory), and then gather data from
      whole array using C/Fortran/etc code.
      
      It shows how to compute mean via NumPy's ndarray.mean()
      
      It also shows that e.g. ndarray.var() wants to create temporaries in
      size of original ndarray and that would fail, because it does not fit
      into RAM.
      
      ndarray.var() should not need to create such temporaries in principle -
      all it has to do is to first compute mean, and then compute
      
          sum (Xi - <X>)^2
      
      in a loop.
      
      <X> is scalar, Xi - is just access to original array.
      
      ~~~~
      
      So this also show NumPy can be incrementally improved to avoid creating
      such temporaries, and then it will work.
      1ee72371
    • Kirill Smelkov's avatar
      Prepare to benchmark python code with py.test · bcab1246
      Kirill Smelkov authored
      For this, a small wrapper over py.test is developed (to discover/collect
      functions to benchmar, etc) and then it runs such functions several
      times in a boxed enveronment.
      
      Benchmarks should be named bench_*.py
      bcab1246
    • Kirill Smelkov's avatar
      BigArray: An ndarray-like on top of BigFile memory mappings · 0c826d5c
      Kirill Smelkov authored
      I.e. something like numpy.memmap for numpy.ndarray and OS files. The whole
      bigarray cannot be used as a drop-in replacement for numpy arrays, but BigArray
      _slices_ are real ndarrays and can be used everywhere ndarray can be used,
      including in C/Fortran code. Slice size is limited by mapping-size (=
      address-space size) limit, i.e. to ~ max 127TB on Linux/amd64.
      
      Changes to bigarray memory are changes to bigfile memory mapping and as such
      can be discarded or saved back to bigfile using mapping (= BigFileH) dirty
      discard/writeout interface.
      
      For the same reason the whole amount of changes to memory is limited by amount
      of physical RAM.
      0c826d5c