1. 30 Jan, 2017 1 commit
    • Kirill Smelkov's avatar
      X Switch Zope to use poll() instead of select() · c6addb05
      Kirill Smelkov authored
      Recently we are starting to get following errors on a production
      instance:
      
            File ".../bin/runzope", line 211, in <module>
              sys.exit(Zope2.Startup.run.run())
            File ".../eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/run.py", line 26, in run
              starter.run()
            File ".../eggs/Zope2-2.13.24-py2.7.egg/Zope2/Startup/__init__.py", line 111, in run
              Lifetime.loop()
            File ".../eggs/Zope2-2.13.24-py2.7.egg/Lifetime/__init__.py", line 43, in loop
              lifetime_loop()
            File ".../eggs/Zope2-2.13.24-py2.7.egg/Lifetime/__init__.py", line 53, in lifetime_loop
              asyncore.poll(timeout, map)
            File ".../parts/python2.7/lib/python2.7/asyncore.py", line 145, in poll
              r, w, e = select.select(r, w, e, timeout)
          ValueError: filedescriptor out of range in select()
      
      Initially we thought the reason here is that number of file descriptors this
      process uses goes beyond allowed limit (65K open files on that instance)
      because wendelin.core used 1 fd per an array view (opening
      /dev/shm/ramh.XXXXX) but that turned out to be not strictly that case:
      
      The reason here is that select() has limit for how many #fd it can
      monitor at all. The limit is system-specific but on Linux it is usually
      1024 - http://man7.org/linux/man-pages/man2/select.2.html#BUGS :
      
          $ cat s.c
          #include <sys/select.h>
          #include <stdio.h>
      
          int main() {
                  printf("%d\n", FD_SETSIZE);
                  return 0;
          }
      
          $ tcc -run s.c
          1024
      
      Also select() can monitor only file descriptors which are by itself are
      "< FD_SETSIZE", e.g. it cannot monitor fd=1025 even if there are only 10
      opened file descriptors because 1025 is not < 1024.
      
      As was said above in wendelin.core every array view uses 1 file
      descriptor, so if we are using not so small amount of arrays, even
      though #fd does not go beyond process-specific ulimit, because of
      select() usage the total number of allowed opened file descriptors
      becomes essentially 1024.
      
      So let's switch from select() to poll() which does not have this 1024
       #fd limit. Asyncore already support using poll() out of the box -
      either via passing use_pull=True to asyncore.loop()
      
      https://docs.python.org/2/library/asyncore.html#asyncore.loop
      
      or by using asyncore.poll2() instead of asyncore.poll()
      
      https://github.com/python/cpython/blob/2.7/Lib/asyncore.py#L170
      https://github.com/python/cpython/blob/2.7/Lib/asyncore.py#L125
      
      --------
      
      This patch switches asyncore.poll() -> asyncore.poll2() for only 1 place
      in Zope. There are however many such places in Zope and other software.
      
      For this reason, for me, what makes sense to do is not to patch all such
      places, but instead change via runtime-patching asyncore.poll to be
      asyncore.poll2 - this way all software will automatically benefit from
      poll() usage instead of select.
      
      P.S.
      
      What might also make sense to do in the future is to actually let
      asyncore.poll to use epoll(), as both select() and poll() are doing all
      fd registration on _every_ call, so when #fd grows this preparation
      time grows too. For epoll() file descriptors are registered only once.
      For this to work asyncore.socket_map has to be patched also, since there
      are places in code which modify this fd registry directly (e.g. remove
      fd from there etc)
      c6addb05
  2. 18 Jan, 2017 2 commits
    • Maurits van Rees's avatar
      Fixed dummy runTest on functional test case. (#89) · ecfd7b1b
      Maurits van Rees authored
      In functional doc tests you can apparently have a test case that has
      no runTest method.  Until now the Testing package added a dummy
      runTest method in that case, and set it to None.
      
      But when this dummy runTest method gets called, you get an error:
      
        Error in test runTest (Testing.ZopeTestCase.ZopeTestCase.FunctionalTestCase)
        Traceback (most recent call last):
          File ".../lib/python2.7/unittest/case.py", line 329, in run
            testMethod()
        TypeError: 'NoneType' object is not callable
      
      Simply importing Testing.ZopeTestCase.FunctionalTestCase in a new test
      file may be enough to trigger this.
      
      So this has something to do with the order in which tests are found.
      
      I fixed it by making the dummy runTest method callable.
      ecfd7b1b
    • Tres Seaver's avatar
      Merge pull request #86 from zopefoundation/apply-plonehotfix-20170717-213 · 554c81bc
      Tres Seaver authored
      Apply plonehotfix 20170717 [2.13]
      554c81bc
  3. 17 Jan, 2017 2 commits
  4. 13 Jan, 2017 7 commits
  5. 27 Dec, 2016 2 commits
  6. 21 Dec, 2016 3 commits
  7. 08 Dec, 2016 1 commit
  8. 15 Sep, 2016 2 commits
  9. 14 Sep, 2016 1 commit
  10. 09 Sep, 2016 1 commit
  11. 08 Sep, 2016 1 commit
  12. 07 Sep, 2016 3 commits
  13. 02 Aug, 2016 1 commit
  14. 01 Aug, 2016 1 commit
  15. 17 Jul, 2016 3 commits
  16. 29 Jun, 2016 2 commits
  17. 04 Mar, 2016 2 commits
  18. 29 Feb, 2016 5 commits