1. 22 Nov, 2018 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components: Fix startup of old Sites not having portal_components yet. · bda37e96
      Arnaud Fontaine authored
      In such case, it must obviously be considered that the ZODB Component Document
      was not found and thus fallback on filesystem Document.
      
      2018-11-21 17:05:01,919 WARNING ERP5Type.Dynamic Could not access Portal Type Object for type 'Category Tool'
      Traceback (most recent call last):
        File "product/ERP5Type/dynamic/lazy_class.py", line 3
      52, in loadClass
          class_definition = generatePortalTypeClass(site, portal_type)
        File "product/ERP5Type/dynamic/portal_type_class.py",
       line 198, in generatePortalTypeClass
          module = erp5.component.document.find_load_module(type_class)
        File "product/ERP5Type/dynamic/component_package.py",
       line 361, in find_load_module
          loader = self.find_module(fullname)
        File "product/ERP5Type/dynamic/component_package.py",
       line 155, in find_module
          component_tool = aq_base(site.portal_components)
      AttributeError: portal_components
      bda37e96
  2. 29 Oct, 2018 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components: Modules loaded by one Request were being GC'ed while executing in another Request. · f523edc4
      Arnaud Fontaine authored
      Until this commit, loading a ZODB Component would add it to the current Request
      object to prevent its reference counter reaching 0 (thus its global being reset to
      None and then being later GC'ed) when a reset happens in another Request.
      
      This problem was found when investigating testFunctionalConfigurator failure
      when installing bt5s as requested by the Configurator:
        1. Request R1 calls Zuite_waitForActivities importing extension.erp5.ERP5Zuite (M1).
           => Add M1 to R1._module_cache_set.
           => M1 ref counter equals to 2 (sys.modules and R1._module_cache_set references).
        2. R1 terminates and is GC'ed.
           => M1 ref counter equals to 1 (sys.modules).
        3. Request R2 runs Configurator configuring the Site.
        4. testFunctionalConfigurator calls Zuite_waitForActivities to wait for the
           Configurator to finish (request R3 which may take ~15 minutes). This calls
           time.sleep() in a loop where 'time' module is imported at top-level.
        5. R2 installs bt5 triggering reset.
           => M1 ref counter equals to 0.
              ===> M1 global variables are reset to None and thus 'time' is set to None
                   raising an Exception in the next call of time.sleep() in the loop.
      
      The easiest way would be to have a hook on sys.modules dict lookup and thus add M1
      to R2._module_cache_set when being imported, but this is not supported... Instead
      create a global cache on erp5.component package.
      f523edc4
  3. 20 Mar, 2018 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components not following ID=(document|extension|test).VERSION.REFERENCE... · cfed9ece
      Arnaud Fontaine authored
      ZODB Components not following ID=(document|extension|test).VERSION.REFERENCE convention are not importable anymore.
      
      Import mechanism now relies on this convention to find the document in
      portal_components.
      
      This is necessary to get rid of _registry_dict (ZODB Components equivalent of
      document_class_registry), along with several locks. This used to be a mapping
      between (reference, version) to ID stored outside of Zope Transactions as a
      dict on global erp5.component.XXX modules.
      
      This did not work for long transactions such as installing many bt5s:
        1. Transaction1: Install a bt5 with ZODB Component "Foo".
           => Trigger a reset and _registry_dict is cleared.
        2. Transaction2: TimerServer kicks in.
           => Generate _registry_dict without "Foo".
        3. Transaction1: Install another bt5 using "Foo".
           => "Foo" not in _registry_dict and thus considered not present.
      cfed9ece
  4. 25 Jul, 2016 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components: Fix bootstrap issue. · 5ba7e6fa
      Arnaud Fontaine authored
      When a Property Sheet with a ScriptConstraint (whose Document is a ZODB
      Component) was added to Memcached Tool: Document Component was partially
      loaded with no Property Sheet yet (and thus without the accessors needed
      to import ScriptConstraint ZODB Component) and ScriptConstraint Document
      was considered not available.
      5ba7e6fa
  5. 25 Nov, 2015 1 commit
  6. 09 Nov, 2015 1 commit
    • Arnaud Fontaine's avatar
      ZODB Components: Fix deadlock between import lock and aq_method_lock. · 11456c69
      Arnaud Fontaine authored
      This only happens when using ZEO (see source code comments).
      
      Steps to reproduce:
        1. Edit a ZODB Component in one tab.
        2. At the same time, run Unit Tests in another tab.
      
      Backtrace:
      
        # ThreadID: 140153540167424
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
          response=b)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 455, in publish_module
          environ, debug, request, response)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 249, in publish_module_standard
          response = publish(request, module_name, after_list, debug=debug)
        File: "parts/erp5/Products/Localizer/patches.py", line 84, in new_publish
          x = zope_publish(request, module_name, after_list, debug)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 138, in publish
          request, bind=1)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/mapply.py", line 44, in mapply
          f, count = zope.publisher.publish.unwrapMethod(object)
        File: "eggs/zope.publisher-3.12.6-py2.7.egg/zope/publisher/publish.py", line 46, in unwrapMethod
          elif getattr(unwrapped, 'func_code', None) is not None:
        File: "eggs/Products.ExternalMethod-2.13.0-py2.7.egg/Products/ExternalMethod/ExternalMethod.py", line 106, in <lambda>
          func_code = ComputedAttribute(lambda self: self.getFuncCode())
        File: "eggs/Products.ExternalMethod-2.13.0-py2.7.egg/Products/ExternalMethod/ExternalMethod.py", line 190, in getFuncCode
          self._v_f = self.getFunction()
        File: "parts/erp5/Products/ERP5Type/patches/ExternalMethod.py", line 29, in getFunction
          level=0)
        File: "parts/erp5/Products/ERP5Type/dynamic/component_package.py", line 407, in load_module
          with aq_method_lock:
        File: "parts/python2.7/lib/python2.7/threading.py", line 174, in acquire
          rc = self.__block.acquire(blocking)
      
        => 1. Acquire Import lock in getFunction() (ZODB Component import)
           2. Try to acquire aq_method_lock
      
        # ThreadID: 140153468495616
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
          response=b)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 455, in publish_module
          environ, debug, request, response)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 249, in publish_module_standard
          response = publish(request, module_name, after_list, debug=debug)
        File: "parts/erp5/Products/Localizer/patches.py", line 84, in new_publish
          x = zope_publish(request, module_name, after_list, debug)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 138, in publish
          request, bind=1)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/mapply.py", line 77, in mapply
          if debug is not None: return debug(object,args,context)
        File: "eggs/Zope2-2.13.22-py2.7.egg/ZPublisher/Publish.py", line 48, in call_object
          result=apply(object,args) # Type s<cr> to step into published object.
        File: "parts/erp5/Products/TimerService/TimerService.py", line 90, in process_timer
          subscriptions.append(self.unrestrictedTraverse(path))
        File: "eggs/Zope2-2.13.22-py2.7.egg/OFS/Traversable.py", line 249, in unrestrictedTraverse
          if getattr(aq_base(obj), name, _marker) is not _marker:
        File: "parts/erp5/Products/ERP5Type/dynamic/lazy_class.py", line 107, in __getattribute__
          self.__class__.loadClass()
        File: "parts/erp5/Products/ERP5Type/dynamic/lazy_class.py", line 326, in loadClass
          class_definition = generatePortalTypeClass(site, portal_type)
        File: "parts/erp5/Products/ERP5Type/dynamic/portal_type_class.py", line 143, in generatePortalTypeClass
          type_class = portal_type.getTypeClass()
        File: "eggs/ZODB3-3.10.5+slapospatched001-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 860, in setstate
          self._setstate(obj)
        File: "eggs/ZODB3-3.10.5+slapospatched001-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 914, in _setstate
          self._reader.setGhostState(obj, p)
        File: "eggs/ZODB3-3.10.5+slapospatched001-py2.7-linux-x86_64.egg/ZODB/serialize.py", line 612, in setGhostState
          state = self.getState(pickle)
        File: "eggs/ZODB3-3.10.5+slapospatched001-py2.7-linux-x86_64.egg/ZODB/serialize.py", line 604, in getState
          unpickler.load() # skip the class metadata
        File: "eggs/ZODB3-3.10.5+slapospatched001-py2.7-linux-x86_64.egg/ZODB/serialize.py", line 474, in find_global
          return factory(conn, modulename, name)
        File: "eggs/Zope2-2.13.22-py2.7.egg/Zope2/App/ClassFactory.py", line 21, in ClassFactory
          m=__import__(module, _globals, _globals, _silly)
      
        => 1. Acquire aq_method_lock (generatePortalTypeClass())
           2. Try to import module and acquire Import lock
      11456c69
  7. 24 Jan, 2014 1 commit
  8. 23 Jan, 2014 4 commits
    • Arnaud Fontaine's avatar
    • Arnaud Fontaine's avatar
      ZODB Components: Display proper traceback and source code if an error occurred on 'exec'. · cb631f83
      Arnaud Fontaine authored
      The initial 'exec' traceback was not passed when raising ImportError (following
      PEP-302), thus it displayed traceback in component_package instead of the actual
      location of the error. Also, Caching source code retrieved from traceback/(i)pdb
      (through get_source()) was done after the actual 'exec', whereas it should have
      always been *before* obviously...
      
      Steps to reproduce:
        1/ Create and validate a Test Component A importing another Component B.
        2/ Invalidate Component B or add a typo not handled by pylint (because of exceptions).
      cb631f83
    • Arnaud Fontaine's avatar
      ZODB Components: Cache source code in the loader to avoid access to ERP5 Site. · 2ac367df
      Arnaud Fontaine authored
      Using DeadlockDebugguer lead to a deadlock when trying to debug a deadlock on
      ZODB Component as it was trying to load the Component. Moreover, this required
      accessing to ERP5 Site whereas DeadlockDebugguer is used outside.
      
      Steps to reproduce:
        1. Create an Extension Component with a sleep and run it.
        2. Call manage_debug_threads.
      2ac367df
    • Arnaud Fontaine's avatar
      ZODB Components: Execute code in two steps: compile(), then 'exec' instead of only 'exec'. · 6b03c39e
      Arnaud Fontaine authored
      This is much faster, eg defining a max function is about 39 times faster and
      even more on more complicated code:
      
      $ python -mtimeit -s 'code = "def max(a, b): return a > b and a or b"' 'exec code'
      10000 loops, best of 3: 25.2 usec per loop
      $ python -mtimeit -s 'code = compile("def max(a, b): return a > b and a or b", "<string>", "exec")' 'exec code'
      1000000 loops, best of 3: 0.632 usec per loop
      
      Another reason is for traceback/pdb readability as __file__ is displayed in
      the backtrace, but with 'exec' __file__ equals to '<string>', compile() allows
      to specify a readable name easily without having to override __file__.
      
      Also, set __file__ to the Component relative URL instead of the ID for
      readability sake.
      6b03c39e
  9. 10 Dec, 2013 2 commits
  10. 10 Sep, 2013 3 commits
    • Arnaud Fontaine's avatar
      ZODB Components: When executing code, module globals disappeared after a reset... · 3a92d8b6
      Arnaud Fontaine authored
      ZODB Components: When executing code, module globals disappeared after a reset performed in another thread.
      
      This fixes failures in erp5_configurator* tests where BTs installed during the
      tests trigger a reset and globals (such as DateTime global import) were reset to
      None.
      3a92d8b6
    • Arnaud Fontaine's avatar
      ZODB Components: Component with the same version/reference as a validated one... · 7d31f4fb
      Arnaud Fontaine authored
      ZODB Components: Component with the same version/reference as a validated one must not be able to be validated.
      7d31f4fb
    • Arnaud Fontaine's avatar
      ZODB Components: Fix deadlock on import lock. · 3181d5ad
      Arnaud Fontaine authored
      Since a29456bc, only import lock is used instead of aq_method_lock, dynamic
      modules locks and import lock. However, it was creating another deadlock:
      
      when the import lock is held in one thread (for instance when trying to
      perform migration to Portal Type as Classes and ZODB Property Sheets), and an
      object is loaded from ZODB, a request is sent to ZEO, and this blocks until
      another thread (asyncore) gets the ZEO reply and sends it back to the first
      thread.
      
      However, if the asyncore thread receives an Exception, it will tries to import
      its module and thus create a deadlock as the import lock is still held by the
      first thread.
      3181d5ad
  11. 09 Sep, 2013 5 commits
  12. 14 Mar, 2012 2 commits
  13. 12 Mar, 2012 4 commits
  14. 07 Mar, 2012 13 commits