1. 01 Dec, 2020 1 commit
    • Kirill Smelkov's avatar
      Connection: Adjust msg_id a bit so it behaves like stream_id in HTTP/2 · cb1c9934
      Kirill Smelkov authored
      This is 2020 edition of my original patch from 2016 ( dd3bb8b4 ).
      
      It was described in my NEO/go article ( https://navytux.spb.ru/~kirr/neo.html )
      in the paragraph quoted below:
      
          NEO/go shifts from thinking about protocol logic as RPC to thinking of it as
          more general network protocol and settles to provide general
          connection-oriented message exchange service[1] : whenever a message with new
          `msg_id` is sent, a new connection is established multiplexed on top of a
          single node-node TCP link. Then it is possible to send/receive arbitrary
          messages over back and forth until so established connection is closed. This
          works transparently to NEO/py who still thinks it operates in simple RPC mode
          because of the way messages are put on the wire and because simple RPC is
          subset of a general exchange.  The `neonet` module also provides `DialLink` and
          `ListenLink` primitives[2] that work similarly to standard Go `net.Dial` and
          `net.Listen` but wrap so created link into the multiplexing layer. What is
          actually done this way is very similar to HTTP/2 which also provides multiple
          general streams multiplexing on top of a single TCP connection ([3], [4]).
          However if connection ids (sent in place of `msg_id` on the wire) are assigned
          arbitrary, there could be a case when two nodes could try to initiate two new
          different connections to each other with the same connection id. To prevent
          such kind of conflict a simple rule to allocate connection ids either even or
          odd, depending on the role peer played while establishing the link, could be
          used. HTTP/2 takes similar approach[5] where `"Streams initiated by a client
          MUST use odd-numbered stream identifiers; those initiated by the server MUST
          use even-numbered stream identifiers."` with NEO/go doing the same
          corresponding to who was originally dialer and who was a listener. However it
          requires small patch to be applied on NEO/py side to increment `msg_id` by 2
          instead of 1.
      
          [1] https://lab.nexedi.com/kirr/neo/blob/463ef9ad/go/neo/neonet/connection.go
          [2] https://lab.nexedi.com/kirr/neo/blob/463ef9ad/go/neo/neonet/newlink.go
          [3] https://tools.ietf.org/html/rfc7540#section-5
          [4] https://http2.github.io/faq/#why-is-http2-multiplexed
          [5] https://tools.ietf.org/html/rfc7540#section-5.1.1
      
      It can be named as "terrible", "irritating", "stupid" or "crazy", but the fact is:
      
      - it does no harm to NEO/py and is backward-compatible: a NEO/py node
        without this patch can still successfully connect and interoperate to
        another NEO/py node with this patch.
      
      - it is required for NEO/go to be able to interoperate with NEO/py.
        Both client and server parts of NEO/go use the same neonet module to exchange messages.
      
      - NEO/go client is used by wendelin.core 2, which organizes access to on-ZODB
        ZBigFile data via WCFS filesystem implemented in Go.
      
      So on one side this patch is small, simple and does not do any harm to NEO/py.
      On the other side it is required for NEO/go and wendelin.core 2.
      
      To me this clearly indicates that there should be NO GOOD REASON to reject
      inclusion of this patch into NEO/py.
      
      --------
      
      My original patch from 2016 came with corresponding adjustments to neo/tests/testConnection.py
      ( dd3bb8b4 )
      but commit f6eb02b4 (Remove packet timeouts; 2017-05-04) removed testConnection.py
      completely and, if I understand correctly, did not add any other test to
      compensate that. This way I'm not trying to restore my tests to
      Connection neither.
      
      Anyway, with this patch there is no regression to all other existing NEO/py tests.
      
      --------
      
      My original patch description from 2016 follows:
      
      - even for server initiated streams
      - odd  for client initiated streams
      
      This way I will be able to use Pkt.msg_id as real stream_id in go's Conn
      because with even / odd scheme there is no possibility for id conflicts
      in between two peers.
      
      /cc @romain, @tomo, @rafael, @arnau, @vpelletier, @klaus, @Tyagov
      cb1c9934
  2. 19 Aug, 2020 4 commits
  3. 22 May, 2020 2 commits
    • Julien Muchembled's avatar
      master: fix crash in STARTING_BACKUP when connecting to an upstream secondary master · 011eba12
      Julien Muchembled authored
      This fixes the following assertion:
      
        Traceback (most recent call last):
          File "neo/master/app.py", line 172, in run
            self._run()
          File "neo/master/app.py", line 182, in _run
            self.playPrimaryRole()
          File "neo/master/app.py", line 302, in playPrimaryRole
            self.backup_app.provideService())
          File "neo/master/backup_app.py", line 114, in provideService
            node, conn = bootstrap.getPrimaryConnection()
          File "neo/lib/bootstrap.py", line 74, in getPrimaryConnection
            poll(1)
          File "neo/lib/event.py", line 160, in poll
            to_process.process()
          File "neo/lib/connection.py", line 504, in process
            self._handlers.handle(self, self._queue.pop(0))
          File "neo/lib/connection.py", line 92, in handle
            self._handle(connection, packet)
          File "neo/lib/connection.py", line 107, in _handle
            pending[0][1].packetReceived(connection, packet)
          File "neo/lib/handler.py", line 125, in packetReceived
            self.dispatch(*args)
          File "neo/lib/handler.py", line 75, in dispatch
            method(conn, *args, **kw)
          File "neo/lib/handler.py", line 159, in notPrimaryMaster
            assert primary != self.app.server
        AttributeError: 'BackupApplication' object has no attribute 'server'
      
      (cherry picked from commit dba07e72)
      011eba12
    • Julien Muchembled's avatar
      mysql: workaround for MDEV-20693 · 70387981
      Julien Muchembled authored
      70387981
  4. 07 Jan, 2020 1 commit
  5. 28 Apr, 2019 1 commit
  6. 27 Apr, 2019 12 commits
  7. 26 Apr, 2019 4 commits
  8. 16 Apr, 2019 5 commits
  9. 05 Apr, 2019 3 commits
  10. 01 Apr, 2019 1 commit
  11. 21 Mar, 2019 2 commits
  12. 16 Mar, 2019 1 commit
    • Julien Muchembled's avatar
      importer: fix possible data loss on writeback · e387ad59
      Julien Muchembled authored
      If the source DB is lost during the import and then restored from a backup,
      all new transactions have to written back again on resume. It is the most
      common case for which the writeback hits the maximum number of transactions
      per partition to process at each iteration; the previous code was buggy in
      that it could skip transactions.
      e387ad59
  13. 13 Mar, 2019 3 commits