1. 30 Dec, 2018 3 commits
  2. 17 Dec, 2018 1 commit
  3. 13 Dec, 2018 3 commits
    • Kirill Smelkov's avatar
      zodbcommit - Tool to commit new transaction into ZODB · 960c5e17
      Kirill Smelkov authored
      Zodbcommit reads transaction description from stdin and commits read data into
      ZODB. The transaction to be committed is read in zodbdump format, but without
      first 'txn' header line. For example:
      
          user "author"
          description "change 123"
          extension ""
          obj 0000000000000001 4 null:00
          ZZZZ
      
      This tool could be useful for testing and for low-level database
      maintenance. Please see zodbcommit.py docstring for more details.
      960c5e17
    • Kirill Smelkov's avatar
      zodbdump += DumpReader - to read/parse zodbdump stream · dd959b28
      Kirill Smelkov authored
      We will likely need this reader for `zodb restore` in the future.
      We will also use this reader for `zodb commit` in the next patch.
      
      pygolang dependency v↑ becuase we use recently introduced
      golang.strconv to unquote user/desc/extension strings.
      
      Python2 works. Python3 support is only minimal and incomplete.
      dd959b28
    • Kirill Smelkov's avatar
      util += hashRegistry, null, adler32, crc32 hashers · e973d519
      Kirill Smelkov authored
      hashRegistry will be needed for zodbdump reader to create a hasher by
      its name. Since std hashlib does not have adler32/crc32 nor null - let's
      add our own implementation for those hashers too.
      
      The code is based on
      
      	kirr/neo@a60c472c, and
      	kirr/neo@3f578560
      e973d519
  4. 12 Dec, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Always end a transaction with LF · 624aeb09
      Kirill Smelkov authored
      Before now we were emitting extra LF only in between transactions as a
      separator. However the dump format states LF always goes after
      transaction, and there is a reason for it:
      
      	without LF in the end, it becomes ambiguous at EOF - whether it
      	is a proper transaction end, or the transaction was cut.
      
      So avoid the ambiguity by always emitting trailing LF after transaction
      record.
      624aeb09
  5. 02 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Switch to using qq from pygolang · b1163449
      Kirill Smelkov authored
      I originally added escapeqq as part of 75c03368 (zodbdump: Start to
      stabilize output format) with the task for this utility to quote string
      into valid "..." string always quoted with ".
      
      This utility was later copied to pygolang:
      
      	kirr/pygolang@afa46cf5
      
      and then further improved there to work under both Python2 and Python3
      and to not escape printable UTF-8 characters:
      
      	kirr/pygolang@02dddb97
      
      So stop the duplication and simply switch to the better version.
      b1163449
  6. 11 May, 2018 2 commits
  7. 02 Nov, 2017 1 commit
    • Kirill Smelkov's avatar
      zodbdump: Start to stabilize output format · 75c03368
      Kirill Smelkov authored
      Since zodbdump start (c0a6299f "zodbdump - Tool to dump content of a
      ZODB database   (draft)") and up till now zodbdump output format was not
      good. For example user and description transaction properties were
      output without proper quoting, which in situation when there would be
      fancy characters in there would break the output.
      
      So start the format stabilization:
      
      - user and description are output as quoted, so now they are guaranteed
        to be on one line. The quoting character is always " (instead of e.g.
        smartly quoting either by ' or " as python does) for easier
        compatibility with ZODB implementations in other languages.
      
      - transaction extension is now printed as raw bytes, not as dict.
        The idea here is that `zodb dump`
      
        * should perform dump of raw data as stored inside ZODB so that later
          `zodb restore` could restore the database identically to the same state.
      
        * we should dump raw data instead of unpickled ones because generally
          on-disk extension's format can be any raw bytes and this information
          should be preserved.
      
      - transaction status is now also output as quoted to preserve line
        breakage on fancy status codes.
      
      - it is documented that sha1 is not the only allowed hash function that
        might be used.
      
      - in hashonly mode we add trailing " -" to obj string so that it is
        possible to distinguish outputs of `zodb dump` and `zodb dump -hashonly`
        without knowing a-priory the way it was produced.
      
        The reason to do so is that it would be not good to e.g. by accident
        feed hashonly output to (future) `zodb restore`, which, without having
        a way to see it should not consume object data would read following
        transaction information as raw object data with confusing later
        errors (and a small chance to restore completely different database
        without reporting error at all).
      
      Because ZODB iteration API gives us already unpickled extension and only
      that, for now to dump it as raw we get a long road to pickle it back
      also caring to try to pickle in stable order.
      
      Hopefully this will be only a fallback because of
      
      https://github.com/zopefoundation/ZODB/pull/183
      
      and next zodbtools patch.
      
      ~~~~
      
      For testing purposes (currently only quoting function unit test) py.test
      usage is introduced.
      
      The code is also generally polished here and there.
      75c03368
  8. 24 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source... · 79cf177a
      Kirill Smelkov authored
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
      
      Nexedi stack is licensed under Free Software licenses with various exceptions
      that cover three business cases:
      
      - Free Software
      - Proprietary Software
      - Rebranding
      
      As long as one intends to develop Free Software based on Nexedi stack, no
      license cost is involved. Developing proprietary software based on Nexedi stack
      may require a proprietary exception license. Rebranding Nexedi stack is
      prohibited unless rebranding license is acquired.
      
      Through this licensing approach, Nexedi expects to encourage Free Software
      development without restrictions and at the same time create a framework for
      proprietary software to contribute to the long term sustainability of the
      Nexedi stack.
      
      Please see https://www.nexedi.com/licensing for details, rationale and options.
      79cf177a
  9. 04 Apr, 2017 4 commits
    • Kirill Smelkov's avatar
      Zodbinfo - Tool to print general information about a ZODB database · 37b9fbde
      Kirill Smelkov authored
      Either all general parameters at once:
      
          $ zodb info neo://neo1@127.0.0.1:24573
          name=NEOStorage(neo1)
          size=0
          last_tid=03be7484ddc7f6ee
      
      or one particular parameter:
      
          $ zodb info neo://neo1@127.0.0.1:24573 last_tid
          03be7484ddc7f6ee
      37b9fbde
    • Kirill Smelkov's avatar
      Allow internal clients to specify intended access mode - read-only or read-write · bfeb1690
      Kirill Smelkov authored
      Most of our tools need only read access for working. However e.g.
      FileStorage, when opened in read-write mode, automatically creates
      database file and index.
      
      This way if database is opened in read-write mode a simple typo in path,
      e.g. to `zodb dump path` would lead to:
      
      - new database at path will be created
      - the dump will print nothing (empty database)
      - exit status will be 0 (ok) and no error will be reported.
      
      For this reason it is better tools declare access level they need so for
      read-only access request we can catch it with an error from storage.
      
      This, however, requires quite recent ZODB to work:
      
      https://github.com/zopefoundation/ZODB/pull/153
      
      P.S.
      
      We don't want to force users to always specify read-only in URLs or
      zconf files because:
      
      - this is error prone
      - URL or zconf can be though as of file
      - when a program opens a file the program, not file, declares which type
        of access it wants.
      
      That's why access mode declaration has to be internal.
      bfeb1690
    • Kirill Smelkov's avatar
      Switch to opening storages by URL · 82b06413
      Kirill Smelkov authored
      Previously for specifying a storage we were requiring users to create
      zconfig file and put information about storage there. This is not always
      convenient e.g. for quickly inspecting a file storage or a NEO instance
      here and there for which address and name is already at hand.
      
      So thanks to zodburi we can switch to specifying storages by URL without
      loosing generality as there is still zconfig:// schema which allows to
      configure storages zconfig way.
      
      P.S. for convenience we allow paths without schema to be treated as
          FileStorage (i.e. file:// schema is implicitly used).
      
      P.P.S. zodbanalyze is not affected (for now ?) as it currently
          works with FileStorage only.
      
      [1] http://docs.pylonsproject.org/projects/zodburi/
      82b06413
    • Kirill Smelkov's avatar
      zodb - Tool to drive them all · 984cfe22
      Kirill Smelkov authored
      We already have 3 commands in zodbtools suite (zodbanalyze, zodbdump &
      zodbcmp) and this is going to grow. And it was already noted some time
      ago with TODO (in 66946b8d) that we need only one command driver to
      invoke everything.
      
      So do it: introduce `zodb` command which can invoke other subcommands
      and show general help or help for subcommand or a topic.
      
      The structure is modelled after `git` and `go` commands. Help topics
      are for now empty but we'll add one help topic in the next patch.
      984cfe22
  10. 30 Mar, 2017 1 commit
    • Kirill Smelkov's avatar
      Clarify licensing · 9e4305b8
      Kirill Smelkov authored
      - Nexedi adheres to GPLv3+ with wide exception for Free/Open-source
        software. Use this for our original work.
      
      - Zodbanalyze was initially developed by Zope corp & co. See
      
        d86d04dc	(initial copy of ZODB3-3.9.7's ZODB/scripts/analyze.py.)
        ab17cf2d	(Hook in Nexedi's zodbanalyze)
      
        So follow original Zope's licensing terms for zodbanalyze.
      9e4305b8
  11. 14 Mar, 2017 1 commit