- refactor usage of TID backup and restore by cutting restored file instead...

 - refactor usage of TID backup and restore by cutting restored file instead of doing backup until found TID
 - spellcheking corrections
 - described new behaviour in documentation
 - create methods to do typical tasks in repozo related scripts
 - forced usage of python2.4 when using ZODB
 - cleaned up files docstrings
 - cleaned up options analysis


git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@25836 20353a03-c40f-0410-a6d1-a30d3c3de9de

product/TIDStorage/README

@@ -28,6 +28,13 @@ because transactions are stored in ZODB in the order they are committed.
 So is T2 is in the backup, a part of T1 will also be, and backup will be
 inconsistent (T1 commit on B never happened).
 
+TIDStorage log and server log
+-----------------------------
+
+TIDStorage uses two logfiles - one which is used to inform administrator
+about server state (logfile_name in configuration) and TIDStorage log to which
+TIDs are appended (status_file in configuration).
+
 USAGE
 =====
 
@@ -49,6 +56,31 @@ Example:
 
 PYTHONPATH=/usr/lib/erp5/lib/python:/usr/lib/erp5/lib/python/Products/TIDStorage
 
+Typical scenario with failure, restoring from backup
+----------------------------------------------------
+
+ * Zopes and Zeos running
+ * TIDStorage running
+ * backups done using repozo/repozo_tidstorage.py (they might contain
+   incoherency), for every backup tidstorage.tid is saved
+ * system failure
+ * restore using repozo/repozo_tidstorage.py with -t tidstorage.tid from last
+   backup
+
+In this scenario only on restoration destination file is cut at point of last
+known TID position. This step is optional, as in some cases administrator
+might want to not cut this file.
+
+Typical scenario with failure, no restoring needed
+--------------------------------------------------
+
+ * Zopes and Zeos running
+ * TIDStorage running
+ * system failure
+ * no need to restore from backup, but there might be some laying transactions
+   in different ZODB files, system is incoherent
+ * administrator use repozo/restore_tidstorage.py to cut not correctly commited
+   transactions, system is coherent again
 
 TECHNICAL DETAILS
 =================
@@ -70,11 +102,13 @@ TIDStorage is composed of 3 parts:
  - A daemon
    This is TIDStorage itself, receiving TIDs from Zopes and delivering
    coherency points to backup scripts.
- - Backup scripts
+ - Backup scripts and other utilities
    Those scripts are (mostly) wrappers for repozo backup script, fetching
    coherency points from TIDStorage daemon and invoking repozo.
-   This requires a patch to be applied to regular repozo, so that it can
-   backup ZODBs only up to a given TID.
+   No changes to repozo.py are needed, as it is used only as subsystem
+   to do reliable backups and restore.
+   Using provided utils in utils/ directory is it possible to query
+   for last known TID from server and operate on TIDStorage log.
 
 Constraints under which TIDStorage was designed:
  - Zope performance
@@ -100,13 +134,15 @@ Constraints under which TIDStorage was designed:
    from crashed ones - as long as they are not corrupted.
 
 Limits:
- - Backup "lag"
-   As TIDStorage can only offer a coherency point when interdependent
-   transactions are all finished (committed or aborted), a backup started at
-   time T might actually contain data from moments before. There are pathologic
-   cases where no coherency point can be found, so no backup can happen.
-   Also, bootstrap can prevent backups from happening if daemon is
-   misconfigured.
+ - Restore "lag"
+   As TIDStorage can only offer a coherency point when inderdependent
+   transactions are all finished (committed or aborted), TIDStorage log file
+   backup from time T might actually contain data from moments before.
+   So while doing restore with -t option data will be cut to state as
+   time T - undefined, small lag.
+
+   There are even pathologic cases where no coherency point can be found,
+   so TIDStorage log file won't have any information.
 
 PROTOCOL SPECIFICATION
 ======================

product/TIDStorage/repozo/from_z2.8.8_repozo.diff

---- /home/vincent/bin/zope2.8/bin/repozo.py	2007-02-09 13:52:35.000000000 +0100
-+++ repozo.py	2007-10-26 15:30:43.311046075 +0200
-@@ -50,6 +50,12 @@
-         Compress with gzip the backup files.  Uses the default zlib
-         compression level.  By default, gzip compression is not used.
- 
-+    -m / --max-tid
-+        Stop at given TID when saving the Data.fs.
-+
-+    -M / --print-max-tid
-+        Print the last saved transaction's tid.
-+
- Options for -R/--recover:
-     -D str
-     --date=str
-@@ -70,6 +76,7 @@
- import time
- import errno
- import getopt
-+import base64
- 
- from ZODB.FileStorage import FileStorage
- 
-@@ -104,10 +111,11 @@
- def parseargs():
-     global VERBOSE
-     try:
--        opts, args = getopt.getopt(sys.argv[1:], 'BRvhf:r:FD:o:Qz',
-+        opts, args = getopt.getopt(sys.argv[1:], 'BRvhf:r:FD:o:Qzm:M',
-                                    ['backup', 'recover', 'verbose', 'help',
-                                     'file=', 'repository=', 'full', 'date=',
--                                    'output=', 'quick', 'gzip'])
-+                                    'output=', 'quick', 'gzip', 'max-tid=',
-+                                    'print-max-tid'])
-     except getopt.error, msg:
-         usage(1, msg)
- 
-@@ -120,6 +128,8 @@
-         output = None       # where to write recovered data; None = stdout
-         quick = False       # -Q flag state
-         gzip = False        # -z flag state
-+        print_tid = False   # -M flag state
-+        max_tid = None      # -m argument, if any
- 
-     options = Options()
- 
-@@ -150,6 +160,10 @@
-             options.output = arg
-         elif opt in ('-z', '--gzip'):
-             options.gzip = True
-+        elif opt in ('-M', '--print-max-tid'):
-+             options.print_tid = True
-+        elif opt in ('-m', '--max-tid'):
-+             options.max_tid = base64.decodestring(arg)
-         else:
-             assert False, (opt, arg)
- 
-@@ -174,6 +188,12 @@
-         if options.file is not None:
-             log('--file option is ignored in recover mode')
-             options.file = None
-+        if options.print_tid:
-+            log('--print-max-tid is ignored in recover mode')
-+            options.print_tid = False
-+        if options.max_tid is not None:
-+            log('--max-tid is ignored in recover mode')
-+            options.max_tid = None
-     return options
- 
- 
-@@ -349,13 +369,19 @@
- 
- def do_full_backup(options):
-     # Find the file position of the last completed transaction.
--    fs = FileStorage(options.file, read_only=True)
-+    fs = FileStorage(options.file, read_only=True, stop=options.max_tid)
-     # Note that the FileStorage ctor calls read_index() which scans the file
-     # and returns "the position just after the last valid transaction record".
-     # getSize() then returns this position, which is exactly what we want,
-     # because we only want to copy stuff from the beginning of the file to the
-     # last valid transaction record.
-     pos = fs.getSize()
-+    if options.print_tid:
-+      undo_log = fs.undoLog(last=-1)
-+      if len(undo_log):
-+        print >> sys.stdout, 'Last TID: %s' % (undo_log[0]['id'], )
-+      else:
-+        print >> sys.stderr, 'Cannot get latest TID'
-     fs.close()
-     options.full = True
-     dest = os.path.join(options.repository, gen_filename(options))
-@@ -375,13 +401,19 @@
- 
- def do_incremental_backup(options, reposz, repofiles):
-     # Find the file position of the last completed transaction.
--    fs = FileStorage(options.file, read_only=True)
-+    fs = FileStorage(options.file, read_only=True, stop=options.max_tid)
-     # Note that the FileStorage ctor calls read_index() which scans the file
-     # and returns "the position just after the last valid transaction record".
-     # getSize() then returns this position, which is exactly what we want,
-     # because we only want to copy stuff from the beginning of the file to the
-     # last valid transaction record.
-     pos = fs.getSize()
-+    if options.print_tid:
-+      undo_log = fs.undoLog(last=-1)
-+      if len(undo_log):
-+        print >> sys.stdout, 'Last TID: %s' % (undo_log[0]['id'], )
-+      else:
-+        print >> sys.stderr, 'Cannot get latest TID'
-     fs.close()
-     options.full = False
-     dest = os.path.join(options.repository, gen_filename(options))

product/TIDStorage/repozo/restore_tidstorage.py

@@ -79,6 +79,19 @@ def parse(status_file):
 
 READCHUNK = 10 * 1024 * 1024
 
+def get_tid_position(filepath,last_tid):
+  tid = pack('>Q', last_tid + 1)
+  # Find the file position of the last completed transaction.
+  fs = FileStorage(filepath, read_only=True, stop=tid)
+  # Note that the FileStorage ctor calls read_index() which scans the file
+  # and returns "the position just after the last valid transaction record".
+  # getSize() then returns this position, which is exactly what we want,
+  # because we only want to copy stuff from the beginning of the file to the
+  # last valid transaction record.
+  pos = fs.getSize()
+  fs.close()
+  return pos
+
 def recover(data_fs_backup_path_dict, status_file):
   last_tid_dict = parse(status_file)
   for storage_id, (file_path, backup_path) in data_fs_backup_path_dict.iteritems():
@@ -105,17 +118,7 @@ def recover(data_fs_backup_path_dict, status_file):
       else:
         print 'Cannot find any file for %r: %r and %r do not exist.' % (storage_id, file_path, backup_path)
     if can_restore:
-      last_tid = last_tid_dict[storage_id] + 1
-      tid = pack('>Q', last_tid)
-      # Find the file position of the last completed transaction.
-      fs = FileStorage(backup_path, read_only=True, stop=tid)
-      # Note that the FileStorage ctor calls read_index() which scans the file
-      # and returns "the position just after the last valid transaction record".
-      # getSize() then returns this position, which is exactly what we want,
-      # because we only want to copy stuff from the beginning of the file to the
-      # last valid transaction record.
-      pos = fs.getSize()
-      fs.close()
+      pos = get_tid_position(backup_path,last_tid_dict[storage_id])
       print 'Restoring backup: %s bytes (transaction %r) from %s to %s' % (pos, tid, backup_path, file_path)
       source_file = open(backup_path, 'rb')
       destination_file = open(file_path, 'wb')

product/TIDStorage/repozo/sample_configuration.py

 # COMMON
-# This part is used both by server_v2.py and repozo_tidstorage_v2.py
+# This part is used both by tidstorage.py and repozo_tidstorage.py
 known_tid_storage_identifier_dict = {
   "((('localhost', 8200),), '2')":
     ('/home/vincent/zeo2/var2/Data.fs',
@@ -30,6 +30,8 @@ burst_period = 30
 full_dump_period = 300
 
 # REPOZO_TIDSTORAGE
-# This part is only used by repozo_tidstorage_v2.py
+# This part is only used by repozo_tidstorage.py
 timestamp_file_path = 'repozo_tidstorage_timestamp.log'
+# place to put backuped TIDStorage status_file logs
+status_file_backup_dir = '/home/vincent/tmp/repozo'