Skip to content

erp5
erp5
Commit b111b6e3 authored by Rafael Monnerat's avatar Rafael Monnerat
Browse files
Options

erp5_disaster_recovery: Initial public release 

See merge request nexedi/erp5!1900
parents 38483df3 751c2516
Hide whitespace changes
Inline Side-by-side
Showing with 697 additions and 0 deletions
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Folder" module="OFS.Folder"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_objects</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>erp5_disaster_recovery</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.py 0 → 100644
View file @ b111b6e3
portal = context.getPortalObject()
document_list = portal.portal_catalog(
limit=limit,
uid={'query': min_uid, 'range': 'nlt'},
sort_on=(('uid', 'ASC'),),
)
result_count = len(document_list)
if result_count:
if result_count == limit:
portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkDeletedDocumentList(document_list[-1].uid, limit, packet_size)
column_list = [(x.path, x.uid) for x in document_list]
for i in xrange(0, result_count, packet_size):
portal.portal_activities.activate(activity='SQLQueue').ERP5Site_unindexDeletedDocumentList(column_list[i:i+packet_size])
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>min_uid, limit, packet_size</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_checkDeletedDocumentList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.py 0 → 100644
View file @ b111b6e3
from Products.ZSQLCatalog.SQLCatalog import Query
portal = context.getPortalObject()
document_list = portal.portal_catalog(
limit=limit,
uid={'query': min_uid, 'range': 'nlt'},
indexation_timestamp=Query(**{'indexation_timestamp': (before, now), 'range': 'minngt'}),
sort_on=(('uid', 'ASC'),),
)
result_count = len(document_list)
if result_count:
if result_count == limit:
portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkLatestModifiedDocumentList(document_list[-1].uid, limit, packet_size, before, now)
column_list = [(x.path, x.uid) for x in document_list]
for i in xrange(0, result_count, packet_size):
portal.portal_activities.activate(activity='SQLQueue').ERP5Site_reindexOrUnindexDocumentList(column_list[i:i+packet_size])
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>min_uid, limit, packet_size, before, now</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_checkLatestModifiedDocumentList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.py 0 → 100644
View file @ b111b6e3
portal = context.getPortalObject()
now = DateTime()
before = now - int(days_before)
strfstring = '%Y-%m-%d %H:%M:%S'
portal.ERP5Site_checkLatestModifiedDocumentList(0, 1000, 100,
before.strftime(strfstring), now.strftime(strfstring))
# Force reindexation of recently created document
# This expect module to use HBTree
# The following category could be consider as "non-optimistic", but
# it is minimal compared to a whole reindex and it is safe to prevent
# Minor inconsistencies.
for module_id in portal.objectIds(("ERP5 Folder",)):
if module_id.endswith("_module"):
portal[module_id].recurseCallMethod(
'recursiveReindexObject',
max_depth=1,
min_depth=1,
max_retry=0,
activity_count=100,
min_id=before.strftime("%Y%m%d"),
)
portal.ERP5Site_checkDeletedDocumentList(0, 1000, 100)
# Whenever we trust that the catalog is consistent and more recent them the
# ZODB, it isn't required to trigger the whole reindexation of the site.
# If the Mariadb is eventually inconsitent or older them the current ZODB,
# It's required reindex the whole site.
if not optimistic:
for module_id in [
'portal_preferences',
'portal_categories',
'portal_alarms',
'portal_simulation']+portal.objectIds(("ERP5 Folder",)):
portal[module_id].recurseCallMethod(
'immediateReindexObject',
min_depth=1,
max_depth=10000,
activate_kw=dict(
group_method_id='portal_catalog/catalogObjectList',
alternate_method_id='alternateReindexObject',
group_method_cost=1,
priority=6,
),
max_retry=0,
activity_count=100,
)
return "OK"
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>optimistic=True, days_before=1</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_recoverFromRestoration</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.py 0 → 100644
View file @ b111b6e3
from erp5.component.module.Log import log
for path, uid in column_list:
try:
ob = context.restrictedTraverse(path)
except KeyError:
log("object not found", path)
context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
else:
ob.reindexObject()
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>column_list</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_reindexOrUnindexDocumentList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.py 0 → 100644
View file @ b111b6e3
from erp5.component.module.Log import log
for path, uid in column_list:
try:
_ = context.restrictedTraverse(path)
except KeyError:
log("object not found", path)
context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>column_list</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ERP5Site_unindexDeletedDocumentList</string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.py 0 → 100644
View file @ b111b6e3
##############################################################################
#
# Copyright (c) 2002-2024 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly adviced to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
##############################################################################
from Products.ERP5Type.tests.ERP5TypeTestCase import ERP5TypeTestCase
class TestDisasterRecovery(ERP5TypeTestCase):
"""
Test Disaster Recover
"""
def test_missing_catalog(self):
document = self.portal.person_module.newContent(
portal_type='Person',
title='Person for Disaster Recovery'
)
self.tic()
_document = self.portal.portal_catalog.getResultValue(
uid=document.getUid()
)
self.assertNotEqual(_document, None)
self.assertEqual(document.getUid(), _document.getUid())
self.portal.portal_catalog.uncatalog_object(uid=document.getUid())
self.tic()
_document = self.portal.portal_catalog.getResultValue(
uid=document.getUid()
)
self.assertEqual(_document, None)
self.portal.ERP5Site_recoverFromRestoration()
self.tic()
_document = self.portal.portal_catalog.getResultValue(
uid=document.getUid()
)
self.assertNotEqual(_document, None)
self.assertEqual(document.getUid(), _document.getUid())
def test_catalog_but_deleted(self):
if self.portal.person_module.getIdGenerator() != '_generatePerDayId':
self.portal.person_module.setIdGenerator('_generatePerDayId')
document = self.portal.person_module.newContent(
portal_type='Person',
title='Person for Disaster Recovery'
)
self.tic()
_document = self.portal.portal_catalog.getResultValue(
uid=document.getUid()
)
self.assertNotEqual(_document, None)
self.assertEqual(document.getUid(), _document.getUid())
# Force remove the object w/o trigger updates on catalog
self.portal.person_module._objects = tuple([
i for i in self.portal.person_module._objects
if i['id'] != document.getId()])
self.portal.person_module._delOb(document.getId())
self.tic()
connection = self.getSQLConnection()
doc_list = connection.manage_test(
"select * from catalog where path = '/%s/%s'" % (
self.portal.getId(), document.getRelativeUrl()))
self.assertEqual(len(doc_list), 1)
doc_list = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
self.assertEqual(len(doc_list), 1)
self.portal.ERP5Site_recoverFromRestoration()
self.tic()
ac = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
self.assertEqual(len(ac), 0)
_document = self.portal.portal_catalog(uid=document.getUid())
self.assertEqual(len(_document), 0)
def test_cataloged_is_inconsistent(self):
document = self.portal.person_module.newContent(
portal_type='Person',
title='Person for Disaster Recovery %s' % (
str(self.portal.portal_ids.generateNewId(
id_group=('erp5_disaster_recovery_test_id')))))
self.tic()
_document = self.portal.portal_catalog.getResultValue(
uid=document.getUid()
)
self.assertNotEqual(_document, None)
self.assertEqual(document.getUid(), _document.getUid())
connection = self.getSQLConnection()
connection.manage_test("update catalog set title = 'modified title' where uid = %s" % document.getUid())
connection.manage_test("commit")
self.commit()
_document = self.portal.portal_catalog(title='modified title')
self.assertEqual(len(_document), 1)
_document = self.portal.portal_catalog(title=document.getTitle())
self.assertEqual(len(_document), 0)
self.portal.ERP5Site_recoverFromRestoration()
self.tic()
_document = self.portal.portal_catalog(title='modified title')
self.assertEqual(len(_document), 0)
_document = self.portal.portal_catalog(title=document.getTitle())
self.assertEqual(len(_document), 1)
_document = self.portal.portal_catalog(title=document.getTitle())
self.assertEqual(_document[0].getUid(), document.getUid())
bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.xml 0 → 100644
View file @ b111b6e3
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Test Component" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>default_reference</string> </key>
<value> <string>testDisasterRecovery</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>test.erp5.testDisasterRecovery</string> </value>
</item>
<item>
<key> <string>sid</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>text_content_error_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>text_content_warning_message</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>version</string> </key>
<value> <string>erp5</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>component_validation_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>validate</string> </value>
</item>
<item>
<key> <string>validation_state</string> </key>
<value> <string>validated</string> </value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_disaster_recovery/bt/dependency_list 0 → 100644
View file @ b111b6e3
erp5_core
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/description 0 → 100644
View file @ b111b6e3
Business template to help to sync mariadb and Datafs consistency, after a restoration.
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/template_skin_id_list 0 → 100644
View file @ b111b6e3
erp5_disaster_recovery
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/template_test_id_list 0 → 100644
View file @ b111b6e3
test.erp5.testDisasterRecovery
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/test_dependency_list 0 → 100644
View file @ b111b6e3
erp5_base
erp5_full_text_mroonga_catalog
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/title 0 → 100644
View file @ b111b6e3
erp5_disaster_recovery
\ No newline at end of file
bt5/erp5_disaster_recovery/bt/version 0 → 100644
View file @ b111b6e3
0.1
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7