Skip to content

erp5
erp5
Commit 1fa75866 authored by Julien Muchembled's avatar Julien Muchembled
Browse files
Options

* ERP5Site_checkCatalogTable: 

  * Add support for 'fixit' parameter. Calls unindexObject for missing objects, and reindexObject when values differ.
  * Fix detection of missing objects in the ZODB. Exclude 'path' from properties to check because it is redundant and "getProperty('path')" does not always work.
  * Fix comparison of float values. Rounding errors could produce false positive.
  * Reduce length of some lines (80 chars).
  * Clean up dead code.
* Delete useless Alarm_activeCheckCatalogTable and update check_catalog alarm.


git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@26854 20353a03-c40f-0410-a6d1-a30d3c3de9de
parent f9097773
Hide whitespace changes
Inline Side-by-side
Showing with 136 additions and 249 deletions
bt5/erp5_administration/PathTemplateItem/portal_alarms/check_catalog.xml
View file @ 1fa75866
......@@ -11,7 +11,15 @@
<dictionary>
<item>
<key> <string>active_sense_method_id</string> </key>
<value> <string>Alarm_activeCheckCatalogTable</string> </value>
<value> <string>Alarm_checkCatalogTable</string> </value>
</item>
<item>
<key> <string>alarm_notification_mode</string> </key>
<value>
<tuple>
<string>problem</string>
</tuple>
</value>
</item>
<item>
<key> <string>description</string> </key>
......@@ -27,6 +35,25 @@
<key> <string>id</string> </key>
<value> <string>check_catalog</string> </value>
</item>
<item>
<key> <string>periodicity_start_date</string> </key>
<value>
<object>
<klass>
<global name="DateTime" module="DateTime.DateTime"/>
</klass>
<tuple>
<none/>
</tuple>
<state>
<tuple>
<float>1230768000.0</float>
<string>GMT</string>
</tuple>
</state>
</object>
</value>
</item>
<item>
<key> <string>portal_type</string> </key>
<value> <string>Alarm</string> </value>
......
bt5/erp5_administration/SkinTemplateItem/portal_skins/erp5_administration/Alarm_activeCheckCatalogTable.xml deleted 100644 → 0
View file @ f9097773
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<tuple>
<global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
<tuple/>
</tuple>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>Script_magic</string> </key>
<value> <int>3</int> </value>
</item>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_body</string> </key>
<value> <string>active_process = context.newActiveProcess()\n
context.Alarm_checkCatalogTable(active_process=\'/\'.join(active_process.getPhysicalPath()), tag=tag)\n
</string> </value>
</item>
<item>
<key> <string>_code</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>fixit=0, tag=\'\'</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>func_code</string> </key>
<value>
<object>
<klass>
<global name="FuncCode" module="Shared.DC.Scripts.Signature"/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>2</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
<value>
<tuple>
<string>fixit</string>
<string>tag</string>
<string>_getattr_</string>
<string>context</string>
<string>active_process</string>
</tuple>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>func_defaults</string> </key>
<value>
<tuple>
<int>0</int>
<string></string>
</tuple>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Alarm_activeCheckCatalogTable</string> </value>
</item>
<item>
<key> <string>warnings</string> </key>
<value>
<tuple/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_administration/SkinTemplateItem/portal_skins/erp5_administration/Alarm_checkCatalogTable.xml
View file @ 1fa75866
......@@ -53,11 +53,9 @@
</item>
<item>
<key> <string>_body</string> </key>
<value> <string>context.ERP5Site_checkCatalogTable(bundle_object_count=bundle_object_count,\n
bundle_offset=bundle_offset,\n
activity_count=1,\n
active_process=active_process,\n
tag=tag)\n
<value> <string>context.ERP5Site_checkCatalogTable(\n
active_process=context.newActiveProcess().getPath(),\n
**kw)\n
</string> </value>
</item>
<item>
......@@ -68,7 +66,7 @@
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>bundle_object_count=100, bundle_offset=0, active_process=None, tag=\'\'</string> </value>
<value> <string>**kw</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
......@@ -88,16 +86,14 @@
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>4</int> </value>
<value> <int>0</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
<value>
<tuple>
<string>bundle_object_count</string>
<string>bundle_offset</string>
<string>active_process</string>
<string>tag</string>
<string>kw</string>
<string>_apply_</string>
<string>_getattr_</string>
<string>context</string>
</tuple>
......@@ -111,12 +107,7 @@
<item>
<key> <string>func_defaults</string> </key>
<value>
<tuple>
<int>100</int>
<int>0</int>
<none/>
<string></string>
</tuple>
<none/>
</value>
</item>
<item>
......
bt5/erp5_administration/SkinTemplateItem/portal_skins/erp5_administration/ERP5Site_checkCatalogTable.xml
View file @ 1fa75866
......@@ -62,160 +62,161 @@
Maximum number of objects to deal with in one transaction. \n
An activity is started after each successfull execution which\n
found bundle_object_count to work on.\n
bundle_offset\n
Offset the bundle, to allow walking along the catalog with\n
bundle_object_count increment.\n
property_override_method_id\n
Id of a method that generates a dictionary of reference values\n
for a particular item in the catalog.\n
catalog_kw\n
Extra parameters passed to catalog\n
retry\n
\n
"""\n
from DateTime import DateTime\n
from Products.CMFActivity.ActiveResult import ActiveResult\n
active_result = ActiveResult()\n
active_context = context.getPortalObject().person_module\n
portal = context.getPortalObject()\n
activate = portal.portal_activities.activate\n
result_list = []\n
if catalog_kw is None:\n
catalog_kw = {}\n
\n
if not catalog_kw.has_key(\'sort_on\'):\n
catalog_kw[\'sort_on\'] = ((\'uid\',\'ascending\'),)\n
catalog_kw.setdefault(\'sort_on\', ((\'uid\',\'ascending\'),))\n
\n
if catalog_uid_list is None:\n
# No uid list was given: fetch work to do from catalog and spawn activities\n
first_run = uid_min is None\n
sql_kw = {}\n
if uid_min is not None:\n
# Check what is after last check\n
catalog_kw[\'uid\'] = {\'query\': uid_min, \'range\': \'nlt\'}\n
catalog_uid_list = [x.uid for x in context.portal_catalog(\n
catalog_uid_list = [x.uid for x in portal.portal_catalog(\n
limit=bundle_object_count * activity_count,\n
**catalog_kw)]\n
# Remove the uid once the parameter was given to catalog\n
if catalog_kw.has_key(\'uid\'):\n
catalog_kw.pop(\'uid\')\n
catalog_kw.pop(\'uid\', None)\n
if len(catalog_uid_list):\n
# Get the last uid this pass will check, so that next pass will check a batch starting after this uid.\n
# Get the last uid this pass will check,\n
# so that next pass will check a batch starting after this uid.\n
uid_min = max(catalog_uid_list)\n
# Spawn activities\n
worker_tag = tag + \'_worker\'\n
activity_kw = {\n
\'activity\': \'SQLQueue\',\n
\'priority\': 4,\n
\'lonely\': 1,\n
}\n
check_kw = {\n
\'property_override_method_id\': property_override_method_id,\n
\'active_process\': active_process,\n
\'activity_count\': activity_count,\n
\'bundle_object_count\' : bundle_object_count,\n
\'tag\': tag\n
\'tag\': tag,\n
\'fixit\': fixit,\n
}\n
for activity in xrange(activity_count):\n
if len(catalog_uid_list) == 0:\n
result_list.append(\'No more uids to check, stop spawning activities.\')\n
break\n
activity_catalog_uid_list, catalog_uid_list = catalog_uid_list[:bundle_object_count], catalog_uid_list[bundle_object_count:]\n
result_list.append(\'Spawning activity for range %i..%i (len=%i)\' % (activity_catalog_uid_list[0], activity_catalog_uid_list[-1], len(activity_catalog_uid_list)))\n
active_context.activate(tag=worker_tag, **activity_kw).ERP5Site_checkCatalogTable(catalog_uid_list=activity_catalog_uid_list,\n
catalog_kw=catalog_kw, **check_kw)\n
activity_catalog_uid_list = catalog_uid_list[:bundle_object_count]\n
catalog_uid_list = catalog_uid_list[bundle_object_count:]\n
result_list.append(\'Spawning activity for range %i..%i (len=%i)\'\n
% (activity_catalog_uid_list[0],\n
activity_catalog_uid_list[-1],\n
len(activity_catalog_uid_list)))\n
activate(tag=worker_tag, **activity_kw) \\\n
.ERP5Site_checkCatalogTable(catalog_uid_list=activity_catalog_uid_list,\n
catalog_kw=catalog_kw, **check_kw)\n
else:\n
result_list.append(\'Spawning an activity to fetch a new batch starting above uid %i\' % (uid_min, ))\n
# For loop was not interrupted by a break, which means that all activities got uids to process. Maybe there is another batch of uids to check besides current one. Spawn an activity to process such batch.\n
active_context.activate(after_tag=worker_tag, tag=tag, **activity_kw).ERP5Site_checkCatalogTable(uid_min=uid_min, \n
catalog_kw=catalog_kw, **check_kw)\n
result_list.append(\'Spawning an activity to fetch a new batch starting\'\n
\' above uid %i\' % uid_min)\n
# For loop was not interrupted by a break, which means that all\n
# activities got uids to process. Maybe there is another batch of uids\n
# to check besides current one. Spawn an activity to process such batch.\n
activate(after_tag=worker_tag, tag=tag, **activity_kw) \\\n
.ERP5Site_checkCatalogTable(uid_min=uid_min,\n
catalog_kw=catalog_kw, **check_kw)\n
else:\n
result_list.append(\'Base_zGetAllFromcatalog found no more line to check.\')\n
active_result.edit(summary=\'Spawning activities\', severity=0, detail=\'\\n\'.join(result_list))\n
# Spawn an activity to save generated active result only if it\'s not the initial run\n
if not first_run:\n
active_context.activate(active_process=active_process, activity=\'SQLQueue\', priority=2, tag=tag).ERP5Site_saveCheckCatalogTableResult(active_result)\n
activate(active_process=active_process, activity=\'SQLQueue\', priority=2, tag=tag) \\\n
.ERP5Site_saveCheckCatalogTableResult(active_result)\n
else:\n
# Process given uid list\n
REFERENCE_DATETIME = DateTime()\n
MARKER = []\n
retry_uid_list = []\n
restrictedTraverse = context.getPortalObject().restrictedTraverse\n
null_value_list = (\'\', None, 0.0, 0) # Values which are all considered equal.\n
catalog_line_list = context.portal_catalog(uid=catalog_uid_list, **catalog_kw)\n
restrictedTraverse = portal.restrictedTraverse\n
catalog_line_list = portal.portal_catalog(uid=catalog_uid_list, **catalog_kw)\n
attribute_id_list = catalog_line_list.names()\n
#attribute_id_list.remove(\'catalog_path\')\n
attribute_id_list.remove(\'path\')\n
\n
def error(message):\n
if retry:\n
retry_uid_list.append(catalog_line[\'uid\'])\n
else:\n
result_list.append(message)\n
return fixit\n
\n
def normalize(value):\n
if value not in (\'\', None, 0.0, 0): # values which are all considered equal\n
if isinstance(value, float):\n
return float(str(value))\n
if isinstance(value, DateTime):\n
return DateTime("%s Universal" % value.toZone("Universal").ISO())\n
return value\n
\n
for catalog_line in catalog_line_list:\n
object_path = catalog_line[\'path\']\n
if object_path is None:\n
if retry:\n
retry_uid_list.append(catalog_line[\'uid\'])\n
else:\n
message = \'Object with uid %r has no path in catalog.\' % (catalog_line[\'uid\'], )\n
result_list.append(message)\n
error(\'Object with uid %r has no path in catalog.\' % catalog_line[\'uid\'])\n
continue\n
elif object_path == "deleted":\n
continue\n
try:\n
actual_object = restrictedTraverse(catalog_line[\'path\'])\n
actual_object = restrictedTraverse(object_path)\n
except KeyError:\n
if retry:\n
retry_uid_list.append(catalog_line[\'uid\'])\n
else:\n
message = \'Object with path %r cannot be found in the ZODB.\' % (object_path, )\n
result_list.append(message)\n
actual_object = None\n
if actual_object is None or actual_object.getPath() != object_path:\n
if error(\'Object with path %r cannot be found in the ZODB.\'\n
% object_path):\n
result_list.append(\'Catalog line will be deleted.\')\n
portal.portal_catalog.activate(activity=\'SQLQueue\') \\\n
.unindexObject(uid=catalog_line[\'uid\'])\n
continue\n
if exception_portal_type_list is not None and \\\n
actual_object.getPortalType() in exception_portal_type_list:\n
continue\n
explanation_value = None\n
try:\n
explanation_value = actual_object.getExplanationValue()\n
except AttributeError:\n
pass\n
explanation_value = None\n
# There is already activity changing the state\n
if actual_object.hasActivity() \\\n
or (explanation_value is not None \\\n
and explanation_value.hasActivity()):\n
continue\n
continue\n
if property_override_method_id is None:\n
reference_dict = {}\n
reference_dict = {\'uid\': actual_object.getUid()}\n
else:\n
reference_dict = getattr(context, property_override_method_id)(instance=actual_object)\n
do_reindex = False\n
for attribute_id in attribute_id_list:\n
if not reference_dict.has_key(attribute_id):\n
reference_value = actual_object.getProperty(attribute_id)\n
else:\n
reference_value = reference_dict[attribute_id]\n
catalog_value = catalog_line[attribute_id]\n
reference_can_be_null_value = False\n
if same_type(reference_value, tuple()) or same_type(reference_value, list()):\n
for reference_value_item in reference_value:\n
# We probably do not need this\n
if same_type(reference_value, REFERENCE_DATETIME):\n
reference_value = DateTime("%s Universal" % reference_value.toZone("Universal").ISO())\n
if reference_value_item in null_value_list:\n
reference_can_be_null_value = True\n
else:\n
if same_type(reference_value, REFERENCE_DATETIME):\n
reference_value = DateTime("%s Universal" % reference_value.toZone("Universal").ISO())\n
if reference_value in null_value_list:\n
reference_can_be_null_value = True\n
if reference_can_be_null_value and catalog_value in null_value_list:\n
continue\n
elif same_type(reference_value, ()):\n
if catalog_value not in reference_value:\n
if retry:\n
retry_uid_list.append(catalog_line[\'uid\'])\n
else:\n
message = \'%s.%s has candidate list %s, but catalog contains %s\' % (actual_object.getRelativeUrl(), attribute_id,\n
repr(reference_value), repr(catalog_value))\n
result_list.append(message)\n
elif reference_value != catalog_value:\n
if retry:\n
retry_uid_list.append(catalog_line[\'uid\'])\n
else:\n
message = \'%s.%s = %s, but catalog contains %s\' % (actual_object.getRelativeUrl(), attribute_id,\n
repr(reference_value), repr(catalog_value))\n
result_list.append(message)\n
catalog_value = normalize(catalog_line[attribute_id])\n
# reference_value may be a list (or tuple) when we don\'t know exactly\n
# what should be the value in the catalog, for example when checking\n
# stocks (1 line with a positive value and another with a negative one).\n
is_reference_value_list = same_type(reference_value, ()) \\\n
or same_type(reference_value, [])\n
if (catalog_value not in map(normalize, not is_reference_value_list\n
and (reference_value,) or reference_value)):\n
if error(\'%s.%s %s %r, but catalog contains %r\'\n
% (actual_object.getRelativeUrl(), attribute_id,\n
is_reference_value_list and \'has candidate list\' or \'=\',\n
reference_value, catalog_line[attribute_id])):\n
do_reindex = True\n
if do_reindex:\n
result_list.append(\'Object %r will be reindexed.\' % object_path)\n
actual_object.reindexObject()\n
\n
summary_list = []\n
begin = catalog_uid_list[0]\n
......@@ -227,8 +228,10 @@ else:\n
summary_list.append(\'Success\')\n
else:\n
summary_list.append(\'Failed\')\n
active_result.edit(summary=\', \'.join(summary_list), severity=severity, detail=\'\\n\'.join(result_list))\n
active_context.activate(active_process=active_process,\n
active_result.edit(summary=\', \'.join(summary_list),\n
severity=severity,\n
detail=\'\\n\'.join(result_list))\n
activate(active_process=active_process,\n
activity=\'SQLQueue\', \n
priority=2,\n
tag=tag).ERP5Site_saveCheckCatalogTableResult(active_result)\n
......@@ -240,18 +243,19 @@ else:\n
activity_kw = {\n
\'activity\': \'SQLQueue\',\n
\'priority\': 4,\n
\'lonely\': 1,\n
}\n
check_kw = {\n
\'property_override_method_id\': property_override_method_id,\n
\'active_process\': active_process,\n
\'activity_count\': activity_count,\n
\'bundle_object_count\' : bundle_object_count,\n
\'tag\': tag\n
\'tag\': tag,\n
\'fixit\': fixit,\n
}\n
active_context.activate(tag=worker_tag, **activity_kw).ERP5Site_checkCatalogTable(catalog_uid_list=retry_uid_list, \n
catalog_kw=catalog_kw, retry=False, **check_kw)\n
\n
activate(tag=worker_tag, **activity_kw) \\\n
.ERP5Site_checkCatalogTable(catalog_uid_list=retry_uid_list, retry=False,\n
catalog_kw=catalog_kw, **check_kw)\n
\n
return active_result\n
</string> </value>
</item>
......@@ -263,7 +267,7 @@ return active_result\n
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>bundle_object_count=100, catalog_uid_list=None, property_override_method_id=None, active_process=None, activity_count=1, get_from_catalog=1, initial_activity_count=None, uid_min=None, tag=\'\', catalog_kw=None, retry=True, exception_portal_type_list=None, **kw</string> </value>
<value> <string>bundle_object_count=100, catalog_uid_list=None, property_override_method_id=None, active_process=None, activity_count=1, uid_min=None, tag=\'\', catalog_kw=None, retry=True, exception_portal_type_list=None, fixit=False, **kw</string> </value>
</item>
<item>
<key> <string>errors</string> </key>
......@@ -283,7 +287,7 @@ return active_result\n
<dictionary>
<item>
<key> <string>co_argcount</string> </key>
<value> <int>12</int> </value>
<value> <int>11</int> </value>
</item>
<item>
<key> <string>co_varnames</string> </key>
......@@ -294,13 +298,12 @@ return active_result\n
<string>property_override_method_id</string>
<string>active_process</string>
<string>activity_count</string>
<string>get_from_catalog</string>
<string>initial_activity_count</string>
<string>uid_min</string>
<string>tag</string>
<string>catalog_kw</string>
<string>retry</string>
<string>exception_portal_type_list</string>
<string>fixit</string>
<string>kw</string>
<string>DateTime</string>
<string>Products.CMFActivity.ActiveResult</string>
......@@ -308,12 +311,12 @@ return active_result\n
<string>active_result</string>
<string>_getattr_</string>
<string>context</string>
<string>active_context</string>
<string>portal</string>
<string>activate</string>
<string>result_list</string>
<string>None</string>
<string>_write_</string>
<string>first_run</string>
<string>sql_kw</string>
<string>_write_</string>
<string>append</string>
<string>$append0</string>
<string>_getiter_</string>
......@@ -328,33 +331,29 @@ return active_result\n
<string>activity</string>
<string>_getitem_</string>
<string>activity_catalog_uid_list</string>
<string>REFERENCE_DATETIME</string>
<string>MARKER</string>
<string>retry_uid_list</string>
<string>restrictedTraverse</string>
<string>null_value_list</string>
<string>catalog_line_list</string>
<string>attribute_id_list</string>
<string>catalog_line</string>
<string>error</string>
<string>normalize</string>
<string>object_path</string>
<string>message</string>
<string>actual_object</string>
<string>KeyError</string>
<string>explanation_value</string>
<string>AttributeError</string>
<string>reference_dict</string>
<string>getattr</string>
<string>False</string>
<string>do_reindex</string>
<string>attribute_id</string>
<string>reference_value</string>
<string>catalog_value</string>
<string>False</string>
<string>reference_can_be_null_value</string>
<string>same_type</string>
<string>tuple</string>
<string>list</string>
<string>reference_value_item</string>
<string>is_reference_value_list</string>
<string>map</string>
<string>True</string>
<string>repr</string>
<string>summary_list</string>
<string>begin</string>
<string>end</string>
......@@ -377,13 +376,12 @@ return active_result\n
<none/>
<none/>
<int>1</int>
<int>1</int>
<none/>
<none/>
<string></string>
<none/>
<int>1</int>
<none/>
<int>0</int>
</tuple>
</value>
</item>
......
bt5/erp5_administration/bt/revision
View file @ 1fa75866
61
\ No newline at end of file
62
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7