Add a new column to catalog which is automatically updated by MySQL when each...

Add a new column to catalog which is automatically updated by MySQL when each catalog row is modified.
Update z_catalog_object_list to explicitely give the updated columns, otherwise MySQL complains that not all columns are updated.


git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@16229 20353a03-c40f-0410-a6d1-a30d3c3de9de

product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_catalog_object_list.xml

@@ -344,6 +344,11 @@ getModificationDate</string> </value>
 
 REPLACE INTO\n
   catalog\n
+  (`uid`, `security_uid`, `owner`, `path`, `relative_url`, `parent_uid`, `id`, `description`, `title`, `meta_type`,\n
+   `portal_type`, `opportunity_state`, `corporate_registration_code`, `ean13_code`, `validation_state`, `simulation_state`,\n
+   `causality_state`, `invoice_state`, `payment_state`, `event_state`, `immobilisation_state`, `reference`, `grouping_reference`,\n
+   `source_reference`, `destination_reference`, `string_index`, `int_index`, `float_index`, `has_cell_content`, `creation_date`,\n
+   `modification_date`)\n
 VALUES\n
 <dtml-in prefix="loop" expr="_.range(_.len(uid))">\n
 (\n
@@ -421,6 +426,11 @@ VALUES\n
 
 REPLACE INTO\n
   catalog\n
+  (`uid`, `security_uid`, `owner`, `path`, `relative_url`, `parent_uid`, `id`, `description`, `title`, `meta_type`,\n
+   `portal_type`, `opportunity_state`, `corporate_registration_code`, `ean13_code`, `validation_state`, `simulation_state`,\n
+   `causality_state`, `invoice_state`, `payment_state`, `event_state`, `immobilisation_state`, `reference`, `grouping_reference`,\n
+   `source_reference`, `destination_reference`, `string_index`, `int_index`, `float_index`, `has_cell_content`, `creation_date`,\n
+   `modification_date`)\n
 VALUES\n
 <dtml-in prefix="loop" expr="_.range(_.len(uid))">\n
 (\n

product/ERP5/bootstrap/erp5_mysql_innodb_catalog/CatalogMethodTemplateItem/portal_catalog/erp5_mysql_innodb/z_create_catalog.xml

@@ -127,6 +127,7 @@ CREATE TABLE `catalog` (\n
   `has_cell_content` bool,\n
   `creation_date` datetime,\n
   `modification_date` datetime,\n
+  `indexation_timestamp` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n
   PRIMARY KEY  (`uid`),\n
   KEY `security_uid` (`security_uid`),\n
   KEY `owner` (`owner`),\n
@@ -214,6 +215,7 @@ CREATE TABLE `catalog` (\n
   `has_cell_content` bool,\n
   `creation_date` datetime,\n
   `modification_date` datetime,\n
+  `indexation_timestamp` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\n
   PRIMARY KEY  (`uid`),\n
   KEY `security_uid` (`security_uid`),\n
   KEY `owner` (`owner`),\n

product/ERP5/bootstrap/erp5_mysql_innodb_catalog/bt/revision

-54
+80
\ No newline at end of file

[Vincent Pelletier]

I just found out that the TIMESTAMP data type seems to be represented in mysql/mariadb as a 32 bits integer, which means this column will stop working in 2038, at which point it seems this column with be set to 0:

MariaDB [erp5]> update catalog set indexation_timestamp="2040/01/01 00:00:00" where path="/erp5/portal_categories";
Query OK, 1 row affected, 1 warning (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 1

MariaDB [erp5]> show warnings;
+---------+------+---------------------------------------------------------------+
| Level   | Code | Message                                                       |
+---------+------+---------------------------------------------------------------+
| Warning | 1264 | Out of range value for column 'indexation_timestamp' at row 1 |
+---------+------+---------------------------------------------------------------+
1 row in set (0.00 sec)

MariaDB [erp5]> select indexation_timestamp from catalog where path="/erp5/portal_categories";
+----------------------+
| indexation_timestamp |
+----------------------+
| 0000-00-00 00:00:00  |
+----------------------+
1 row in set (0.00 sec)

There may be some more subtleties on this impact, as the documentation also mentions that since mariadb 10.1.2 a different internal representation is used for all time types, which uses more space but allows for more precision and negative values. I cannot find any specific description in the documentation though, so I do not know if this changes such limitation. Of course, the impact depends on when the table was created (or had its schema last altered) rather than the current version.

In any case, DATETIME is documented as being stored as a 64 bits integer, and CURRENT_TIMESTAMP can be used with it, so I would like to switch this column's type to DATETIME. But then, any instance upgrading to such modified version will trigger (in the post-upgrade) an ALTER TABLE on their catalog table, which will prevent any indexation from happening until the table is fully processed (which can take a non-trivial amount of time, hurting availability). With it also comes the (at least transient) roughly doubling of disk space use for the catalog table, which could cause more headaches for an upgrading instance.

A nicer alternative solution would be to run pt-online-schema-change (part of perconna toolkit, which has been part of ERP5 SR for years) ahead of the post-upgrade alarm, which allows writes to the table being altered. But this is not automated. While this does not resolve the double-space-usage, it at least should remove the surprise effect (as the admin is aware that a schema change is needed, and should realize the impact on disk usage).

Are there suggestions on how this should be handled ?

/to: @jerome @romain

[Jérome Perrin]

I don't think we should worry about schema changes, they are supported automatically by upgrader. They cause a short downtime, but in practice for "medium sized" ERP5 it has always been just a few minutes, I don't see a problem here.

[Jérome Perrin]

BTW, I have an ERP5 running with mariadb 10.4.28, I tried update catalog set indexation_timestamp="2040/01/01 00:00:00" where path="/erp5/portal_categories" just in case and the behavior is same as with 10.3.

[Jérome Perrin]

"I don't see a problem" was a bad way of saying, it may sound "I don't care it's not my problem", I did not want to say that :). What I was thinking was that doing the change in the create_catalog.sql seems good, today we have an ALTER TABLE based solution which works and does not cause problem for most of our installations. Maybe in the future we can improve this to integrate pt-online-schema-change instead of ALTER TABLE, this would (if I understand correctly) also work by comparing the create_catalog.sql with the actual catalog table, so we can continue our approach of changing the create sql as a "source" and we can use "better" tools instead when the ALTER TABLE solution is not possible