ods_style: use a script to properly get all editable fields

bt5/erp5_ods_style/SkinTemplateItem/portal_skins/erp5_ods_style/Listbox_getEditableFieldDict.xml

+<?xml version="1.0"?>
+<ZopeData>
+  <record id="1" aka="AAAAAAAAAAE=">
+    <pickle>
+      <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
+    </pickle>
+    <pickle>
+      <dictionary>
+        <item>
+            <key> <string>Script_magic</string> </key>
+            <value> <int>3</int> </value>
+        </item>
+        <item>
+            <key> <string>_bind_names</string> </key>
+            <value>
+              <object>
+                <klass>
+                  <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
+                </klass>
+                <tuple/>
+                <state>
+                  <dictionary>
+                    <item>
+                        <key> <string>_asgns</string> </key>
+                        <value>
+                          <dictionary>
+                            <item>
+                                <key> <string>name_container</string> </key>
+                                <value> <string>container</string> </value>
+                            </item>
+                            <item>
+                                <key> <string>name_context</string> </key>
+                                <value> <string>context</string> </value>
+                            </item>
+                            <item>
+                                <key> <string>name_m_self</string> </key>
+                                <value> <string>script</string> </value>
+                            </item>
+                            <item>
+                                <key> <string>name_subpath</string> </key>
+                                <value> <string>traverse_subpath</string> </value>
+                            </item>
+                          </dictionary>
+                        </value>
+                    </item>
+                  </dictionary>
+                </state>
+              </object>
+            </value>
+        </item>
+        <item>
+            <key> <string>_body</string> </key>
+            <value> <string>request = container.REQUEST\n
+\n
+# XXX this is a restricted environment available reimplementation of ListboxRenderer.getEditableField\n
+editable_columns = set(context.get_value(\'columns\', REQUEST=request)\n
+  + context.get_value(\'all_columns\', REQUEST=request))\n
+\n
+editable_fields = {}\n
+\n
+def getEditableField(alias):\n
+  """Get an editable field for column, using column alias.\n
+  Return None if a field for this column does not exist.\n
+  """\n
+  field = context\n
+  original_field_id = field.id\n
+  while True:\n
+    for field_id in {original_field_id, field.id}:\n
+      if field.aq_parent.has_field("%s_%s" % (field_id, alias), include_disabled=1):\n
+        return field.aq_parent.get_field("%s_%s" % (field_id, alias),\n
+                                         include_disabled=1)\n
+    if field.meta_type != \'ProxyField\':\n
+      return None\n
+    # if we are rendering a proxy field, also look for editable fields from\n
+    # the template field\'s form. This editable field can be prefixed either\n
+    # by the template field listbox id or by the proxy field listbox id.\n
+    field = field.getTemplateField().aq_inner\n
+\n
+for column, _ in editable_columns:\n
+  field = getEditableField(column)\n
+  if field is not None:\n
+    editable_fields[column] = field\n
+  \n
+return editable_fields\n
+</string> </value>
+        </item>
+        <item>
+            <key> <string>_params</string> </key>
+            <value> <string></string> </value>
+        </item>
+        <item>
+            <key> <string>id</string> </key>
+            <value> <string>Listbox_getEditableFieldDict</string> </value>
+        </item>
+      </dictionary>
+    </pickle>
+  </record>
+</ZopeData>

bt5/erp5_ods_style/SkinTemplateItem/portal_skins/erp5_ods_style/listbox_ods_macro.xml

@@ -54,7 +54,7 @@
                selection_name python: listbox.get_value(\'selection_name\', REQUEST=request);\n
                editable_columns python: listbox.get_value(\'editable_columns\', REQUEST=request);\n
                untranslatable_columns python: [x[0] for x in listbox.get_value(\'untranslatable_columns\', REQUEST=request)];\n
-               editable_fields python: dict([(column[0], getattr(listbox.aq_parent, \'listbox_%s\' % column[0], None)) for column in editable_columns]);\n
+               editable_fields listbox/Listbox_getEditableFieldDict;\n
                selection python: here.portal_selections.getSelectionFor(selection_name, REQUEST=request);\n
                is_report_tree_mode selection/report_tree_mode | python: 0;\n
                is_domain_tree_mode selection/domain_tree_mode | python: 0;\n

[Georgios Dagkakis]

@jerome sometimes we have a column id with '.', so that we join to a table, e.g. delivery.start_date. In these cases, the 'editable' field would be listbox_delivery_start_date, but it would not be found here. Shouldn't it? Should we do something like:

field = getEditableField(column.replace('.', '_'))

[Jérome Perrin]

@georgios.dagkakis you are absolutely right. This method is same as getEditableField from Listbox, but when listbox calls getEditableField, it uses getColumnAliasList ( see this code block ) and getColumnAliasList is:

  @lazyMethod
  def getColumnAliasList(self):
    """Return the list of column aliases for SQL, because SQL does not allow a symbol to contain dots.
    """
    alias_list = []
    for sql, title in self.getSelectedColumnList():
      alias_list.append(sql.replace('.', '_'))
    return alias_list

so we need to do this replacement somewhere.

So this must be the reason why some listboxs do not use preferences configuration to display date order from editable fields :)

[Georgios Dagkakis]

so we need to do this replacement somewhere.

I would do here if no objection:

- field = getEditableField(column)
+ field = getEditableField(column.replace('.', '_'))

So this must be the reason why some listboxs do not use preferences configuration to display date order from editable fields :)

apparently checking this is how I spotted. But unfortunately not, I get same issue even after this fix and also in 'simple' columns like start_date. But anyway, it is some progress :)

By the way, the name 'ediable' confused me here initially. I thought it was actual editable columns it referred to, but it is not the case, it is listbox columns foo for which we have some utility field listbox_foo

[Jérome Perrin]

Yes, editable columns are both for making an editable <input> field and also just to set the styling when the column is not marked editable. That's true that this seems a step in the good direction anyway.

[Georgios Dagkakis]

So where I am in it...

With the fix above, all the cases would invoke field_ods_macro that is good. Then the code that renders the date is here, 2 branches, one if we have date_only and the other if we do not. I copy one branch for convenience:

                    <tal:block tal:condition="python:not field.get_value('date_only')">
                      <table:table-cell tal:attributes="office:date-value python: context.ERP5Site_formatDateForODF(value);
                                      table:style-name string:${style_prefix}date_with_time;"
                                      table:style-name="date_with_time"
                                      office:value-type="date">
                        <text:p tal:content="python: field.render_pdf(value)"/>
                      </table:table-cell>
                    </tal:block>

my understanding after some pdb-ing is that field.render_pdf(value) works fine to get the order defined in the field BUT this is not what is rendered in the end. It is only what is returned by office:date-value python: context.ERP5Site_formatDateForODF(value); and then this would be formatted somehow indepentently by what the order is (my testing was in Spreadsheet and PDF formats). I checked also style_macros to play with some stuff, but I could not change the order.

So I believe there is something in libre-office xml that somehow defines the output date order. But here my knowledge is limited, so I hit a wall. Any idea where I can find info about this spec?

[Jérome Perrin]

from what I remember we render something like this:

      <table:table-row>
        <table:table-cell office:value-type="date" office:date-value="2023-11-30">
          <text:p>2023-11-30<!-- value rendered by field.render_pdf(value) --></text:p>
        </table:table-cell>
      </table:table-row>

and what Libreoffice uses is the office:date-value ( the content of the text:p is not really used ) and it applies a style for this, so the XML is actually a bit more complex, we have a style:

      <table:table-row>
        <table:table-cell office:value-type="date" office:date-value="2023-11-30" table:style-name="date_with_time">
          <text:p>2023-11-30<!-- value rendered by field.render_pdf(value) --></text:p>
        </table:table-cell>
      </table:table-row>

and the corresponding style, defined here:

    <number:date-style style:name="date_style" number:automatic-order="true">
      <number:day number:style="long"/>
      <number:text>/</number:text>
      <number:month number:style="long"/>
      <number:text>/</number:text>
      <number:year number:style="long"/>
      <number:text> </number:text>
      <number:hours number:style="long"/>
      <number:text>:</number:text>
      <number:minutes number:style="long"/>
      <number:text>:</number:text>
      <number:seconds number:style="long"/>
    </number:date-style>
    <style:style style:name="date_with_time" style:family="table-cell" style:parent-style-name="Default" style:data-style-name="date_style">
      <style:table-cell-properties style:vertical-align='middle' style:repeat-content='false' style:text-align-source='fix' fo:border-right='0.002cm solid #000000' />
    </style:style>

and it's probably this style which defines the ymd or dmy order ... but in reality I don't know when/if this is used.

There is detailed spec at https://docs.oasis-open.org/office/OpenDocument/v1.3/os/part3-schema/OpenDocument-v1.3-os-part3-schema.html#attribute-office_date-value and there must be some introduction documents on internet, there was efforts in making a standard format for document when office softwares were popular some years ago.

I tried to configure person module listbox to display modification_date field and then access URL /person_module/PersonModule_viewPersonList?format=txt&portal_skin=ODS, this gives me a text and when I change date format in preferences the format is updated ... but maybe that does not work when converting to excel ? what was the problem again ? :)

[Jérome Perrin]

ah yes that seems to be the problem, if I use /person_module/PersonModule_viewPersonList?format=xls&portal_skin=ODS then the excel file is always mdy whatever the preferences.

We can probably use this approach to reproduce the problem in a test:

• set preferred date format
• use self.publish('/person_module/PersonModule_viewPersonList?format=xls&portal_skin=ODS') to get a listbox with a date field in excel format
• use DMS or portal_transforms API to convert the excel spreadsheet to text
• assert the text content
• repeat for another date format

[Georgios Dagkakis]

There is detailed spec at https://docs.oasis-open.org/office/OpenDocument/v1.3/os/part3-schema/OpenDocument-v1.3-os-part3-schema.html#attribute-office_date-value and there must be some introduction documents on internet

Thanks, this was exactly what I searched for! Checking a bit I saw I could turn it to %d/%m/%Y format with this change in style_macros:

-    <number:date-style style:name="date_style" number:automatic-order="true">
+    <number:date-style style:name="date_style" number:automatic-order="false" number:country="FRA">

We could probably try to respect the selected date format of the field with some mapping like:

contry = {
  'dmy': 'FRA',
  'mdy': 'USA'
  ...
}.get(selected_date_format, 'dmy')

Not sure if we can access the selected date format of the field from style_macros though.

We can probably use this approach to reproduce the problem in a test: [...]

• use self.publish('/person_module/PersonModule_viewPersonList?format=xls&portal_skin=ODS') to get a listbox with a date field in excel format

This would download file to the system though, right? Wouldn't it be 'easier' to actually run a report and then check the attachment of the last mail send by mailhost? Maybe there is also some way (by preference?) to store the report as document in ERP5?

[Georgios Dagkakis]

By the way, the 'solution' above is only for the not date_only branch of code. The other has table:style-name string:${style_prefix}date;. style_prefix is empty generally, not sure where it is used. Then there is no 'date' style. I added like:

    <number:date-style style:name="only_date_style" number:automatic-order="false" number:country="FRA">
      <number:day number:style="long"/>
      <number:text>/</number:text>
      <number:month number:style="long"/>
      <number:text>/</number:text>
      <number:year number:style="long"/>
    </number:date-style>
    <style:style style:name="date" style:family="table-cell" style:parent-style-name="Default" style:data-style-name="only_date_style">
      <style:table-cell-properties style:vertical-align='middle' style:repeat-content='false' style:text-align-source='fix' fo:border-right='0.002cm solid #000000' />
    </style:style>

and it works similarly

[Jérome Perrin]

yes I was thinking of including <number:date-style> for each of dmy, ymd and mdy and select the good one based on field.get_value("input_order").

About using self.publish vs checking mailhost, it seemed easier to use self.publish because other tests in product/ERP5OOo/tests/testOOoStyle.py do this way. Some of these tests use FooModule_viewFooList which has a listbox_delivery_start_date field, so it seems perfect.

As you can see there are a base class for tests applying to both ods and odt styles, this sounds like a test only for TestODSStyle, so we can add the test directly in TestODSStyle I guess

[Georgios Dagkakis]

yes I was thinking of including <number:date-style> for each of dmy, ymd and mdy and select the good one based on field.get_value("input_order").

Not sure if you mean this, but changing stuff like put number:day first etc does not make a change. I mean:

      <number:day number:style="long"/>
      <number:text>/</number:text>
      <number:month number:style="long"/>
      <number:text>/</number:text>

and

      <number:month number:style="long"/>
      <number:text>/</number:text>
      <number:day number:style="long"/>
      <number:text>/</number:text>

lead to same result. This is why the best way I find for the moment is to set number:country.

About using self.publish vs checking mailhost, it seemed easier to use self.publish because other tests in product/ERP5OOo/tests/testOOoStyle.py do this way. Some of these tests use FooModule_viewFooList which has a listbox_delivery_start_date field, so it seems perfect.

As you can see there are a base class for tests applying to both ods and odt styles, this sounds like a test only for TestODSStyle, so we can add the test directly in TestODSStyle I guess

OK thanks. So I put on my to-do to think of the generic solution for this and pack it in a merge request with a test. Not sure how fast it would be since there are other pending tasks.

The + field = getEditableField(column.replace('.', '_')) part I plan to commit now, without adding test for it, if that's OK

[Jérome Perrin]

ah that's strange if number:country works but not changing order of number:day and number:month, because it seems that the style is applied somehow. One thing that helped me when working with ods was to use libreoffice and inspect the saved files, an ods style is just a zip file with xml files inside (with one for the styles and one for the content if I remember)

[Georgios Dagkakis]

Yes, and other changes work, e.g.:

      <number:year number:style="long"/>
-      <number:text> </number:text>
+      <number:text>-</number:text>
      <number:hours number:style="long"/>

does work, the - s added.

My best guess is that there is some internal code on libreoffice conversion that respects the date format order that is supposed to be defined by the country, no matter what the order of the style is. And it somehow makes sense, otherwise one would need different styles for each date format order.