Stop including empty items in dialog multilistfields

The meaning of empty items for **multi** list fields for categories is
not clear for dialogs (if user does not want to apply any filter, then
the natural way would be to select nothing).

This also caused issues with category fields, when the action script
uses restrictedTraverse to get the uids corresponding to the category
but a path is empty, like for example in 
https://lab.nexedi.com/nexedi/erp5/blob/d51bb0413a806b3db0c5eb69dec06065b9601322/bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/AccountModule_getTrialBalanceReportSectionList.py#L40-48

which does this:

```python
# optional GAP filter
node_uid = []
gap_uid_list = []
for gap in request.get('gap_list', ()):
  gap_uid_list.append(portal.portal_categories.gap.restrictedTraverse(gap).getUid())
if gap_uid_list:
  node_uid = [x.uid for x in portal.portal_catalog(
                                   portal_type='Account',
                                   default_gap_uid=gap_uid_list)] or -1
```

If an empty item is selected, then `gap_uid_list` will contain an entry for 
`portal.portal_categories.gap.restrictedTraverse('').getUid()` which will be the
uid of the gap base category. Searching with a base category uid nowadays does not
match any document, but before 95e3eaec (CMFCategory: Do not index any Base Category
as a related document., 2016-12-21), it was matching all documents having a relation
from this base category and in the case of this trial balance report it was matching all
accounts.

This was a problem for old instances with accounts created before 95e3eaec, because when
they were first indexed, they had the record in category table, so they were matched, but
once they get re-indexed, they no longer had the record, so the result of this report when
selecting the empty item became different, because accounts were no longer included.

Looking back at this, maybe when updating to get 95e3eaec, we should have ran a migration
to delete all these records (re-indexing every document in the background should be enough)
so that if there's a problem, the problem happens right now and not after a few months
after accounts are modified and re-indexed.

When looking at this from end user level, theses empty items not only cause this problem,
but also does not have a clear behaviour and are not needed, so the changes here are about
removing these empty items.

In accounting reports, there was a multi listfield showing all gap categories, "grouped" by
chart of account - but the name of the chart of account was not displayed. This change to
use a None item, which is rendered as disabled to display the chart of account name, but to
do this we had to fix a bug in Formulator, these disabled items were only working properly
for single item widgets, not multiple items widgets.

See merge request !1572

bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/AccountModule_viewDialogFieldLibrary/your_mirror_section_category_list.xml

@@ -263,7 +263,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python:here.portal_categories.role.getCategoryChildTranslatedLogicalPathItemList(base=True)</string> </value>
+            <value> <string>python:here.portal_categories.role.getCategoryChildTranslatedLogicalPathItemList(base=True, display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/Account_getGapItemList.py

@@ -7,32 +7,35 @@ categories from all available GAP.
 
 portal = context.getPortalObject()
 
-display_cache = {}
 def display(x):
-  if x not in display_cache:
-    gap_id = x.getReference()
-    if gap_id:
-      display_cache[x] = '%s - %s' % (
-        gap_id,
-        x.getTranslatedShortTitle() or x.getTranslatedTitle())
-    else:
-      display_cache[x] = x.getIndentedTitle()
+  gap_id = x.getReference()
+  if gap_id:
+    return '%s - %s' % (
+      gap_id,
+      x.getTranslatedShortTitle() or x.getTranslatedTitle())
+  return x.getIndentedTitle()
 
-  return display_cache[x]
 
 def getGapItemList(only_preferred_gap, gap_root=None):
-  if only_preferred_gap:
-    if gap_root:
-      return portal.portal_categories.resolveCategory(gap_root).getCategoryChildItemList(
-        base=False, is_self_excluded=True, display_method=display,
+  if only_preferred_gap and gap_root:
+    return portal.portal_categories.resolveCategory(gap_root).getCategoryChildItemList(
+        base=False,
+        is_self_excluded=True,
+        display_method=display,
+        display_none_category=False,
         local_sort_id=('int_index', 'reference', 'id'))
 
   result = []
-  for country in portal.portal_categories.gap.contentValues():
-    for gap_root in country.contentValues():
-      result.extend(gap_root.getCategoryChildItemList(
-        base=False, is_self_excluded=True, display_method=display,
-        local_sort_id=('int_index', 'reference', 'id')))
+  for gap_root_title, gap_root in context.AccountModule_getAvailableGapList():
+    if gap_root:
+      result.append((gap_root_title, None))
+      result.extend(
+          portal.portal_categories.resolveCategory(gap_root).getCategoryChildItemList(
+              base=False,
+              is_self_excluded=True,
+              display_method=display,
+              display_none_category=False,
+              local_sort_id=('int_index', 'reference', 'id')))
   return result
 
 from Products.ERP5Type.Cache import CachingMethod

bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/Account_hashGapItemList.py

@@ -12,6 +12,8 @@ def getSubFieldDict():
     for item in item_list:
       # Get value of the item
       item_value = item[int(not is_right_display)]
+      if item_value is None:
+        continue
 
       # Hash key from item_value
       item_split = item_value.split('/')

bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/AccountingTransactionModule_viewSearchDialog/your_node.xml

@@ -278,7 +278,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: context.ERP5Site_getAccountItemList(section_category=request.get(\'your_section_category\', preferences.getPreferredAccountingTransactionSectionCategory()), section_category_strict=request.get(\'your_section_category_strict\', preferences.getPreferredAccountingSectionCategoryStrict()), from_date=request.get(\'your_from_date\', preferences.getPreferredAccountingTransactionFromDate()))</string> </value>
+            <value> <string>python: [item for item in context.ERP5Site_getAccountItemList(section_category=request.get(\'your_section_category\', preferences.getPreferredAccountingTransactionSectionCategory()), section_category_strict=request.get(\'your_section_category_strict\', preferences.getPreferredAccountingSectionCategoryStrict()), from_date=request.get(\'your_from_date\', preferences.getPreferredAccountingTransactionFromDate())) if item[0]]</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_accounting/SkinTemplateItem/portal_skins/erp5_accounting/AccountingTransactionModule_viewSearchDialog/your_simulation_state.xml

@@ -278,7 +278,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: context.ERP5Site_getWorkflowStateItemList(portal_type=\'Accounting Transaction\', state_var=\'simulation_state\')</string> </value>
+            <value> <string>python: context.ERP5Site_getWorkflowStateItemList(portal_type=\'Accounting Transaction\', state_var=\'simulation_state\', display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_budget/SkinTemplateItem/portal_skins/erp5_budget/BudgetModule_viewBudgetConsumptionReportDialog/your_specialise.xml

@@ -114,7 +114,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: [(\'\', \'\')] + [(x.getTitle(), x.getRelativeUrl()) for x in context.portal_catalog(portal_type=\'Budget Model\')]</string> </value>
+            <value> <string>python: [(x.getTranslatedTitle(), x.getRelativeUrl()) for x in context.portal_catalog(portal_type=\'Budget Model\')]</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_budget/SkinTemplateItem/portal_skins/erp5_budget/BudgetModule_viewBudgetConsumptionReportDialog/your_validation_state.xml

@@ -280,7 +280,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: context.ERP5Site_getWorkflowStateItemList(portal_type=\'Budget\', state_var=\'validation_state\')</string> </value>
+            <value> <string>python: context.ERP5Site_getWorkflowStateItemList(portal_type=\'Budget\', state_var=\'validation_state\', display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_core_proxy_field_legacy/SkinTemplateItem/portal_skins/erp5_core_proxy_field_legacy/Base_viewDialogFieldLibrary/your_category_list.xml

@@ -263,7 +263,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: here.portal_categories[field.getId().replace(\'your_\', \'\', 1).replace(\'_list\', \'\')].getCategoryChildCompactLogicalPathItemList(local_sort_id="int_index", checked_permission=\'View\')</string> </value>
+            <value> <string>python: here.portal_categories[field.getId().replace(\'your_\', \'\', 1).replace(\'_list\', \'\')].getCategoryChildCompactLogicalPathItemList(local_sort_id="int_index", checked_permission=\'View\', display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_core_test/TestTemplateItem/portal_components/test.erp5.testFields.py

@@ -50,6 +50,7 @@ from Products.ERP5Form.Form import field_value_cache
 from Products.ERP5Form.Form import getFieldValue
 from Products.ERP5Form import ProxyField
 from DateTime import DateTime
+import lxml.html
 
 from Products.Formulator.Widget import NSMAP
 ODG_XML_WRAPPING_XPATH = 'draw:text-box/text:p/text:span'
@@ -534,6 +535,8 @@ class TestListField(ERP5TypeTestCase):
 
   def afterSetUp(self):
     self.field = ListField('test_field')
+    self.field.values['items'] = [('My first Line', '1'), ('My Second Line', '2')]
+    self.field.values['default'] = '2'
     self.widget = self.field.widget
     self.createCategories()
     self.tic()
@@ -553,9 +556,6 @@ class TestListField(ERP5TypeTestCase):
                                       int_index=2)
 
   def test_render_odt(self):
-    items = [('My first Line', '1'), ('My Second Line', '2')]
-    self.field.values['items'] = items
-    self.field.values['default'] = '2'
     element = self.field.render_odt(as_string=False)
     self.assertEqual('{%(text)s}p' % NSMAP, element.tag)
     self.assertEqual('My Second Line', element.text)
@@ -565,6 +565,44 @@ class TestListField(ERP5TypeTestCase):
     element = self.field.render_odt(as_string=False)
     self.assertEqual('??? (3)', element.text)
 
+  def test_render(self):
+    select, input_element, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    # listfields render an input to confirm that the field was posted
+    # in the form's action script
+    self.assertEqual(input_element.name, 'default_field_test_field:int')
+    self.assertEqual(input_element.type, 'hidden')
+
+    self.assertEqual(select.tag, 'select')
+    first, second = select
+    self.assertEqual(first.tag, 'option')
+    self.assertEqual(first.text_content(), 'My first Line')
+    self.assertEqual(first.attrib['value'], '1')
+
+    self.assertEqual(second.tag, 'option')
+    self.assertEqual(second.text_content(), 'My Second Line')
+    self.assertEqual(second.attrib['value'], '2',)
+    self.assertTrue(second.attrib['selected'])
+
+  def test_render_escape_html(self):
+    self.field.values['default'] = ''
+    self.field.values['items'] = [
+        ('<script>alert("text content")</script>', '<script>alert("value")</script>'),]
+    (script, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertEqual(script.attrib['value'], '<script>alert("value")</script>')
+    self.assertEqual(script.text_content(), '<script>alert("text content")</script>')
+    # selected
+    self.field.values['default'] = self.field.values['items'][0][1]
+    (script, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertEqual(script.attrib['value'], '<script>alert("value")</script>')
+    self.assertEqual(script.text_content(), '<script>alert("text content")</script>')
+
+  def test_render_disabled(self):
+    self.field.values['default'] = ''
+    # None items are rendered as disabled
+    self.field.values['items'] = [('Disabled', None)]
+    (disabled, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertTrue(disabled.attrib['disabled'])
+    self.assertFalse(disabled.attrib.get('value'))
 
   def test_listField_value_order(self):
     '''This test check the list field value order
@@ -606,6 +644,46 @@ class TestMultiListField(ERP5TypeTestCase):
     self.field.values['items'] = [('A', 'a',), ('B', 'b')]
     self.field.values['default'] = ['a', 'b']
 
+  def test_render(self):
+    select, input_element, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    # listfields render an input to confirm that the field was posted
+    # in the form's action script
+    self.assertEqual(input_element.name, 'default_field_test_field:int')
+    self.assertEqual(input_element.type, 'hidden')
+
+    self.assertEqual(select.tag, 'select')
+    first, second = select
+    self.assertEqual(first.tag, 'option')
+    self.assertEqual(first.text_content(), 'A')
+    self.assertEqual(first.attrib['value'], 'a')
+    self.assertTrue(second.attrib['selected'])
+
+    self.assertEqual(second.tag, 'option')
+    self.assertEqual(second.text_content(), 'B')
+    self.assertEqual(second.attrib['value'], 'b',)
+    self.assertTrue(second.attrib['selected'])
+
+  def test_render_escape_html(self):
+    self.field.values['default'] = []
+    self.field.values['items'] = [
+        ('<script>alert("text content")</script>', '<script>alert("value")</script>')]
+    (script, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertEqual(script.attrib['value'], '<script>alert("value")</script>')
+    self.assertEqual(script.text_content(), '<script>alert("text content")</script>')
+    # selected
+    self.field.values['default'] = [self.field.values['items'][0][1]]
+    (script, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertEqual(script.attrib['value'], '<script>alert("value")</script>')
+    self.assertEqual(script.text_content(), '<script>alert("text content")</script>')
+
+  def test_render_disabled(self):
+    # None items are rendered as disabled
+    self.field.values['default'] = []
+    self.field.values['items'] = [('Disabled', None)]
+    (disabled, ), _, = lxml.html.fragments_fromstring(self.field.render())  # pylint:disable=unbalanced-tuple-unpacking
+    self.assertTrue(disabled.attrib['disabled'])
+    self.assertFalse(disabled.attrib.get('value'))
+
   def test_render_view(self):
     self.assertEqual('A<br />\nB', self.field.render_view(value=['a', 'b']))
 

bt5/erp5_dms/SkinTemplateItem/portal_skins/erp5_dms/Base_viewAdvancedSearchDialog/your_publication_section_relative_url.xml

@@ -118,7 +118,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: getattr(here.portal_categories[field.getId().replace(\'your_\', \'\', 1).replace(\'_relative_url\', \'\')], preferences.getPreference(\'preferred_category_child_item_list_method_id\', \'getCategoryChildCompactLogicalPathItemList\'))(local_sort_id=(\'int_index\', \'translated_title\'), checked_permission=\'View\', base=1)</string> </value>
+            <value> <string>python: getattr(here.portal_categories[field.getId().replace(\'your_\', \'\', 1).replace(\'_relative_url\', \'\')], preferences.getPreference(\'preferred_category_child_item_list_method_id\', \'getCategoryChildCompactLogicalPathItemList\'))(local_sort_id=(\'int_index\', \'translated_title\'), checked_permission=\'View\', base=1,  display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_trade/SkinTemplateItem/portal_skins/erp5_trade/Base_viewTradeFieldLibrary/my_dialog_mode_use_list.xml

@@ -118,7 +118,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: getattr(here.portal_categories.use, preferences.getPreference(\'preferred_category_child_item_list_method_id\', \'getCategoryChildCompactLogicalPathItemList\'))(local_sort_id=(\'int_index\', \'translated_title\'), checked_permission=\'View\', base=True)</string> </value>
+            <value> <string>python: getattr(here.portal_categories.use, preferences.getPreference(\'preferred_category_child_item_list_method_id\', \'getCategoryChildCompactLogicalPathItemList\'))(local_sort_id=(\'int_index\', \'translated_title\'), checked_permission=\'View\', base=True,  display_none_category=False)</string> </value>
         </item>
       </dictionary>
     </pickle>

product/Formulator/Widget.py

@@ -1027,7 +1027,7 @@ class MultiItemsWidget(ItemsWidget):
       if item_value in value:
         rendered_item = self.render_selected_item(
             escape(ustr(item_text)),
-            escape(ustr(item_value)),
+            item_value,
             key,
             css_class,
             extra_item)
@@ -1037,7 +1037,7 @@ class MultiItemsWidget(ItemsWidget):
       else:
         rendered_item = self.render_item(
             escape(ustr(item_text)),
-            escape(ustr(item_value)),
+            item_value,
             key,
             css_class,
             extra_item)