WIP [erp5_web_renderjs_ui] Replace appcache by a service worker

Fetch usage can be bypassed to do not use service worker when not needed.

As appcache has been dropped on Firefox, this change will improve the speed on Firefox.

No change is expected on Chrome/Safari.

[erp5_web_renderjs_ui] Replace Base_getListFileFromAppcache with Base_getTranslationSourceFileList.

Collect a list of files from service worker code.

[erp5_web_renderjs_ui] Fix rjs_gadget_erp5_serviceworker.js. Use cache.add because safari does not support cache.addAll.

[erp5_web_renderjs_ui] Fix rjs_gadget_erp5_serviceworker.js to fix Firefox Cache Storage problem.

Don't give request object itself to cache.match. Firefox's Cache Storage does not work properly when VARY contains Accept-Language. Give URL string instead, then cache.match works on both Firefox and Chrome.

[erp5_web_renderjs_ui] Stop calling skipWaiting() and clients.claim() for the new service worker.

To preserve the consistency of code and data, let the new service worker wait until all tabs and windows of the old version are closed.

[erp5_web_renderjs_ui] New client can use the latest cache without waiting for the new service worker to be activated.

And once client was associated with a cache, client keeps using the same cache.

[erp5_web_renderjs_ui] Fix translation script. Get service worker filename from layout property. Don't hardcode it.

[erp5_web_renderjs_ui] Fix service worker. Don't hardcode the special cache name.

The name must be different per web site, else if the same service worker code is used by multiple web sites in ERP5 web site module, service worker does not work correctly.

[erp5_web_renderjs_ui] Add more comments because service worker is unstable and hard to use safely.

[erp5_web_renderjs_ui] If service worker failed to install cache, unregister this service worker explicitly.

[erp5_web_renderjs_ui] Clean up service worker code. Remove many `if`.

[erp5_web_renderjs_ui] Update service worker code. Use lower case for non-global variables.

[erp5_web_renderjs_ui] Update service worker code. Client_id is null when it is the first request, in other words if request is navigate mode. Since major web browsers already implement client_id, if client_is is null, let's use the latest cache and don't get cache_key from CACHE_MAP and erp5js_cache.

bt5/erp5_web_renderjs_ui/PathTemplateItem/web_page_module/rjs_gadget_erp5_serviceworker.xml

@@ -228,7 +228,7 @@
             </item>
             <item>
                 <key> <string>serial</string> </key>
-                <value> <string>952.5891.40125.8465</string> </value>
+                <value> <string>976.56996.62202.65433</string> </value>
             </item>
             <item>
                 <key> <string>state</string> </key>
@@ -246,7 +246,7 @@
                     </tuple>
                     <state>
                       <tuple>
-                        <float>1467035757.8</float>
+                        <float>1562322109.66</float>
                         <string>UTC</string>
                       </tuple>
                     </state>

bt5/erp5_web_renderjs_ui/PathTemplateItem/web_site_module/renderjs_runner.xml

@@ -279,6 +279,16 @@
                       <value> <string>string</string> </value>
                   </item>
                 </dictionary>
+                <dictionary>
+                  <item>
+                      <key> <string>id</string> </key>
+                      <value> <string>configuration_service_worker_url</string> </value>
+                  </item>
+                  <item>
+                      <key> <string>type</string> </key>
+                      <value> <string>string</string> </value>
+                  </item>
+                </dictionary>
               </tuple>
             </value>
         </item>
@@ -353,12 +363,18 @@
         </item>
         <item>
             <key> <string>configuration_manifest_url</string> </key>
-            <value> <string>gadget_erp5.appcache</string> </value>
+            <value>
+              <none/>
+            </value>
         </item>
         <item>
             <key> <string>configuration_panel_gadget_url</string> </key>
             <value> <string>gadget_erp5_panel.html</string> </value>
         </item>
+        <item>
+            <key> <string>configuration_service_worker_url</string> </key>
+            <value> <string>gadget_erp5_serviceworker.js</string> </value>
+        </item>
         <item>
             <key> <string>configuration_translation_gadget_url</string> </key>
             <value> <string>gadget_translation.html</string> </value>
@@ -600,7 +616,7 @@
             </item>
             <item>
                 <key> <string>serial</string> </key>
-                <value> <string>965.19244.13768.6758</string> </value>
+                <value> <string>973.33482.4166.8669</string> </value>
             </item>
             <item>
                 <key> <string>state</string> </key>
@@ -618,7 +634,7 @@
                     </tuple>
                     <state>
                       <tuple>
-                        <float>1517843997.55</float>
+                        <float>1553593373.77</float>
                         <string>UTC</string>
                       </tuple>
                     </state>

bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/Base_createTranslateData.py

@@ -15,7 +15,7 @@ tmp_re = re.compile(r"""/[{}]/g, """"")
 translate_word = []
 
 for web_page in portal.web_page_module.searchFolder(portal_type='Web Page',
-                                                    reference=context.Base_getListFileFromAppcache(only_html=1)):
+                                                    reference=context.Base_getTranslationSourceFileList(only_html=1)):
   data = attribute_filter_re.findall(web_page.getTextContent())
   for attribute in data:
     a = re.sub(r'[{|}]', "", attribute[1])

bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/Base_getListFileFromAppcache.py

-appcache_file = context.getLayoutProperty("configuration_manifest_url", default="gadget_erp5.appcache")
-
-
-text_content = context.web_page_module.searchFolder(
-  portal_type= 'Web Manifest',
-  reference = appcache_file)[0].getTextContent()
-
-translation_data_file = []
-file_list = []
-for file in text_content.split('\n'):
-  file = file.split('/')[-1]
-  if file.endswith('.html'):
-    file_list.append(file)
-    continue
-
-  if file.endswith('.js') and not only_html:
-    if file.endswith('translation_data.js'):
-      translation_data_file = [file]
-      continue
-    file_list.append(file)
-return translation_data_file + file_list

bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/Base_getTranslationSourceFileList.py

+import re
+
+service_worker_reference = context.getLayoutProperty("configuration_service_worker_url", default="gadget_erp5_serviceworker.js")
+
+text_content = context.web_page_module.searchFolder(
+  portal_type='Web Script',
+  reference=service_worker_reference)[0].getTextContent()
+
+filename_pattern = re.compile("'(?P<filename>[a-zA-Z0-9-_\.\?=]*)'")
+filename_list = []
+
+start = False
+for line in text_content.split('\n'):
+  if start is False and 'REQUIRED_FILES' in line:
+    start = True
+    continue
+  if not line:
+    continue
+  if start:
+    if ']' in line:
+      break
+    matched = filename_pattern.search(line)
+    if matched is not None:
+      filename = matched.groupdict().get('filename')
+      if filename:
+        filename_list.append(filename)
+
+file_list = []
+translation_data_file = []
+for filename in filename_list:
+  if filename.endswith('.html'):
+    file_list.append(filename)
+    continue
+
+  if filename.endswith('.js') and not only_html:
+    if filename.endswith('translation_data.js'):
+      translation_data_file = [filename]
+      continue
+    file_list.append(filename)
+return translation_data_file + file_list

bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/Base_getListFileFromAppcache.xml → bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/Base_getTranslationSourceFileList.xml

@@ -50,11 +50,11 @@
         </item>
         <item>
             <key> <string>_params</string> </key>
-            <value> <string>only_html = 0</string> </value>
+            <value> <string>only_html=0</string> </value>
         </item>
         <item>
             <key> <string>id</string> </key>
-            <value> <string>Base_getListFileFromAppcache</string> </value>
+            <value> <string>Base_getTranslationSourceFileList</string> </value>
         </item>
       </dictionary>
     </pickle>

bt5/erp5_web_renderjs_ui/SkinTemplateItem/portal_skins/erp5_web_renderjs/WebSite_createTranslationData/your_translation_data_file.xml

@@ -54,6 +54,10 @@
                     <key> <string>field_id</string> </key>
                     <value> <string></string> </value>
                 </item>
+                <item>
+                    <key> <string>first_item</string> </key>
+                    <value> <string></string> </value>
+                </item>
                 <item>
                     <key> <string>form_id</string> </key>
                     <value> <string></string> </value>
@@ -64,6 +68,10 @@
                       <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
                     </value>
                 </item>
+                <item>
+                    <key> <string>title</string> </key>
+                    <value> <string></string> </value>
+                </item>
               </dictionary>
             </value>
         </item>
@@ -107,7 +115,7 @@
       <dictionary>
         <item>
             <key> <string>_text</string> </key>
-            <value> <string>python: [ (x,x) for x in here.Base_getListFileFromAppcache()]</string> </value>
+            <value> <string>python: [ (x,x) for x in here.Base_getTranslationSourceFileList()]</string> </value>
         </item>
       </dictionary>
     </pickle>