Migration to ERP5 Catalog

Migrating current portal_catalog to erp5 object.

Major changes :

We have new classes for various objects based on base classes which we use for ERP5 objects.

Old class	New class	Parent Class	Portal Type
CatalogTool	ERP5CatalogTool	BaseTool	Catalog Tool
SQLCatalog	Catalog	ERP5Type.Folder	Catalog
ZSQLMethod	SQLMethod	XMLObject	SQL method

All the major attributes for these objects has been changed to properties. For these, we have some property_sheets added.
- Catalog : Properties like sql_clear_catalog, sql_search_result_keys, etc
- CatalogTool : Properties like default_erp5_catalog_id, archive_path, etc
- SQLMethod : Properties like arguments, templates, etc
- CatalogFilter : For CatalogMethod objects, we have added filter properties which earlier used to be an attribute of SQLCatalog saving object in PersistentMapping.

Installation :

New ERP5 Instance : For a new ERP5 instance, you get installed ERP5 Catalog Tool and ERP5 Catalog by default. Concerning sub-objects and python scripts, we have added an extra step in installation of CatalogMethodTemplateItem (mentioned in next section). This part is conversion of catalog methods to ERP5 objects. This adds little extra time to installation of BT5 containing catalog methods.
Running ERP5 Instance: We migrate the objects(portal_catalog as well as its sub objects) dynamically to the new portal_types, thanks to usage of __of__ and __bootstrap. Concerning dynamic migration, it is handled via synchronizeDynamicModule function, which cover all steps in a transaction, hence any error in between rolls back the changes.

Migration Steps for running instance :

Rebase your erp5 repository on new code.
Restart zope process.
Wait for activities to finish. Thereafter you can check from portal_catalog/manage_main that we have catalog and sub-objects as ERP5 objects. But you still won't be able to see the view for those objects, so we move to next step of upgrading Business Templates. It is normal to get error that catalog view is unaccessible. You can check if you get this error both for portal_catalog as well as erp5_mysql_innodb also.
Restart zope process again.
Update genbt5list.
In template tool, upgrade business templates and upgrade erp5_core Business Templates. Wait for activities to finish(may take some time, 1-2 mins). After that, you are good to go and use ERP5 Catalog.

Changes in bt5 installation for CatalogMethodTemplateItem

When we don't have default catalog installaed :
Use case: Installation of BT5 during ERP5 setup
In this case, while installation of erp5_mysql_innodb_catalog, we create the catalog and install all CatalogMethodTemplateItem there. Also, in the mean process, we convert the zsql methods and script(python) to erp5 objects. This process is only carried once, i.e at the installation of catalog. This is done in ObjectTemplateItem.install function
When we have ERP5 Catalog installed :
Use case: Installing BT5 containing ZSQL Method while we have ERP5 Catalog installed
We use install function for Class CatalogMethodTemplateItem in this case. Here, we check for aq_parent of the catalog method and if it is an 'ERP5 Catalog', we convert it to erp5 objects.

Performance :

Adding extra step in BT5 installation adds some extra time to performance.
Because, in every erp5 objects, object lookup is done via _getOb rather than __getattr__, so this adds a bit in improving performance.
Overall, we lose around 2~3% performance.

Problem pertaining/Limitations :

Need to restart erp5_site before upgrading BT5(s) - Current structure of migration uses __of__ function to migrate catalog which in turn makes a call to generation of dynamic classes for catalog object. This is where it calls _importClass method, which ends up calling setDefaultClassProperties function on multiple layer of classes including ERP5Site. Thus, we end up having isRADContent for portal object which comes as a hindrance when we try to index objects in acquisition such as portal_skins, etc.

Extra points to be noted :

Catalog object shouldn't be indexable(because of the problem which may arise due to circular dependency).
Dynamic migration keeps all the old attributes for now, so as not to lose any user made change.

Concerning Links :

Documentation WebPage : https://www.erp5.com/erp5-Migrating.erp5.catalog
Test Suite : https://nexedi.erp5.net/test_suite_module/284
Latest Test Result : https://nexedi.erp5.net/test_result_module/20171120-61F85D0C
Latest Performance Result : https://nexedi.erp5.net/test_result_module/20171120-2615A7B

Migrating current portal_catalog to erp5 object.

# Major changes :

- We have new classes for various objects based on base classes which we use for ERP5 objects.

| Old class | New class | Parent Class  | Portal Type |
| ------------ | ------------- | ----------------- | ----------------|
| CatalogTool | ERP5CatalogTool | BaseTool | Catalog Tool |
| SQLCatalog | Catalog | ERP5Type.Folder | Catalog |
| ZSQLMethod | SQLMethod | XMLObject | SQL method |

- All the major attributes for these objects has been changed to properties. For these, we have some property_sheets added.  
  - **Catalog** : Properties like sql_clear_catalog, sql_search_result_keys, etc
  - **CatalogTool** : Properties like default_erp5_catalog_id, archive_path, etc
  - **SQLMethod** : Properties like arguments, templates, etc
  - **CatalogFilter** :  For CatalogMethod objects, we have added filter properties which earlier used to be an attribute of SQLCatalog saving object in PersistentMapping.

# Installation :  
- **New ERP5 Instance** : For a new ERP5 instance, you get installed ERP5 Catalog Tool and ERP5 Catalog by default. Concerning sub-objects and python scripts, we have added an extra step in installation of `CatalogMethodTemplateItem` (mentioned in next section). This part is conversion of catalog methods to ERP5 objects.
This adds little extra time to installation of BT5 containing catalog methods.
- **Running ERP5 Instance**:  We migrate the objects(`portal_catalog` as well as its sub objects) dynamically to the new portal_types, thanks to usage of `__of__` and `__bootstrap`.  Concerning dynamic migration,  it is handled via `synchronizeDynamicModule` function, which cover all steps in a transaction, hence any error in between rolls back the changes.

# Migration Steps for running instance :  
1. Rebase your erp5 repository on new code.
2. Restart `zope` process.
3. Wait for activities to finish. Thereafter you can check from `portal_catalog/manage_main` that we have catalog and sub-objects as ERP5 objects.
But you still won't be able to see the `view` for those objects, so we move to next step of upgrading Business Templates. *It is normal to get error that catalog view is unaccessible*. You can check if you get this error both for `portal_catalog` as well as `erp5_mysql_innodb` also.
4. Restart `zope` process **again**.
5. Update `genbt5list`.
6. In template tool, upgrade business templates and upgrade `erp5_core` Business Templates. Wait for activities to finish(may take some time, 1-2 mins). After that, you are good to go and use ERP5 Catalog.

**Changes in bt5 installation for CatalogMethodTemplateItem**  
- When we don't have default catalog installaed :   
*Use case: Installation of BT5 during  ERP5 setup*  
In this case, while installation of `erp5_mysql_innodb_catalog`, we create the catalog and install all CatalogMethodTemplateItem there. Also, in the mean process, we convert the zsql methods and script(python) to erp5 objects. This process is only carried once, i.e at the installation of catalog. This is done in `ObjectTemplateItem.install` function
- When we have `ERP5 Catalog` installed :  
*Use case: Installing BT5 containing ZSQL Method while we have ERP5 Catalog installed*  
We use `install` function for `Class CatalogMethodTemplateItem` in this case. Here, we check for `aq_parent` of the catalog method and if it is an 'ERP5 Catalog', we convert it to erp5 objects.

# Performance :   
- Adding extra step in BT5 installation adds some extra time to performance.
- Because, in every erp5 objects, object lookup is done via `_getOb` rather than `__getattr__`, so this adds a bit in improving performance.
- Overall, we lose around 2~3% performance.

# Problem pertaining/Limitations :  
- Need to restart erp5_site before upgrading BT5(s) -  Current structure of migration uses `__of__` function to migrate catalog which in turn makes a call to generation of dynamic classes for catalog object. This is where it calls `_importClass` method, which ends up calling `setDefaultClassProperties` function on multiple layer of classes including `ERP5Site`. Thus, we end up having `isRADContent` for portal object which comes as a hindrance when we try to index objects in acquisition such as `portal_skins`, etc.

# Extra points to be noted :  
- `Catalog` object shouldn't be indexable(because of the problem which may arise due to circular dependency).
- Dynamic migration keeps all the old attributes for now, so as not to lose any user made change.

# Concerning Links :  
- Documentation WebPage                   :  https://www.erp5.com/erp5-Migrating.erp5.catalog  
- Test Suite                                                    :  https://nexedi.erp5.net/test_suite_module/284   
- Latest Test Result                                    :  https://nexedi.erp5.net/test_result_module/20171120-61F85D0C
- Latest Performance Result                  :  https://nexedi.erp5.net/test_result_module/20171120-2615A7B