Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Léo-Paul Géneau
wendelin
Commits
fd728553
Commit
fd728553
authored
Apr 30, 2020
by
Ivan Tyagov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Add a new interaction workflow which will publish / invalidate all Data Streams per Data Set.
parent
562b39c3
Changes
16
Hide whitespace changes
Inline
Side-by-side
Showing
16 changed files
with
560 additions
and
1 deletion
+560
-1
bt5/erp5_wendelin_data_lake_ingestion/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
...rtalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
+1
-1
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
...kins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
+13
-0
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
...ins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow.xml
...ateItem/portal_workflow/data_set_interaction_workflow.xml
+46
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions.xml
...l_workflow/data_set_interaction_workflow/interactions.xml
+28
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/invalidate.xml
...data_set_interaction_workflow/interactions/invalidate.xml
+100
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/publish.xml
...ow/data_set_interaction_workflow/interactions/publish.xml
+100
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts.xml
...portal_workflow/data_set_interaction_workflow/scripts.xml
+28
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.py
...tion_workflow/scripts/DataSet_invalidateDataStreamList.py
+6
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml
...ion_workflow/scripts/DataSet_invalidateDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.py
...raction_workflow/scripts/DataSet_publishDataStreamList.py
+6
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml
...action_workflow/scripts/DataSet_publishDataStreamList.xml
+62
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/variables.xml
...rtal_workflow/data_set_interaction_workflow/variables.xml
+22
-0
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/worklists.xml
...rtal_workflow/data_set_interaction_workflow/worklists.xml
+22
-0
bt5/erp5_wendelin_data_lake_ingestion/bt/template_portal_type_workflow_chain_list
...ake_ingestion/bt/template_portal_type_workflow_chain_list
+1
-0
bt5/erp5_wendelin_data_lake_ingestion/bt/template_workflow_id_list
...wendelin_data_lake_ingestion/bt/template_workflow_id_list
+1
-0
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PortalTypeWorkflowChainTemplateItem/workflow_chain_type.xml
View file @
fd728553
<workflow_chain>
<chain>
<type>
Data Set
</type>
<workflow>
-validation_workflow, data_validation_workflow
</workflow>
<workflow>
-validation_workflow, data_
set_interaction_workflow, data_
validation_workflow
</workflow>
</chain>
<chain>
<type>
Data Stream
</type>
...
...
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.py
0 → 100644
View file @
fd728553
"""
Return list of Data Streams belonging to a Date Set.
Data Ingestion line aggregates both Data Set and Data Stream.
Note: This code is quite computationally costly (for Data Streams having thousands of iles) as it needs to:
1. Query MariaDB to find ingestion lines
2. Read from ZODB both Data Ingestion Lines and Data Streams (whoch itself can be big too)
"""
data_ingestion_line_list
=
context
.
portal_catalog
(
portal_type
=
"Data Ingestion Line"
,
aggregate_uid
=
context
.
getUid
())
return
[
x
.
getAggregateValue
(
portal_type
=
"Data Stream"
)
\
for
x
in
data_ingestion_line_list
]
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/DataSet_getDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_getDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionWorkflowDefinition"
module=
"Products.ERP5.InteractionWorkflow"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
creation_guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
groups
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
data_set_interaction_workflow
</string>
</value>
</item>
<item>
<key>
<string>
manager_bypass
</string>
</key>
<value>
<int>
0
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string>
Data Set Interaction Workflow
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Interaction"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
interactions
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/invalidate.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionDefinition"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
actbox_category
</string>
</key>
<value>
<string>
workflow
</string>
</value>
</item>
<item>
<key>
<string>
actbox_name
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
actbox_url
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
activate_script_name
</string>
</key>
<value>
<list>
<string>
DataSet_invalidateDataStreamList
</string>
</list>
</value>
</item>
<item>
<key>
<string>
after_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
before_commit_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
invalidate
</string>
</value>
</item>
<item>
<key>
<string>
method_id
</string>
</key>
<value>
<list>
<string>
invalidate
</string>
</list>
</value>
</item>
<item>
<key>
<string>
once_per_transaction
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
portal_type_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
portal_type_group_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
temporary_document_disallowed
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
trigger_type
</string>
</key>
<value>
<int>
2
</int>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/interactions/publish.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"InteractionDefinition"
module=
"Products.ERP5.Interaction"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
actbox_category
</string>
</key>
<value>
<string>
workflow
</string>
</value>
</item>
<item>
<key>
<string>
actbox_name
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
actbox_url
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
activate_script_name
</string>
</key>
<value>
<list>
<string>
DataSet_publishDataStreamList
</string>
</list>
</value>
</item>
<item>
<key>
<string>
after_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
before_commit_script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
guard
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
publish
</string>
</value>
</item>
<item>
<key>
<string>
method_id
</string>
</key>
<value>
<list>
<string>
publish
</string>
</list>
</value>
</item>
<item>
<key>
<string>
once_per_transaction
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
portal_type_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
portal_type_group_filter
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
script_name
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
temporary_document_disallowed
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
trigger_type
</string>
</key>
<value>
<int>
2
</int>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Scripts"
module=
"Products.DCWorkflow.Scripts"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
_objects
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
scripts
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.py
0 → 100644
View file @
fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set
=
state_change
[
'object'
]
for
data_stream
in
data_set
.
DataSet_getDataStreamList
():
data_stream
.
activate
().
invalidate
()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_invalidateDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
state_change,**kw
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_invalidateDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.py
0 → 100644
View file @
fd728553
"""
Publish all Data Streams for a Data Set.
"""
data_set
=
state_change
[
'object'
]
for
data_stream
in
data_set
.
DataSet_getDataStreamList
():
data_stream
.
activate
().
publish
()
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/scripts/DataSet_publishDataStreamList.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PythonScript"
module=
"Products.PythonScripts.PythonScript"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
state_change,**kw
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DataSet_publishDataStreamList
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/variables.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Variables"
module=
"Products.DCWorkflow.Variables"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
variables
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/WorkflowTemplateItem/portal_workflow/data_set_interaction_workflow/worklists.xml
0 → 100644
View file @
fd728553
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Worklists"
module=
"Products.DCWorkflow.Worklists"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_mapping
</string>
</key>
<value>
<dictionary/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
worklists
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/bt/template_portal_type_workflow_chain_list
View file @
fd728553
Data Set | -validation_workflow
Data Set | data_set_interaction_workflow
Data Set | data_validation_workflow
Data Stream | -validation_workflow
Data Stream | data_validation_workflow
\ No newline at end of file
bt5/erp5_wendelin_data_lake_ingestion/bt/template_workflow_id_list
View file @
fd728553
data_set_interaction_workflow
data_validation_workflow
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment