Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Léo-Paul Géneau
wendelin
Commits
67bd4087
Commit
67bd4087
authored
Jun 25, 2020
by
Roque
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
erp5_wendelin_data_lake_ingestion: data sets and data streams have consistent states
- state after ingestion is "validated"
parent
98c8e221
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
2 additions
and
6 deletions
+2
-6
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/DataIngestionLine_writeEbulkIngestionToDataStream.py
...bles/DataIngestionLine_writeEbulkIngestionToDataStream.py
+1
-2
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_stopIngestionList.py
...ins/erp5_wendelin_data_lake/ERP5Site_stopIngestionList.py
+0
-2
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/IngestionPolicy_getIngestionOperationAndParameterDictEbulk.py
...stionPolicy_getIngestionOperationAndParameterDictEbulk.py
+1
-2
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/DataIngestionLine_writeEbulkIngestionToDataStream.py
View file @
67bd4087
...
...
@@ -30,7 +30,7 @@ portal = context.getPortalObject()
portal_catalog
=
portal
.
portal_catalog
reference_end_split
=
portal
.
ERP5Site_getIngestionReferenceDictionary
()[
"split_end_suffix"
]
#if last chunk of split ingestion -> validate all related data streams
and publish the current one:
#if last chunk of split ingestion -> validate all related data streams
if
data_stream
.
getId
().
endswith
(
reference_end_split
):
query
=
Query
(
portal_type
=
"Data Stream"
,
reference
=
data_stream
.
getReference
(),
validation_state
=
"draft"
)
split_ingestion_data_stream_list
=
portal_catalog
(
query
=
query
,
sort_on
=
((
'creation_date'
,
'ascending'
),))
...
...
@@ -41,4 +41,3 @@ if data_stream.getId().endswith(reference_end_split):
chunk_data_stream
.
validate
()
if
data_stream
.
getValidationState
()
!=
"validated"
:
data_stream
.
validate
()
data_stream
.
publish
()
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/ERP5Site_stopIngestionList.py
View file @
67bd4087
...
...
@@ -69,8 +69,6 @@ for data_ingestion in portal_catalog(portal_type = "Data Ingestion",
data_stream
.
setVersion
(
hash_value
)
if
data_stream
.
getValidationState
()
!=
"validated"
and
data_stream
.
getValidationState
()
!=
"published"
:
data_stream
.
validate
()
if
data_stream
.
getValidationState
()
!=
"published"
:
data_stream
.
publish
()
if
data_ingestion
.
getSimulationState
()
==
"started"
:
data_ingestion
.
stop
()
except
Exception
as
e
:
...
...
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/IngestionPolicy_getIngestionOperationAndParameterDictEbulk.py
View file @
67bd4087
...
...
@@ -117,9 +117,8 @@ data_ingestion.start()
data_operation
=
operation_line
.
getResourceValue
()
data_stream
=
input_line
.
getAggregateDataStreamValue
()
# if not split (one single ingestion) validate
and publish
the data stream
# if not split (one single ingestion) validate the data stream
if
eof
==
reference_end_single
:
data_stream
.
validate
()
data_stream
.
publish
()
return
data_operation
,
{
'data_stream'
:
data_stream
}
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment