Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Xiaowu Zhang
wendelin
Commits
85d24341
Commit
85d24341
authored
Jul 28, 2020
by
Roque
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
erp5_wendelin_data_lake_ingestion: rename data supply
parent
d2600881
Changes
8
Hide whitespace changes
Inline
Side-by-side
Showing
8 changed files
with
19 additions
and
19 deletions
+19
-19
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk.xml
...ion/PathTemplateItem/data_supply_module/default_ebulk.xml
+5
-5
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk/1.xml
...n/PathTemplateItem/data_supply_module/default_ebulk/1.xml
+2
-2
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/default_ebulk/2.xml
...n/PathTemplateItem/data_supply_module/default_ebulk/2.xml
+2
-2
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/IngestionPolicy_getIngestionOperationAndParameterDictEbulk.py
...stionPolicy_getIngestionOperationAndParameterDictEbulk.py
+1
-1
bt5/erp5_wendelin_data_lake_ingestion/TestTemplateItem/portal_components/test.erp5.testDataLakeIngestion.py
...Item/portal_components/test.erp5.testDataLakeIngestion.py
+3
-3
bt5/erp5_wendelin_data_lake_ingestion/bt/template_keep_last_workflow_history_only_path_list
...ion/bt/template_keep_last_workflow_history_only_path_list
+2
-2
bt5/erp5_wendelin_data_lake_ingestion/bt/template_keep_workflow_path_list
...n_data_lake_ingestion/bt/template_keep_workflow_path_list
+2
-2
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
+2
-2
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
em
bulk.xml
→
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
default_e
bulk.xml
View file @
85d24341
...
@@ -86,7 +86,7 @@
...
@@ -86,7 +86,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
default_reference
</string>
</key>
<key>
<string>
default_reference
</string>
</key>
<value>
<string>
em
bulk
</string>
</value>
<value>
<string>
default_e
bulk
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
description
</string>
</key>
<key>
<string>
description
</string>
</key>
...
@@ -132,7 +132,7 @@
...
@@ -132,7 +132,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
id
</string>
</key>
<key>
<string>
id
</string>
</key>
<value>
<string>
em
bulk
</string>
</value>
<value>
<string>
default_e
bulk
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
language
</string>
</key>
<key>
<string>
language
</string>
</key>
...
@@ -146,7 +146,7 @@
...
@@ -146,7 +146,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
title
</string>
</key>
<key>
<string>
title
</string>
</key>
<value>
<string>
E
m
bulk
</string>
</value>
<value>
<string>
Ebulk
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
version
</string>
</key>
<key>
<string>
version
</string>
</key>
...
@@ -329,7 +329,7 @@
...
@@ -329,7 +329,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
serial
</string>
</key>
<key>
<string>
serial
</string>
</key>
<value>
<string>
9
60.15698.49376.7287
</string>
</value>
<value>
<string>
9
85.36429.29664.46233
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
state
</string>
</key>
<key>
<string>
state
</string>
</key>
...
@@ -347,7 +347,7 @@
...
@@ -347,7 +347,7 @@
</tuple>
</tuple>
<state>
<state>
<tuple>
<tuple>
<float>
1
498225133.32
</float>
<float>
1
595958580.55
</float>
<string>
UTC
</string>
<string>
UTC
</string>
</tuple>
</tuple>
</state>
</state>
...
...
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
em
bulk/1.xml
→
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
default_e
bulk/1.xml
View file @
85d24341
...
@@ -214,7 +214,7 @@
...
@@ -214,7 +214,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
serial
</string>
</key>
<key>
<string>
serial
</string>
</key>
<value>
<string>
9
59.7816.8415.26299
</string>
</value>
<value>
<string>
9
85.17805.27243.30754
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
state
</string>
</key>
<key>
<string>
state
</string>
</key>
...
@@ -232,7 +232,7 @@
...
@@ -232,7 +232,7 @@
</tuple>
</tuple>
<state>
<state>
<tuple>
<tuple>
<float>
1
498225125.16
</float>
<float>
1
595958567.14
</float>
<string>
UTC
</string>
<string>
UTC
</string>
</tuple>
</tuple>
</state>
</state>
...
...
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
em
bulk/2.xml
→
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/data_supply_module/
default_e
bulk/2.xml
View file @
85d24341
...
@@ -220,7 +220,7 @@
...
@@ -220,7 +220,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
serial
</string>
</key>
<key>
<string>
serial
</string>
</key>
<value>
<string>
9
59.7816.8415.26299
</string>
</value>
<value>
<string>
9
85.17805.27243.30754
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
state
</string>
</key>
<key>
<string>
state
</string>
</key>
...
@@ -238,7 +238,7 @@
...
@@ -238,7 +238,7 @@
</tuple>
</tuple>
<state>
<state>
<tuple>
<tuple>
<float>
1
498225125
.15
</float>
<float>
1
595958567
.15
</float>
<string>
UTC
</string>
<string>
UTC
</string>
</tuple>
</tuple>
</state>
</state>
...
...
bt5/erp5_wendelin_data_lake_ingestion/SkinTemplateItem/portal_skins/erp5_wendelin_data_lake/IngestionPolicy_getIngestionOperationAndParameterDictEbulk.py
View file @
85d24341
...
@@ -33,7 +33,7 @@ if data_ingestion is not None:
...
@@ -33,7 +33,7 @@ if data_ingestion is not None:
specialise_value_list
=
[
x
.
getObject
()
for
x
in
portal_catalog
.
searchResults
(
specialise_value_list
=
[
x
.
getObject
()
for
x
in
portal_catalog
.
searchResults
(
portal_type
=
'Data Supply'
,
portal_type
=
'Data Supply'
,
reference
=
'
em
bulk'
,
reference
=
'
default_e
bulk'
,
validation_state
=
'validated'
)]
validation_state
=
'validated'
)]
# create a new data ingestion
# create a new data ingestion
...
...
bt5/erp5_wendelin_data_lake_ingestion/TestTemplateItem/portal_components/test.erp5.testDataLakeIngestion.py
View file @
85d24341
...
@@ -216,11 +216,11 @@ class TestDataIngestion(SecurityTestCase):
...
@@ -216,11 +216,11 @@ class TestDataIngestion(SecurityTestCase):
self
.
assertNotEqual
(
None
,
self
.
assertNotEqual
(
None
,
getattr
(
self
.
portal
.
portal_ingestion_policies
,
"default_ebulk"
,
None
))
getattr
(
self
.
portal
.
portal_ingestion_policies
,
"default_ebulk"
,
None
))
self
.
assertNotEqual
(
None
,
self
.
assertNotEqual
(
None
,
getattr
(
self
.
portal
.
data_supply_module
,
"
em
bulk"
,
None
))
getattr
(
self
.
portal
.
data_supply_module
,
"
default_e
bulk"
,
None
))
def
test_04_D
efaultModelSecurityModel
(
self
):
def
test_04_D
atasetAndDatastreamsConsistency
(
self
):
"""
"""
Test
default security model : 'All can download, only contributors can upload.'
Test
that data set state transition also changes its data streams states
"""
"""
data_set
,
data_stream_list
=
self
.
stepIngest
(
self
.
CSV
,
","
,
randomize_ingestion_reference
=
True
)
data_set
,
data_stream_list
=
self
.
stepIngest
(
self
.
CSV
,
","
,
randomize_ingestion_reference
=
True
)
self
.
tic
()
self
.
tic
()
...
...
bt5/erp5_wendelin_data_lake_ingestion/bt/template_keep_last_workflow_history_only_path_list
View file @
85d24341
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
data_product_module/fif_array
data_product_module/fif_array
data_product_module/fif_data
data_product_module/fif_data
data_product_module/fif_descriptor
data_product_module/fif_descriptor
data_supply_module/
em
bulk
data_supply_module/
default_e
bulk
data_supply_module/
em
bulk/**
data_supply_module/
default_e
bulk/**
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/default_ebulk
portal_ingestion_policies/default_ebulk
portal_categories/function/**
portal_categories/function/**
...
...
bt5/erp5_wendelin_data_lake_ingestion/bt/template_keep_workflow_path_list
View file @
85d24341
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
data_product_module/fif_array
data_product_module/fif_array
data_product_module/fif_data
data_product_module/fif_data
data_product_module/fif_descriptor
data_product_module/fif_descriptor
data_supply_module/
em
bulk
data_supply_module/
default_e
bulk
data_supply_module/
em
bulk/**
data_supply_module/
default_e
bulk/**
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/default_ebulk
portal_ingestion_policies/default_ebulk
portal_categories/use/**
portal_categories/use/**
\ No newline at end of file
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
View file @
85d24341
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
...
@@ -3,8 +3,8 @@ data_operation_module/transform_fif
data_product_module/fif_array
data_product_module/fif_array
data_product_module/fif_data
data_product_module/fif_data
data_product_module/fif_descriptor
data_product_module/fif_descriptor
data_supply_module/
em
bulk
data_supply_module/
default_e
bulk
data_supply_module/
em
bulk/**
data_supply_module/
default_e
bulk/**
portal_alarms/wendelin_check_datastream_consistency
portal_alarms/wendelin_check_datastream_consistency
portal_alarms/wendelin_check_datastream_consistency/**
portal_alarms/wendelin_check_datastream_consistency/**
portal_alarms/wendelin_data_lake_handle_analysis
portal_alarms/wendelin_data_lake_handle_analysis
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment