Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Léo-Paul Géneau
wendelin
Commits
c632c012
Commit
c632c012
authored
Apr 10, 2020
by
Ivan Tyagov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Keep naming conventions.
parent
3dc90e9d
Changes
4
Show whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
242 additions
and
102 deletions
+242
-102
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.py
...portal_callables/IngestionPolicy_handleEmbulkIngestion.py
+37
-0
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.xml
...ortal_callables/IngestionPolicy_handleEmbulkIngestion.xml
+95
-0
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_ingestion_policies/wendelin_embulk.xml
...emplateItem/portal_ingestion_policies/wendelin_embulk.xml
+109
-101
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
+1
-1
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.py
0 → 100644
View file @
c632c012
portal
=
context
.
getPortalObject
()
reference_separator
=
portal
.
getIngestionReferenceDictionary
()[
"reference_separator"
]
reference_length
=
portal
.
getIngestionReferenceDictionary
()[
"reference_length"
]
invalid_chars
=
portal
.
getIngestionReferenceDictionary
()[
"invalid_chars"
]
record
=
reference
.
rsplit
(
reference_separator
)
length
=
len
(
record
)
if
(
length
<
reference_length
):
context
.
logEntry
(
"[ERROR] In HandleFifEmbulkIngestion: Data Ingestion reference is not well formated"
)
raise
ValueError
(
"Data Ingestion reference is not well formated."
)
for
char
in
invalid_chars
:
if
char
in
reference
:
context
.
logEntry
(
"[ERROR] In HandleFifEmbulkIngestion: Data Ingestion reference contains chars that are not allowed"
)
raise
ValueError
(
"Data Ingestion reference contains chars that are not allowed."
)
supplier
=
record
[
0
]
dataset_reference
=
record
[
1
]
filename
=
reference_separator
.
join
(
record
[
2
:
-
4
])
extension
=
record
[
length
-
4
]
eof
=
record
[
length
-
3
]
size
=
record
[
length
-
2
]
hash
=
record
[
length
-
1
]
dict
=
{
'filename'
:
filename
,
'extension'
:
extension
,
'eof'
:
eof
,
'supplier'
:
supplier
,
'dataset_reference'
:
dataset_reference
,
'resource_reference'
:
'fif'
,
'size'
:
size
,
'hash'
:
hash
}
return
dict
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.xml
0 → 100644
View file @
c632c012
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PyData Script"
module=
"erp5.portal_type"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_local_properties
</string>
</key>
<value>
<tuple>
<dictionary>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
reference
</string>
</value>
</item>
<item>
<key>
<string>
type
</string>
</key>
<value>
<string>
string
</string>
</value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
reference
</string>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string>
Handles the ingestion and returns the dictionary with the corresponding references.
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
IngestionPolicy_handleEmbulkIngestion
</string>
</value>
</item>
<item>
<key>
<string>
portal_type
</string>
</key>
<value>
<string>
PyData Script
</string>
</value>
</item>
<item>
<key>
<string>
reference
</string>
</key>
<value>
<string>
HandleFifEmbulkIngestion
</string>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string>
Handle Fif Embulk Ingestion
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_ingestion_policies/wendelin_embulk.xml
View file @
c632c012
...
@@ -74,7 +74,7 @@
...
@@ -74,7 +74,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
script_id
</string>
</key>
<key>
<string>
script_id
</string>
</key>
<value>
<string>
HandleFif
EmbulkIngestion
</string>
</value>
<value>
<string>
IngestionPolicy_handle
EmbulkIngestion
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
title
</string>
</key>
<key>
<string>
title
</string>
</key>
...
@@ -123,11 +123,13 @@
...
@@ -123,11 +123,13 @@
</record>
</record>
<record
id=
"3"
aka=
"AAAAAAAAAAM="
>
<record
id=
"3"
aka=
"AAAAAAAAAAM="
>
<pickle>
<pickle>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
patches.WorkflowTool
"
/>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
Workflow
"
/>
</pickle>
</pickle>
<pickle>
<pickle>
<tuple>
<dictionary>
<none/>
<item>
<key>
<string>
_log
</string>
</key>
<value>
<list>
<list>
<dictionary>
<dictionary>
<item>
<item>
...
@@ -150,7 +152,7 @@
...
@@ -150,7 +152,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
serial
</string>
</key>
<key>
<string>
serial
</string>
</key>
<value>
<string>
976.34131.824.20292
</string>
</value>
<value>
<string>
982.48448.18949.19490
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
state
</string>
</key>
<key>
<string>
state
</string>
</key>
...
@@ -168,7 +170,7 @@
...
@@ -168,7 +170,7 @@
</tuple>
</tuple>
<state>
<state>
<tuple>
<tuple>
<float>
1561536855.3
9
</float>
<float>
1586521981.0
9
</float>
<string>
UTC
</string>
<string>
UTC
</string>
</tuple>
</tuple>
</state>
</state>
...
@@ -177,16 +179,20 @@
...
@@ -177,16 +179,20 @@
</item>
</item>
</dictionary>
</dictionary>
</list>
</list>
</tuple>
</value>
</item>
</dictionary>
</pickle>
</pickle>
</record>
</record>
<record
id=
"4"
aka=
"AAAAAAAAAAQ="
>
<record
id=
"4"
aka=
"AAAAAAAAAAQ="
>
<pickle>
<pickle>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
patches.WorkflowTool
"
/>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
Workflow
"
/>
</pickle>
</pickle>
<pickle>
<pickle>
<tuple>
<dictionary>
<none/>
<item>
<key>
<string>
_log
</string>
</key>
<value>
<list>
<list>
<dictionary>
<dictionary>
<item>
<item>
...
@@ -230,7 +236,9 @@
...
@@ -230,7 +236,9 @@
</item>
</item>
</dictionary>
</dictionary>
</list>
</list>
</tuple>
</value>
</item>
</dictionary>
</pickle>
</pickle>
</record>
</record>
</ZopeData>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
View file @
c632c012
...
@@ -9,7 +9,7 @@ portal_alarms/wendelin_data_lake_handle_analysis
...
@@ -9,7 +9,7 @@ portal_alarms/wendelin_data_lake_handle_analysis
portal_alarms/wendelin_data_lake_handle_analysis/**
portal_alarms/wendelin_data_lake_handle_analysis/**
portal_callables/DataIngestionLine_writeEmbulkIngestionToDataStream
portal_callables/DataIngestionLine_writeEmbulkIngestionToDataStream
portal_callables/DataTransformation_transformFIFDataToDataArray
portal_callables/DataTransformation_transformFIFDataToDataArray
portal_callables/
HandleFif
EmbulkIngestion
portal_callables/
IngestionPolicy_handle
EmbulkIngestion
portal_categories/function/**
portal_categories/function/**
portal_categories/use/**
portal_categories/use/**
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/wendelin_embulk
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment