Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Ivan Tyagov
wendelin
Commits
c632c012
Commit
c632c012
authored
Apr 10, 2020
by
Ivan Tyagov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Keep naming conventions.
parent
3dc90e9d
Changes
4
Show whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
242 additions
and
102 deletions
+242
-102
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.py
...portal_callables/IngestionPolicy_handleEmbulkIngestion.py
+37
-0
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.xml
...ortal_callables/IngestionPolicy_handleEmbulkIngestion.xml
+95
-0
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_ingestion_policies/wendelin_embulk.xml
...emplateItem/portal_ingestion_policies/wendelin_embulk.xml
+109
-101
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
+1
-1
No files found.
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.py
0 → 100644
View file @
c632c012
portal
=
context
.
getPortalObject
()
reference_separator
=
portal
.
getIngestionReferenceDictionary
()[
"reference_separator"
]
reference_length
=
portal
.
getIngestionReferenceDictionary
()[
"reference_length"
]
invalid_chars
=
portal
.
getIngestionReferenceDictionary
()[
"invalid_chars"
]
record
=
reference
.
rsplit
(
reference_separator
)
length
=
len
(
record
)
if
(
length
<
reference_length
):
context
.
logEntry
(
"[ERROR] In HandleFifEmbulkIngestion: Data Ingestion reference is not well formated"
)
raise
ValueError
(
"Data Ingestion reference is not well formated."
)
for
char
in
invalid_chars
:
if
char
in
reference
:
context
.
logEntry
(
"[ERROR] In HandleFifEmbulkIngestion: Data Ingestion reference contains chars that are not allowed"
)
raise
ValueError
(
"Data Ingestion reference contains chars that are not allowed."
)
supplier
=
record
[
0
]
dataset_reference
=
record
[
1
]
filename
=
reference_separator
.
join
(
record
[
2
:
-
4
])
extension
=
record
[
length
-
4
]
eof
=
record
[
length
-
3
]
size
=
record
[
length
-
2
]
hash
=
record
[
length
-
1
]
dict
=
{
'filename'
:
filename
,
'extension'
:
extension
,
'eof'
:
eof
,
'supplier'
:
supplier
,
'dataset_reference'
:
dataset_reference
,
'resource_reference'
:
'fif'
,
'size'
:
size
,
'hash'
:
hash
}
return
dict
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_callables/IngestionPolicy_handleEmbulkIngestion.xml
0 → 100644
View file @
c632c012
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"PyData Script"
module=
"erp5.portal_type"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
Script_magic
</string>
</key>
<value>
<int>
3
</int>
</value>
</item>
<item>
<key>
<string>
_bind_names
</string>
</key>
<value>
<object>
<klass>
<global
name=
"NameAssignments"
module=
"Shared.DC.Scripts.Bindings"
/>
</klass>
<tuple/>
<state>
<dictionary>
<item>
<key>
<string>
_asgns
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
name_container
</string>
</key>
<value>
<string>
container
</string>
</value>
</item>
<item>
<key>
<string>
name_context
</string>
</key>
<value>
<string>
context
</string>
</value>
</item>
<item>
<key>
<string>
name_m_self
</string>
</key>
<value>
<string>
script
</string>
</value>
</item>
<item>
<key>
<string>
name_subpath
</string>
</key>
<value>
<string>
traverse_subpath
</string>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key>
<string>
_local_properties
</string>
</key>
<value>
<tuple>
<dictionary>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
reference
</string>
</value>
</item>
<item>
<key>
<string>
type
</string>
</key>
<value>
<string>
string
</string>
</value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key>
<string>
_params
</string>
</key>
<value>
<string>
reference
</string>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<string>
Handles the ingestion and returns the dictionary with the corresponding references.
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
IngestionPolicy_handleEmbulkIngestion
</string>
</value>
</item>
<item>
<key>
<string>
portal_type
</string>
</key>
<value>
<string>
PyData Script
</string>
</value>
</item>
<item>
<key>
<string>
reference
</string>
</key>
<value>
<string>
HandleFifEmbulkIngestion
</string>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string>
Handle Fif Embulk Ingestion
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/PathTemplateItem/portal_ingestion_policies/wendelin_embulk.xml
View file @
c632c012
...
@@ -74,7 +74,7 @@
...
@@ -74,7 +74,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
script_id
</string>
</key>
<key>
<string>
script_id
</string>
</key>
<value>
<string>
HandleFif
EmbulkIngestion
</string>
</value>
<value>
<string>
IngestionPolicy_handle
EmbulkIngestion
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
title
</string>
</key>
<key>
<string>
title
</string>
</key>
...
@@ -123,11 +123,13 @@
...
@@ -123,11 +123,13 @@
</record>
</record>
<record
id=
"3"
aka=
"AAAAAAAAAAM="
>
<record
id=
"3"
aka=
"AAAAAAAAAAM="
>
<pickle>
<pickle>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
patches.WorkflowTool
"
/>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
Workflow
"
/>
</pickle>
</pickle>
<pickle>
<pickle>
<tuple>
<dictionary>
<none/>
<item>
<key>
<string>
_log
</string>
</key>
<value>
<list>
<list>
<dictionary>
<dictionary>
<item>
<item>
...
@@ -150,7 +152,7 @@
...
@@ -150,7 +152,7 @@
</item>
</item>
<item>
<item>
<key>
<string>
serial
</string>
</key>
<key>
<string>
serial
</string>
</key>
<value>
<string>
976.34131.824.20292
</string>
</value>
<value>
<string>
982.48448.18949.19490
</string>
</value>
</item>
</item>
<item>
<item>
<key>
<string>
state
</string>
</key>
<key>
<string>
state
</string>
</key>
...
@@ -168,7 +170,7 @@
...
@@ -168,7 +170,7 @@
</tuple>
</tuple>
<state>
<state>
<tuple>
<tuple>
<float>
1561536855.3
9
</float>
<float>
1586521981.0
9
</float>
<string>
UTC
</string>
<string>
UTC
</string>
</tuple>
</tuple>
</state>
</state>
...
@@ -177,16 +179,20 @@
...
@@ -177,16 +179,20 @@
</item>
</item>
</dictionary>
</dictionary>
</list>
</list>
</tuple>
</value>
</item>
</dictionary>
</pickle>
</pickle>
</record>
</record>
<record
id=
"4"
aka=
"AAAAAAAAAAQ="
>
<record
id=
"4"
aka=
"AAAAAAAAAAQ="
>
<pickle>
<pickle>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
patches.WorkflowTool
"
/>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.
Workflow
"
/>
</pickle>
</pickle>
<pickle>
<pickle>
<tuple>
<dictionary>
<none/>
<item>
<key>
<string>
_log
</string>
</key>
<value>
<list>
<list>
<dictionary>
<dictionary>
<item>
<item>
...
@@ -230,7 +236,9 @@
...
@@ -230,7 +236,9 @@
</item>
</item>
</dictionary>
</dictionary>
</list>
</list>
</tuple>
</value>
</item>
</dictionary>
</pickle>
</pickle>
</record>
</record>
</ZopeData>
</ZopeData>
bt5/erp5_wendelin_data_lake_ingestion/bt/template_path_list
View file @
c632c012
...
@@ -9,7 +9,7 @@ portal_alarms/wendelin_data_lake_handle_analysis
...
@@ -9,7 +9,7 @@ portal_alarms/wendelin_data_lake_handle_analysis
portal_alarms/wendelin_data_lake_handle_analysis/**
portal_alarms/wendelin_data_lake_handle_analysis/**
portal_callables/DataIngestionLine_writeEmbulkIngestionToDataStream
portal_callables/DataIngestionLine_writeEmbulkIngestionToDataStream
portal_callables/DataTransformation_transformFIFDataToDataArray
portal_callables/DataTransformation_transformFIFDataToDataArray
portal_callables/
HandleFif
EmbulkIngestion
portal_callables/
IngestionPolicy_handle
EmbulkIngestion
portal_categories/function/**
portal_categories/function/**
portal_categories/use/**
portal_categories/use/**
portal_ingestion_policies/wendelin_embulk
portal_ingestion_policies/wendelin_embulk
\ No newline at end of file
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment