Skip to content

W
wendelin-telecom
Commit 369139eb authored by Xiaowu Zhang's avatar Xiaowu Zhang
Browse files
Options

ors_wendelin: directly create 2 array instead of resampling

parent 17715694
Expand all Show whitespace changes
Inline Side-by-side
Showing with 59 additions and 1232 deletions
bt5/ors_wendelin/PathTemplateItem/data_product_module/ors_resample.xml → bt5/ors_wendelin/PathTemplateItem/data_product_module/ors_kpi.xml
View file @ 369139eb
......@@ -89,7 +89,7 @@
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>ors_resample</string> </value>
<value> <string>ors_kpi</string> </value>
</item>
<item>
<key> <string>description</string> </key>
......@@ -99,7 +99,7 @@
</item>
<item>
<key> <string>id</string> </key>
<value> <string>ors_resample</string> </value>
<value> <string>ors_kpi</string> </value>
</item>
<item>
<key> <string>individual_variation_base_category_list</string> </key>
......@@ -111,7 +111,7 @@
</item>
<item>
<key> <string>title</string> </key>
<value> <string>ORS Resample</string> </value>
<value> <string>ORS KPI</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
......@@ -203,7 +203,7 @@
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1007.25757.13273.63744</string> </value>
<value> <string>1007.25758.5119.31470</string> </value>
</item>
<item>
<key> <string>state</string> </key>
......@@ -223,7 +223,7 @@
</tuple>
<state>
<tuple>
<float>1680184161.47</float>
<float>1681374109.0</float>
<string>UTC</string>
</tuple>
</state>
......
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_resample/1.xml deleted 100644 → 0
View file @ 17715694
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Data Transformation Operation Line" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_identity_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>_range_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>resource/data_operation_module/ors_resample_kpi</string>
<string>trade_phase/data/subsample</string>
</tuple>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>data_operation</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>1</string> </value>
</item>
<item>
<key> <string>int_index</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>quantity</string> </key>
<value> <float>1.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Resample Raw Array</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_resample/2.xml deleted 100644 → 0
View file @ 17715694
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Data Transformation Resource Line" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_identity_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>_range_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>aggregated_portal_type</string> </key>
<value>
<tuple>
<string>Data Array</string>
<string>Progress Indicator</string>
</tuple>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>resource/data_product_module/ors_convert_to_data_array</string>
<string>trade_phase/data/subsample</string>
<string>quantity_unit/unit/piece</string>
</tuple>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>in_array</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>2</string> </value>
</item>
<item>
<key> <string>int_index</string> </key>
<value> <int>1</int> </value>
</item>
<item>
<key> <string>quantity</string> </key>
<value> <float>-1.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Raw Array</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_transformation.xml
View file @ 369139eb
......@@ -240,7 +240,7 @@
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1007.25783.62488.22715</string> </value>
<value> <string>1007.25837.54663.11793</string> </value>
</item>
<item>
<key> <string>state</string> </key>
......@@ -260,7 +260,7 @@
</tuple>
<state>
<tuple>
<float>1680186026.6</float>
<float>1681374021.36</float>
<string>UTC</string>
</tuple>
</state>
......@@ -355,7 +355,7 @@
<key> <string>object_ids</string> </key>
<value>
<list>
<string>4</string>
<string>2</string>
</list>
</value>
</item>
......
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_transformation/2.xml deleted 100644 → 0
View file @ 17715694
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Data Transformation Resource Line" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_identity_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
<item>
<key> <string>_range_criterion</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
<item>
<key> <string>aggregated_portal_type</string> </key>
<value>
<tuple>
<string>Data Array</string>
</tuple>
</value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple>
<string>trade_phase/data/convert</string>
<string>use/big_data/ingestion/stream</string>
<string>resource/data_product_module/ors_convert_to_data_array</string>
<string>variation/data_product_module/ors_convert_to_data_array/1</string>
</tuple>
</value>
</item>
<item>
<key> <string>default_reference</string> </key>
<value> <string>out_array</string> </value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>2</string> </value>
</item>
<item>
<key> <string>int_index</string> </key>
<value> <int>2</int> </value>
</item>
<item>
<key> <string>language</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>quantity</string> </key>
<value> <float>1.0</float> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>Out Array</string> </value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_resample/4.xml → bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_transformation/4.xml
View file @ 369139eb
......@@ -31,10 +31,10 @@
<value>
<tuple>
<string>quantity_unit/unit/piece</string>
<string>resource/data_product_module/ors_resample</string>
<string>resource/data_product_module/ors_kpi</string>
<string>trade_phase/data/subsample</string>
<string>use/big_data/ingestion/stream</string>
<string>variation/data_product_module/ors_resample/2</string>
<string>variation/data_product_module/ors_kpi/2</string>
</tuple>
</value>
</item>
......
bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_resample/3.xml → bt5/ors_wendelin/PathTemplateItem/data_transformation_module/ors_data_transformation/5.xml
View file @ 369139eb
......@@ -31,8 +31,8 @@
<value>
<tuple>
<string>quantity_unit/unit/piece</string>
<string>resource/data_product_module/ors_resample</string>
<string>variation/data_product_module/ors_resample/1</string>
<string>resource/data_product_module/ors_kpi</string>
<string>variation/data_product_module/ors_kpi/1</string>
<string>trade_phase/data/subsample</string>
<string>use/big_data/ingestion/stream</string>
</tuple>
......@@ -50,7 +50,7 @@
</item>
<item>
<key> <string>id</string> </key>
<value> <string>3</string> </value>
<value> <string>5</string> </value>
</item>
<item>
<key> <string>int_index</string> </key>
......
bt5/ors_wendelin/PathTemplateItem/portal_callables/DataAnalysisLine_calculateKPI.py
View file @ 369139eb
import numpy as np
from DateTime import DateTime
progress_indicator = in_stream["Progress Indicator"]
in_data_stream = in_stream["Data Stream"]
out_data_array = out_array["Data Array"]
context.log(in_data_stream.getRelativeUrl())
context.log(out_data_array.getRelativeUrl())
start = progress_indicator.getIntOffsetIndex()
end = in_data_stream.getSize()
if start >= end:
return
e_rab_data_array = None
e_utran_data_array = None
for array in out_array:
if array['variation'] == 'e_rab':
e_rab_data_array = array['Data Array']
if array['variation'] == 'e_utran':
e_utran_data_array = array['Data Array']
data = in_data_stream.getData().decode('utf8')
vt, vInititialEPSBEstabSR, vAddedEPSBEstabSR, evt, vIPThp_qci = context.Base_getORSKPIValue(data)
dtype = np.dtype([
e_rab_dtype = np.dtype([
('vt', 'float'),
('vInititialEPSBEstabSR_lo', 'float64'),
('vInititialEPSBEstabSR_hi', 'float64'),
('vAddedEPSBEstabSR_lo', 'float64'),
('vAddedEPSBEstabSR_hi', 'float64'),
])
e_utran_dtype =np.dtype([
('evt', 'float'),
('vInititialEPSBEstabSR_dl_lo', 'float64'),
('vInititialEPSBEstabSR_dl_hi', 'float64'),
('vAddedEPSBEstabSR_ul_lo', 'float64'),
('vAddedEPSBEstabSR_ul_hi', 'float64'),
('dl_lo', 'float64'),
('dl_hi', 'float64'),
('ul_lo', 'float64'),
('ul_hi', 'float64'),
])
tmp = []
e_rab_array = e_rab_data_array.getArray()
#XXXXXXXXXX initialise every time because of kpi calculation
if not e_rab_array or True:
e_rab_array = e_rab_data_array.initArray(shape=(0,), dtype=e_rab_dtype)
e_rab_array_data = []
e_utran_array = e_utran_data_array.getArray()
#XXXXXXXXXX initialise every time because of kpi calculation
if not e_utran_array or True:
e_utran_array = e_utran_data_array.initArray(shape=(0,), dtype=e_utran_dtype )
e_utran_array_data = []
for i in range(len(vt)):
tmp.append((vt[i], 0, vInititialEPSBEstabSR[i][0], vInititialEPSBEstabSR[i][1], vAddedEPSBEstabSR[i][0], vAddedEPSBEstabSR[i][1]))
e_rab_array_data.append((vt[i], vInititialEPSBEstabSR[i][0], vInititialEPSBEstabSR[i][1], vAddedEPSBEstabSR[i][0], vAddedEPSBEstabSR[i][1]))
for i in range(len(evt)):
for j in vIPThp_qci[i]:
tmp.append((0, evt[i], j[0], j[1], j[2], j[3]))
ndarray = np.ndarray((len(tmp), ), dtype, np.array(tmp))
zbigarray = out_data_array.initArray(shape=(0,), dtype=dtype)
zbigarray.append(ndarray)
e_utran_array_data.append((evt[i], j[0], j[1], j[2], j[3]))
if e_rab_array_data:
e_rab_array_data = np.ndarray((len(e_rab_array_data), ), e_rab_dtype, np.array(e_rab_array_data))
e_rab_array.append(e_rab_array_data)
if e_utran_array_data:
e_utran_array_data = np.ndarray((len(e_utran_array_data), ), e_utran_dtype, np.array(e_utran_array_data))
e_utran_array.append(e_utran_array_data)
progress_indicator.setIntOffsetIndex(end)
......
bt5/ors_wendelin/PathTemplateItem/portal_callables/DataAnalysisLine_calculateKPI.xml
View file @ 369139eb
......@@ -67,7 +67,7 @@
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>in_stream=None, out_array=None</string> </value>
<value> <string>in_stream=None, out_array=[]</string> </value>
</item>
<item>
<key> <string>description</string> </key>
......
bt5/ors_wendelin/PathTemplateItem/portal_callables/DataAnalysisLine_resampleKPI.py deleted 100644 → 0
View file @ 17715694
import numpy as np
in_data_array = in_array["Data Array"]
in_zbigarray = in_data_array.getArray()
if in_zbigarray is None:
return
if in_zbigarray.shape[0] == 0:
return
for array in out_array:
if array['variation'] == 'e_rab':
e_rab_data_array = array['Data Array']
if array['variation'] == 'e_utran':
e_utran_data_array = array['Data Array']
if e_rab_data_array is None:
return
if e_utran_data_array is None:
return
e_rab_dtype = np.dtype([
('vt', 'float'),
('vInititialEPSBEstabSR_lo', 'float64'),
('vInititialEPSBEstabSR_hi', 'float64'),
('vAddedEPSBEstabSR_lo', 'float64'),
('vAddedEPSBEstabSR_hi', 'float64'),
])
e_utran_dtype =np.dtype([
('evt', 'float'),
('dl_lo', 'float64'),
('dl_hi', 'float64'),
('ul_lo', 'float64'),
('ul_hi', 'float64'),
])
e_rab_array = e_rab_data_array.getArray()
if not e_rab_array:
e_rab_array = e_rab_data_array.initArray(shape=(0,), dtype=e_rab_dtype)
e_rab_array_data = []
e_utran_array = e_utran_data_array.getArray()
if not e_utran_array:
e_utran_array = e_utran_data_array.initArray(shape=(0,), dtype=e_utran_dtype )
e_utran_array_data = []
progress_indicator = in_array["Progress Indicator"]
index = progress_indicator.getIntOffsetIndex()
max_len = len(in_zbigarray)
if index >= max_len:
return
end = index + 50000
if end > max_len:
end = max_len
for line in in_zbigarray[index: end]:
if line['vt']:
e_rab_array_data.append((line['vt'], line['vInititialEPSBEstabSR_dl_lo'], line['vInititialEPSBEstabSR_dl_hi'], line['vAddedEPSBEstabSR_ul_lo'], line['vAddedEPSBEstabSR_ul_hi']))
elif line['evt']:
e_utran_array_data.append((line['evt'], line['vInititialEPSBEstabSR_dl_lo'], line['vInititialEPSBEstabSR_dl_hi'], line['vAddedEPSBEstabSR_ul_lo'], line['vAddedEPSBEstabSR_ul_hi']))
if e_rab_array_data:
e_rab_array_data = np.ndarray((len(e_rab_array_data), ), e_rab_dtype, np.array(e_rab_array_data))
e_rab_array.append(e_rab_array_data)
if e_utran_array_data:
e_utran_array_data = np.ndarray((len(e_utran_array_data), ), e_utran_dtype, np.array(e_utran_array_data))
e_utran_array.append(e_utran_array_data)
progress_indicator.setIntOffsetIndex(end)
bt5/ors_wendelin/PathTemplateItem/portal_callables/DataAnalysisLine_resampleKPI.xml deleted 100644 → 0
View file @ 17715694
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="Python Script" module="erp5.portal_type"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_bind_names</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<dictionary>
<item>
<key> <string>_asgns</string> </key>
<value>
<dictionary>
<item>
<key> <string>name_container</string> </key>
<value> <string>container</string> </value>
</item>
<item>
<key> <string>name_context</string> </key>
<value> <string>context</string> </value>
</item>
<item>
<key> <string>name_m_self</string> </key>
<value> <string>script</string> </value>
</item>
<item>
<key> <string>name_subpath</string> </key>
<value> <string>traverse_subpath</string> </value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</state>
</object>
</value>
</item>
<item>
<key> <string>_local_properties</string> </key>
<value>
<tuple>
<dictionary>
<item>
<key> <string>id</string> </key>
<value> <string>reference</string> </value>
</item>
<item>
<key> <string>type</string> </key>
<value> <string>string</string> </value>
</item>
</dictionary>
</tuple>
</value>
</item>
<item>
<key> <string>_params</string> </key>
<value> <string>in_array = {}, out_array = []</string> </value>
</item>
<item>
<key> <string>categories</string> </key>
<value>
<tuple/>
</value>
</item>
<item>
<key> <string>description</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>DataAnalysisLine_resampleKPI</string> </value>
</item>
<item>
<key> <string>reference</string> </key>
<value> <string>DataAnalysisLine_resampleKPI</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string>DataAnalysisLine_resampleKPI</string> </value>
</item>
<item>
<key> <string>workflow_history</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="2" aka="AAAAAAAAAAI=">
<pickle>
<global name="PersistentMapping" module="Persistence.mapping"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>data</string> </key>
<value>
<dictionary>
<item>
<key> <string>edit_workflow</string> </key>
<value>
<persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record id="3" aka="AAAAAAAAAAM=">
<pickle>
<global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_log</string> </key>
<value>
<list>
<dictionary>
<item>
<key> <string>action</string> </key>
<value> <string>edit</string> </value>
</item>
<item>
<key> <string>actor</string> </key>
<value> <unicode>zope</unicode> </value>
</item>
<item>
<key> <string>comment</string> </key>
<value>
<none/>
</value>
</item>
<item>
<key> <string>error_message</string> </key>
<value> <string></string> </value>
</item>
<item>
<key> <string>serial</string> </key>
<value> <string>1007.25772.60736.16059</string> </value>
</item>
<item>
<key> <string>state</string> </key>
<value> <string>current</string> </value>
</item>
<item>
<key> <string>time</string> </key>
<value>
<object>
<klass>
<global name="_reconstructor" module="copy_reg"/>
</klass>
<tuple>
<global name="DateTime" module="DateTime.DateTime"/>
<global name="object" module="__builtin__"/>
<none/>
</tuple>
<state>
<tuple>
<float>1680187081.99</float>
<string>UTC</string>
</tuple>
</state>
</object>
</value>
</item>
</dictionary>
</list>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/ors_wendelin/PathTemplateItem/portal_callables/IngestionPolicy_parseORSTag.py
View file @ 369139eb
if not reference:
raise ValueError('reference is not defined')
return {'resource_reference' : context.data_product_module.ors_end_device.getReference(),
'specialise_reference': reference,
'reference': reference}
bt5/ors_wendelin/bt/template_keep_last_workflow_history_only_path_list
View file @ 369139eb
data_operation_module/ors_*
data_product_module/ors_*
data_transformation_module/ors_data_resample
data_transformation_module/ors_data_resample/**
data_transformation_module/ors_data_transformation
data_transformation_module/ors_data_transformation/**
organisation_module/ors_*
portal_callables/DataAnalysisLine_calculateKPI
portal_callables/DataAnalysisLine_resampleKPI
portal_callables/IngestionPolicy_parseORSTag
portal_ingestion_policies/ors_ingestion
web_page_module/ndarray_bundle.js
......
bt5/ors_wendelin/bt/template_path_list
View file @ 369139eb
data_operation_module/ors_*
data_product_module/ors_*
data_product_module/ors_resample/**
data_transformation_module/ors_data_resample
data_transformation_module/ors_data_resample/**
data_product_module/ors_kpi/**
data_transformation_module/ors_data_transformation
data_transformation_module/ors_data_transformation/**
organisation_module/ors_*
portal_callables/DataAnalysisLine_calculateKPI
portal_callables/DataAnalysisLine_resampleKPI
portal_callables/IngestionPolicy_parseORSTag
portal_ingestion_policies/ors_ingestion
web_page_module/ndarray_bundle.js
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7