<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/wait_for_activities"> Wait for activities </span></td>
</tr>
<trclass=""style="opacity: 1;">
<trclass=""style="opacity: 1;">
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/wait_for_activities"> Wait for activities </span></td>
</tr>
<trclass=""style="opacity: 1;">
<trclass=""style="opacity: 1;">
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/wait_for_activities"> Wait for activities </span></td>
</tr>
<trclass=""style="opacity: 1;">
<trclass=""style="opacity: 1;">
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/wait_for_activities"> Wait for activities </span></td>
</tr>
<trclass=""style="opacity: 1;">
<trclass=""style="opacity: 1;">
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
<p><strong>Script ID </strong>- <strong>DataAnalysisLine_convertEnvironmentDataStreamToArray </strong>: this script will do all the magic. It doesn't exist yet, we will create it later in this tutorial. </p>
</strong>: this script will do all the magic. It doesn't exist yet, we will create it later in this
tutorial. </p>
<p><strong><labelfor="field_my_aggregated_portal_type_list">Item Types </label></strong><labelfor="field_my_aggregated_portal_type_list">- <strong>Data Array </strong>: we choose Data Array as we will convert the raw data to Data Array</label></p>
<p>Create a new Data Product as described in <ahref="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data Product</a> tutorial with following values</p>
<p>Create a new Data Product as described in <ahref="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data
Product</a> tutorial with following values</p>
<p>Title - <strong>Environment Raw Array</strong></p>
<p>Title - <strong>Environment Raw Array</strong></p>
<p>After Data Product is created and validated, navigate to page called Portal Callables by clicking on <strong>Callables </strong> on the left side panel.</p>
<p>After Data Product is created and validated, navigate to page called Portal Callables by clicking
on <strong>Callables </strong> on the left side panel.</p>
<p>There we will create and store a python script that will transform raw data to arrays. </p>
<p>There we will create and store a python script that will transform raw data to arrays. </p>
<detailsopen="">The script we write in the textbox area at the bottom of the page.</details>
<detailsopen="">The script we write in the textbox area at the bottom of the page.</details>
</section>
</section>
<sectionclass="screenshot">
<sectionclass="screenshot">
<h1>Transformation Script Cont.</h1>
<h1>Transformation Script Cont.</h1>
<pre>
<pre>
<code>
<code>
import pandas as pd
import pandas as pd
...
@@ -796,20 +790,25 @@ if end < in_data_stream.getSize():
...
@@ -796,20 +790,25 @@ if end < in_data_stream.getSize():
</code>
</code>
</pre>
</pre>
<detailsopen="">
<detailsopen="">
<p>This script will simply write raw data to Data Array. More sophisticated example like resampling data or data predictions are shown in following tutorials. </p>
<p>This script will simply write raw data to Data Array. More sophisticated example like resampling data or
data predictions are shown in following tutorials. </p>
<p>First we extract data from the Data stream and read a chunk of data at a time. </p>
<p>First we extract data from the Data stream and read a chunk of data at a time. </p>
<p>As data is in <ahref="https://msgpack.org/">msgpack</a> format we unpack it using iterUnpack method of wendelin Data Stream which is based on Unpacker of msgpack. </p>
<p>As data is in <ahref="https://msgpack.org/">msgpack</a> format we unpack it using iterUnpack method of wendelin
Data Stream which is based on Unpacker of msgpack. </p>
<p>Next we create a data frame using <ahref="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">pandas DataFrame</a>. </p>
<p>Chose descriptive title and reference. For example</p>
<p>Chose descriptive title and reference. For example</p>
<p>Title : Convert Environment Raw Data</p>
<p>Title : Convert Environment Raw Data</p>
<p>Reference : convert-environment-raw-data</p>
<p>Reference : convert-environment-raw-data</p>
<p>For Initial Product we choose the Data Product we created in <ahref="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data Product</a> tutorial. </p>
<p>For Initial Product we choose the Data Product we created in <a
href="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data Product</a> tutorial. </p>
<p>At the end click Save to save the changes. </p>
<p>At the end click Save to save the changes. </p>
</details>
</details>
<test>
<test>
<tablestyle="display: none;"class="test">
<tablestyle="display: none;"class="test">
<tbody>
<tbody>
<tr>
<tr>
...
@@ -984,13 +984,13 @@ if end < in_data_stream.getSize():
...
@@ -984,13 +984,13 @@ if end < in_data_stream.getSize():
<td></td>
<td></td>
</tr>
</tr>
<tr>
<tr>
<td>waitForElementPresent</td>
<tdcolspan="3"><tal:block
<td>//input[@type="submit" and @value="Create Document"]</td>
<p>For Data Operation put the name of the Data operation that we created at the beginning of this tutorial - <strong>Convert Raw Environment Data to Array</strong>.</p>
<p>For Data Operation put the name of the Data operation that we created at the beginning of this tutorial -
<strong>Convert Raw Environment Data to Array</strong>.</p>
<p>At the end click on Save to save the changes. </p>
<p>At the end click on Save to save the changes. </p>
<p>For <strong>Data Product</strong> put the name of the Data Product that we created in <ahref="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data Product</a> tutorial.</p>
<p>For <strong>Data Product</strong> put the name of the Data Product that we created in <a
href="wendelin-Tutorial.Howto.Create.Data.Product">HowTo Create Data Product</a> tutorial.</p>
<p>After filling in the Data Product name click Save to save the intermediate changes. </p>
<p>After filling in the Data Product name click Save to save the intermediate changes. </p>
<p>After saving a new field <strong>Item Types </strong>will appear. </p>
<p>After saving a new field <strong>Item Types </strong>will appear. </p>
<p>For <strong>Data Product</strong> put the name of the Data Product that we created earlier in this tutorial - <strong>Environment Raw Array</strong>.</p>
<p>For <strong>Data Product</strong> put the name of the Data Product that we created earlier in this tutorial -
<strong>Environment Raw Array</strong>.</p>
<p>Once filling in the Data Product name click Save to save the intermediate changes. </p>
<p>Once filling in the Data Product name click Save to save the intermediate changes. </p>
<p>After saving a new field <strong>Item Types </strong>will appear. </p>
<p>After saving a new field <strong>Item Types </strong>will appear. </p>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<tdcolspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/init_wendelin_test_environment">Init Wendlin Test Environment</span></td>
<td colspan="3"><span metal:use-macro="container/Zuite_viewTestMacroLibrary/macros/wait_for_activities"> Wait for activities </span></td>
</tr>
<tr class="" style="opacity: 1;">
<td colspan="3"><span metal:use-macro="container/Zuite_viewWendelinTestMacroLibrary/macros/login_as_functional_test_user">Login As Functional Test User</span></td>
