Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
E
erp5_rtl_support
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Romain Courteaud
erp5_rtl_support
Commits
e9f4d77f
Commit
e9f4d77f
authored
Aug 19, 2014
by
Jérome Perrin
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Proof of concept to reuse testnode infrastructure to run DREAM simulation scenarios
parent
1c827f20
Changes
12
Hide whitespace changes
Inline
Side-by-side
Showing
12 changed files
with
730 additions
and
22 deletions
+730
-22
bt5/erp5_test_result/ActionTemplateItem/portal_types/DREAM%20Simulation%20Distributor/view.xml
...em/portal_types/DREAM%20Simulation%20Distributor/view.xml
+85
-0
bt5/erp5_test_result/DocumentTemplateItem/portal_components/document.erp5.DREAMSimulationDistributor.py
...al_components/document.erp5.DREAMSimulationDistributor.py
+241
-0
bt5/erp5_test_result/DocumentTemplateItem/portal_components/document.erp5.DREAMSimulationDistributor.xml
...l_components/document.erp5.DREAMSimulationDistributor.xml
+123
-0
bt5/erp5_test_result/PortalTypeAllowedContentTypeTemplateItem/allowed_content_types.xml
...eAllowedContentTypeTemplateItem/allowed_content_types.xml
+1
-0
bt5/erp5_test_result/PortalTypeTemplateItem/portal_types/DREAM%20Simulation%20Distributor.xml
...ateItem/portal_types/DREAM%20Simulation%20Distributor.xml
+64
-0
bt5/erp5_test_result/bt/template_action_path_list
bt5/erp5_test_result/bt/template_action_path_list
+1
-0
bt5/erp5_test_result/bt/template_document_id_list
bt5/erp5_test_result/bt/template_document_id_list
+1
-0
bt5/erp5_test_result/bt/template_portal_type_allowed_content_type_list
..._result/bt/template_portal_type_allowed_content_type_list
+1
-0
bt5/erp5_test_result/bt/template_portal_type_id_list
bt5/erp5_test_result/bt/template_portal_type_id_list
+1
-0
erp5/util/testnode/DREAMSimulationRunner.py
erp5/util/testnode/DREAMSimulationRunner.py
+169
-0
erp5/util/testnode/SlapOSControler.py
erp5/util/testnode/SlapOSControler.py
+23
-19
erp5/util/testnode/testnode.py
erp5/util/testnode/testnode.py
+20
-3
No files found.
bt5/erp5_test_result/ActionTemplateItem/portal_types/DREAM%20Simulation%20Distributor/view.xml
0 → 100644
View file @
e9f4d77f
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"ActionInformation"
module=
"Products.CMFCore.ActionInformation"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
action
</string>
</key>
<value>
<persistent>
<string
encoding=
"base64"
>
AAAAAAAAAAI=
</string>
</persistent>
</value>
</item>
<item>
<key>
<string>
categories
</string>
</key>
<value>
<tuple>
<string>
action_type/object_view
</string>
</tuple>
</value>
</item>
<item>
<key>
<string>
category
</string>
</key>
<value>
<string>
object_view
</string>
</value>
</item>
<item>
<key>
<string>
condition
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
icon
</string>
</key>
<value>
<string></string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
view
</string>
</value>
</item>
<item>
<key>
<string>
permissions
</string>
</key>
<value>
<tuple>
<string>
View
</string>
</tuple>
</value>
</item>
<item>
<key>
<string>
portal_type
</string>
</key>
<value>
<string>
Action Information
</string>
</value>
</item>
<item>
<key>
<string>
priority
</string>
</key>
<value>
<float>
1.0
</float>
</value>
</item>
<item>
<key>
<string>
title
</string>
</key>
<value>
<string>
View
</string>
</value>
</item>
<item>
<key>
<string>
visible
</string>
</key>
<value>
<int>
1
</int>
</value>
</item>
</dictionary>
</pickle>
</record>
<record
id=
"2"
aka=
"AAAAAAAAAAI="
>
<pickle>
<global
name=
"Expression"
module=
"Products.CMFCore.Expression"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
text
</string>
</key>
<value>
<string>
string:${object_url}/ERP5ProjectUnitTestDistributor_view
</string>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/DocumentTemplateItem/portal_components/document.erp5.DREAMSimulationDistributor.py
0 → 100644
View file @
e9f4d77f
##############################################################################
# Copyright (c) 2014 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
#
##############################################################################
from
Products.ERP5.Document.ERP5ProjectUnitTestDistributor
import
ERP5ProjectUnitTestDistributor
from
AccessControl
import
ClassSecurityInfo
from
Products.ERP5Type
import
Permissions
import
json
class
DREAMSimulationDistributor
(
ERP5ProjectUnitTestDistributor
):
security
=
ClassSecurityInfo
()
security
.
declareObjectProtected
(
Permissions
.
AccessContentsInformation
)
security
.
declarePublic
(
"getTestType"
)
def
getTestType
(
self
,
batch_mode
=
0
):
"""getTestType : return a string defining the type of tests
"""
return
'DREAMSimulation'
security
.
declarePublic
(
"requestSimulationRun"
)
def
requestSimulationRun
(
self
,
scenario_list
):
"""Request to run a list of scenarios, to be called by ManPy during ACO
This will create a planned simulation run document with lines and returns a job_id to the caller
The testnodes will start the simulation run and start the lines one by one, and stop when posting back the result.
The last testnode will mark the simulation run as completed.
The caller will getJobResult with this job_id until the simulation run is finished.
"""
tr
=
self
.
getPortalObject
().
test_result_module
.
newContent
(
portal_type
=
'Test Result'
,
title
=
'DREAM Simulation Run'
,
)
for
i
,
scenario
in
enumerate
(
scenario_list
):
tr
.
newContent
(
portal_type
=
'Test Result Line'
,
dream_scenario
=
scenario
,
int_index
=
i
)
tr
.
start
()
return
tr
.
getId
()
security
.
declarePublic
(
'getJobResult'
)
def
getJobResult
(
self
,
job_id
):
" "
test_result
=
self
.
getPortalObject
().
test_result_module
.
_getOb
(
job_id
)
if
test_result
.
getSimulationState
()
!=
'stopped'
:
return
return
[
test_result_line
.
getProperty
(
'dream_output'
)
for
test_result_line
in
test_result
.
contentValues
()]
security
.
declarePublic
(
'createTestResult'
)
def
createTestResult
(
self
,
name
,
revision
,
test_name_list
,
allow_restart
,
test_title
=
None
,
node_title
=
None
,
project_title
=
None
):
"""Overriden not to lookup project and not to create test result because it's created by requestSimulationRun
"""
self
.
log
(
'DREAMSimulationDistributor.createTestResult'
,
0
,
(
name
,
revision
,
test_title
,
project_title
))
portal
=
self
.
getPortalObject
()
if
test_title
is
None
:
test_title
=
name
def
createNode
(
test_result
,
node_title
):
if
node_title
is
not
None
:
node
=
self
.
_getTestResultNode
(
test_result
,
node_title
)
if
node
is
None
:
node
=
test_result
.
newContent
(
portal_type
=
'Test Result Node'
,
title
=
node_title
)
node
.
start
()
def
createTestResultLineList
(
test_result
,
test_name_list
):
duration_list
=
[]
previous_test_result_list
=
portal
.
test_result_module
.
searchFolder
(
title
=
'="%s"'
%
test_result
.
getTitle
(),
sort_on
=
[(
'creation_date'
,
'descending'
)],
simulation_state
=
(
'stopped'
,
'public_stopped'
),
limit
=
1
)
if
len
(
previous_test_result_list
):
previous_test_result
=
previous_test_result_list
[
0
].
getObject
()
for
line
in
previous_test_result
.
objectValues
():
if
line
.
getSimulationState
()
in
(
'stopped'
,
'public_stopped'
):
duration_list
.
append
((
line
.
getTitle
(),
line
.
getProperty
(
'duration'
)))
duration_list
.
sort
(
key
=
lambda
x
:
-
x
[
1
])
sorted_test_list
=
[
x
[
0
]
for
x
in
duration_list
]
# Sort tests by name to have consistent numbering of test result line on
# a test suite.
for
test_name
in
sorted
(
test_name_list
):
index
=
0
if
sorted_test_list
:
try
:
index
=
sorted_test_list
.
index
(
test_name
)
except
ValueError
:
pass
line
=
test_result
.
newContent
(
portal_type
=
'Test Result Line'
,
title
=
test_name
,
int_index
=
index
)
reference_list_string
=
None
if
type
(
revision
)
is
str
and
'='
in
revision
:
reference_list_string
=
revision
int_index
,
reference
=
None
,
revision
elif
type
(
revision
)
is
str
:
# backward compatibility
int_index
,
reference
=
revision
,
None
else
:
# backward compatibility
int_index
,
reference
=
revision
result_list
=
portal
.
test_result_module
.
searchFolder
(
portal_type
=
"Test Result"
,
title
=
'="%s"'
%
test_title
,
sort_on
=
((
"creation_date"
,
"descending"
),),
limit
=
1
)
if
result_list
:
test_result
=
result_list
[
0
].
getObject
()
if
test_result
is
not
None
:
last_state
=
test_result
.
getSimulationState
()
last_revision
=
str
(
test_result
.
getIntIndex
())
# XXX we don't need that
if
last_state
==
'started'
:
createNode
(
test_result
,
node_title
)
reference
=
test_result
.
getReference
()
if
reference_list_string
:
last_revision
=
reference
elif
reference
:
last_revision
=
last_revision
,
reference
if
len
(
test_result
.
objectValues
(
portal_type
=
"Test Result Line"
))
==
0
\
and
len
(
test_name_list
):
test_result
.
serialize
()
# prevent duplicate test result lines
createTestResultLineList
(
test_result
,
test_name_list
)
return
test_result
.
getRelativeUrl
(),
last_revision
if
last_state
in
(
'stopped'
,
'public_stopped'
):
if
reference_list_string
is
not
None
:
if
reference_list_string
==
test_result
.
getReference
()
\
and
not
allow_restart
:
return
elif
last_revision
==
int_index
and
not
allow_restart
:
return
assert
0
,
"Test result should already be created"
test_result
=
portal
.
test_result_module
.
newContent
(
portal_type
=
'Test Result'
,
title
=
test_title
,
reference
=
reference
,
is_indexable
=
False
)
if
int_index
is
not
None
:
test_result
.
_setIntIndex
(
int_index
)
if
project_title
is
not
None
:
project_list
=
portal
.
portal_catalog
(
portal_type
=
'Project'
,
title
=
'="%s"'
%
project_title
)
if
len
(
project_list
)
!=
1
:
raise
ValueError
(
'found this list of project : %r for title %r'
%
\
([
x
.
path
for
x
in
project_list
],
project_title
))
test_result
.
_setSourceProjectValue
(
project_list
[
0
].
getObject
())
test_result
.
updateLocalRolesOnSecurityGroups
()
# XXX
test_result
.
start
()
del
test_result
.
isIndexable
test_result
.
immediateReindexObject
()
self
.
serialize
()
# prevent duplicate test result
# following 2 functions only call 'newContent' on test_result
createTestResultLineList
(
test_result
,
test_name_list
)
createNode
(
test_result
,
node_title
)
return
test_result
.
getRelativeUrl
(),
revision
security
.
declarePublic
(
"startTestSuite"
)
def
saveDREAMSimulationResult
(
self
,
test_result_line
,
output
):
'''Save DREAM simulation result'''
test_result_line
=
self
.
getPortalObject
().
unrestrictedTraverse
(
test_result_line
)
test_result_line
.
setProperty
(
'dream_output'
,
output
)
test_result_line
.
stop
()
test_result
=
test_result_line
.
getParentValue
()
for
test_result_line
in
test_result
.
contentValues
(
portal_type
=
'Test Result Line'
):
if
test_result_line
.
getSimulationState
()
!=
'stopped'
:
return
# everything finished.
test_result
.
stop
()
security
.
declarePublic
(
"startTestSuite"
)
def
startTestSuite
(
self
,
title
,
computer_guid
=
'unknown'
,
batch_mode
=
0
,
**
kw
):
"""startTestSuite : subscribe node + return testsuite list to the master. XXX what is a master ?
"""
ERP5ProjectUnitTestDistributor
.
subscribeNode
(
self
,
title
=
title
,
computer_guid
=
computer_guid
,
batch_mode
=
batch_mode
)
dream_repo
=
{
'branch'
:
'dream'
,
'buildout_section_id'
:
'slapos'
,
'url'
:
'http://git.erp5.org/repos/slapos.git'
,
'profile_path'
:
'software/dream/software.cfg'
}
result_list
=
self
.
getPortalObject
().
portal_catalog
.
unrestrictedSearchResults
(
portal_type
=
"Test Result"
,
title
=
'DREAM Simulation Run'
,
# XXX use exact match
sort_on
=
((
"creation_date"
,
"descending"
),),
simulation_state
=
'started'
,
limit
=
1
)
if
result_list
:
test_result
=
result_list
[
0
].
getObject
()
for
scenario
in
test_result
.
contentValues
(
portal_type
=
'Test Result Line'
):
if
scenario
.
getSimulationState
()
==
'draft'
:
scenario
.
start
(
comment
=
"OK"
)
return
json
.
dumps
([{
'project_title'
:
'DREAM Simulation Distribution Project'
,
'test_suite_reference'
:
'dream_runner'
,
'test_suite_title'
:
'DREAM Simulation Run'
,
'test_result_line_id'
:
scenario
.
getId
(),
'scenario'
:
scenario
.
getProperty
(
'dream_scenario'
),
'vcs_repository_list'
:
[
dream_repo
]
}])
# always return the same test_suite_reference otherwise software is removed.
return
json
.
dumps
([{
'project_title'
:
'DREAM Simulation Distribution Project'
,
'test_suite_reference'
:
'dream_runner'
,
'test_suite_title'
:
'DREAM Simulation Run'
,
'scenario'
:
None
,
'vcs_repository_list'
:
[
dream_repo
]
}])
bt5/erp5_test_result/DocumentTemplateItem/portal_components/document.erp5.DREAMSimulationDistributor.xml
0 → 100644
View file @
e9f4d77f
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Document Component"
module=
"erp5.portal_type"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
_recorded_property_dict
</string>
</key>
<value>
<persistent>
<string
encoding=
"base64"
>
AAAAAAAAAAI=
</string>
</persistent>
</value>
</item>
<item>
<key>
<string>
default_reference
</string>
</key>
<value>
<string>
DREAMSimulationDistributor
</string>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
document.erp5.DREAMSimulationDistributor
</string>
</value>
</item>
<item>
<key>
<string>
portal_type
</string>
</key>
<value>
<string>
Document Component
</string>
</value>
</item>
<item>
<key>
<string>
sid
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
text_content_error_message
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
text_content_warning_message
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
version
</string>
</key>
<value>
<string>
erp5
</string>
</value>
</item>
<item>
<key>
<string>
workflow_history
</string>
</key>
<value>
<persistent>
<string
encoding=
"base64"
>
AAAAAAAAAAM=
</string>
</persistent>
</value>
</item>
</dictionary>
</pickle>
</record>
<record
id=
"2"
aka=
"AAAAAAAAAAI="
>
<pickle>
<global
name=
"PersistentMapping"
module=
"Persistence.mapping"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
data
</string>
</key>
<value>
<dictionary/>
</value>
</item>
</dictionary>
</pickle>
</record>
<record
id=
"3"
aka=
"AAAAAAAAAAM="
>
<pickle>
<global
name=
"PersistentMapping"
module=
"Persistence.mapping"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
data
</string>
</key>
<value>
<dictionary>
<item>
<key>
<string>
component_validation_workflow
</string>
</key>
<value>
<persistent>
<string
encoding=
"base64"
>
AAAAAAAAAAQ=
</string>
</persistent>
</value>
</item>
</dictionary>
</value>
</item>
</dictionary>
</pickle>
</record>
<record
id=
"4"
aka=
"AAAAAAAAAAQ="
>
<pickle>
<global
name=
"WorkflowHistoryList"
module=
"Products.ERP5Type.patches.WorkflowTool"
/>
</pickle>
<pickle>
<tuple>
<none/>
<list>
<dictionary>
<item>
<key>
<string>
action
</string>
</key>
<value>
<string>
validate
</string>
</value>
</item>
<item>
<key>
<string>
validation_state
</string>
</key>
<value>
<string>
validated
</string>
</value>
</item>
</dictionary>
</list>
</tuple>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/PortalTypeAllowedContentTypeTemplateItem/allowed_content_types.xml
View file @
e9f4d77f
...
...
@@ -7,6 +7,7 @@
</portal_type>
<portal_type
id=
"Task Distribution Tool"
>
<item>
Cloud Performance Unit Test Distributor
</item>
<item>
DREAM Simulation Distributor
</item>
<item>
ERP5 Project Unit Test Distributor
</item>
<item>
ERP5 Scalability Distributor
</item>
</portal_type>
...
...
bt5/erp5_test_result/PortalTypeTemplateItem/portal_types/DREAM%20Simulation%20Distributor.xml
0 → 100644
View file @
e9f4d77f
<?xml version="1.0"?>
<ZopeData>
<record
id=
"1"
aka=
"AAAAAAAAAAE="
>
<pickle>
<global
name=
"Base Type"
module=
"erp5.portal_type"
/>
</pickle>
<pickle>
<dictionary>
<item>
<key>
<string>
content_icon
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
description
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
factory
</string>
</key>
<value>
<string>
addXMLObject
</string>
</value>
</item>
<item>
<key>
<string>
id
</string>
</key>
<value>
<string>
DREAM Simulation Distributor
</string>
</value>
</item>
<item>
<key>
<string>
init_script
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
permission
</string>
</key>
<value>
<none/>
</value>
</item>
<item>
<key>
<string>
portal_type
</string>
</key>
<value>
<string>
Base Type
</string>
</value>
</item>
<item>
<key>
<string>
type_class
</string>
</key>
<value>
<string>
DREAMSimulationDistributor
</string>
</value>
</item>
<item>
<key>
<string>
type_interface
</string>
</key>
<value>
<tuple/>
</value>
</item>
<item>
<key>
<string>
type_mixin
</string>
</key>
<value>
<tuple/>
</value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_test_result/bt/template_action_path_list
View file @
e9f4d77f
Benchmark Result Line | view
Benchmark Result | view
Cloud Performance Unit Test Distributor | view
DREAM Simulation Distributor | view
ERP5 Project Unit Test Distributor | view
ERP5 Scalability Distributor | view
Scalability Test Suite | vcs_repository
...
...
bt5/erp5_test_result/bt/template_document_id_list
View file @
e9f4d77f
document.erp5.TestNode
document.erp5.TestSuite
document.erp5.ERP5ScalabilityDistributor
document.erp5.DREAMSimulationDistributor
\ No newline at end of file
bt5/erp5_test_result/bt/template_portal_type_allowed_content_type_list
View file @
e9f4d77f
Benchmark Result | Benchmark Result Line
Scalability Test Suite | Test Suite Repository
Task Distribution Tool | Cloud Performance Unit Test Distributor
Task Distribution Tool | DREAM Simulation Distributor
Task Distribution Tool | ERP5 Project Unit Test Distributor
Task Distribution Tool | ERP5 Scalability Distributor
Test Node Module | Test Node
...
...
bt5/erp5_test_result/bt/template_portal_type_id_list
View file @
e9f4d77f
Benchmark Result
Benchmark Result Line
Cloud Performance Unit Test Distributor
DREAM Simulation Distributor
ERP5 Project Unit Test Distributor
ERP5 Scalability Distributor
Scalability Test Suite
...
...
erp5/util/testnode/DREAMSimulationRunner.py
0 → 100644
View file @
e9f4d77f
##############################################################################
#
# Copyright (c) 2014 Nexedi SA and Contributors. All Rights Reserved.
#
# WARNING: This program as such is intended to be used by professional
# programmers who take the whole responsibility of assessing all potential
# consequences resulting from its eventual inadequacies and bugs
# End users who are looking for a ready-to-use solution with commercial
# guarantees and support are strongly advised to contract a Free Software
# Service Company
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 3
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
##############################################################################
from
datetime
import
datetime
,
timedelta
import
os
import
subprocess
import
sys
import
time
import
tempfile
import
glob
import
SlapOSControler
import
json
import
time
import
shutil
import
logging
import
string
import
random
from
ProcessManager
import
SubprocessError
,
ProcessManager
,
CancellationError
from
subprocess
import
CalledProcessError
from
NodeTestSuite
import
SlapOSInstance
from
Updater
import
Updater
from
Utils
import
dealShebang
from
erp5.util
import
taskdistribution
class
DREAMSimulationRunner
():
def
__init__
(
self
,
testnode
):
self
.
testnode
=
testnode
def
_getSlapOSControler
(
self
,
working_directory
):
"""
Create a SlapOSControler for this working dir
"""
return
SlapOSControler
.
SlapOSControler
(
working_directory
,
self
.
testnode
.
config
,
self
.
testnode
.
log
)
def
_prepareSlapOS
(
self
,
working_directory
,
slapos_instance
,
log
,
build_software
=
1
,
software_path_list
=
None
,
**
kw
):
"""Launch slapos to build software and partitions
XXX: only build & create partition once for DREAM
"""
slapproxy_log
=
os
.
path
.
join
(
self
.
testnode
.
config
[
'log_directory'
],
'slapproxy.log'
)
log
(
'Configured slapproxy log to %r'
%
slapproxy_log
)
reset_software
=
slapos_instance
.
retry_software_count
>
10
if
reset_software
:
slapos_instance
.
retry_software_count
=
0
reset_software
=
False
# Never delete ...
log
(
'testnode, retry_software_count : %r'
%
\
slapos_instance
.
retry_software_count
)
# XXX Create a new controler because working_directory can be
# Different depending of the preparation
slapos_controler
=
self
.
_getSlapOSControler
(
working_directory
)
slapos_controler
.
initializeSlapOSControler
(
slapproxy_log
=
slapproxy_log
,
process_manager
=
self
.
testnode
.
process_manager
,
reset_software
=
reset_software
,
software_path_list
=
software_path_list
)
self
.
testnode
.
process_manager
.
supervisord_pid_file
=
os
.
path
.
join
(
\
slapos_controler
.
instance_root
,
'var'
,
'run'
,
'supervisord.pid'
)
# XXX If all software looks already build, we will not run soft again
soft_list
=
glob
.
glob
(
os
.
path
.
join
(
slapos_instance
.
working_directory
,
'soft'
,
'*'
))
completed_soft_list
=
glob
.
glob
(
os
.
path
.
join
(
slapos_instance
.
working_directory
,
'soft'
,
'*'
,
'.completed'
))
if
soft_list
and
len
(
soft_list
)
==
len
(
completed_soft_list
):
log
(
'testnode: all software seem built, will not build again'
)
build_software
=
False
if
build_software
:
status_dict
=
slapos_controler
.
runSoftwareRelease
(
self
.
testnode
.
config
,
environment
=
self
.
testnode
.
config
[
'environment'
])
if
status_dict
[
'status_code'
]
!=
0
:
slapos_instance
.
retry
=
True
slapos_instance
.
retry_software_count
+=
1
raise
SubprocessError
(
status_dict
)
slapos_instance
.
retry_software_count
=
0
status_dict
=
slapos_controler
.
runComputerPartition
(
self
.
testnode
.
config
,
environment
=
self
.
testnode
.
config
[
'environment'
],
implicit_erp5_config
=
False
)
if
status_dict
[
'status_code'
]
!=
0
:
slapos_instance
.
retry
=
True
slapos_instance
.
retry_software_count
+=
1
raise
SubprocessError
(
status_dict
)
slapos_instance
.
retry_software_count
=
0
return
status_dict
def
prepareSlapOSForTestNode
(
self
,
test_node_slapos
):
"""
We will build slapos software needed by the testnode itself,
like the building of selenium-runner by default
"""
# I don't think we need that
return
{}
return
self
.
_prepareSlapOS
(
self
.
testnode
.
config
[
'slapos_directory'
],
test_node_slapos
,
self
.
testnode
.
log
,
create_partition
=
0
,
software_path_list
=
self
.
testnode
.
config
.
get
(
"software_list"
))
def
prepareSlapOSForTestSuite
(
self
,
node_test_suite
):
"""
Build softwares needed by testsuites
"""
log
=
self
.
testnode
.
log
if
log
is
None
:
log
=
self
.
testnode
.
log
return
self
.
_prepareSlapOS
(
node_test_suite
.
working_directory
,
node_test_suite
,
log
,
software_path_list
=
[
node_test_suite
.
custom_profile_path
])
def
runSimulationScenario
(
self
,
node_test_suite
,
portal_url
,
test_result
):
if
not
node_test_suite
.
scenario
:
return
# nothing to do
dream_simulation
=
glob
.
glob
(
"%s/inst/*/etc/run/dream_simulation"
%
\
node_test_suite
.
working_directory
)[
0
]
with
tempfile
.
NamedTemporaryFile
()
as
tf
:
tf
.
write
(
json
.
dumps
(
json
.
loads
(
node_test_suite
.
scenario
)[
'input'
]))
tf
.
flush
()
invocation_list
=
[
dream_simulation
,
tf
.
name
]
status_dict
=
self
.
testnode
.
process_manager
.
spawn
(
*
invocation_list
,
cwd
=
node_test_suite
.
working_directory
,
log_prefix
=
'dream_simulation'
,
get_output
=
True
)
output
=
status_dict
[
'stdout'
]
assert
output
self
.
testnode
.
log
(
'Posting back result for %s/%s'
%
(
test_result
.
test_result_path
,
node_test_suite
.
test_result_line_id
))
test_result
.
_retryRPC
(
'saveDREAMSimulationResult'
,
(
'%s/%s'
%
(
test_result
.
test_result_path
,
node_test_suite
.
test_result_line_id
),
output
))
def
getRelativePathUsage
(
self
):
"""
Used by the method testnode.constructProfile() to know
if the software.cfg have to use relative path or not.
"""
return
False
erp5/util/testnode/SlapOSControler.py
View file @
e9f4d77f
...
...
@@ -350,7 +350,7 @@ class SlapOSControler(object):
# so be tolerant and run it a few times before giving up
for
runs
in
range
(
0
,
MAX_SR_RETRIES
):
status_dict
=
self
.
spawn
(
config
[
'slapos_binary'
],
'node'
,
'software'
,
'--all'
,
'node'
,
'software'
,
'--all'
,
'--pidfile'
,
'%s/software.pid'
%
self
.
software_root
,
'--cfg'
,
self
.
slapos_config
,
raise_error_if_fail
=
False
,
log_prefix
=
'slapgrid_sr'
,
get_output
=
False
)
...
...
@@ -359,25 +359,29 @@ class SlapOSControler(object):
return
status_dict
def
runComputerPartition
(
self
,
config
,
environment
,
stdout
=
None
,
stderr
=
None
):
stdout
=
None
,
stderr
=
None
,
implicit_erp5_config
=
True
):
self
.
log
(
"SlapOSControler.runComputerPartition"
)
# cloudooo-json is required but this is a hack which should be removed
config
[
'instance_dict'
][
'cloudooo-json'
]
=
"{}"
# report-url, report-project and suite-url are required to seleniumrunner
# instance. This is a hack which must be removed.
config
[
'instance_dict'
][
'report-url'
]
=
config
.
get
(
"report-url"
,
""
)
config
[
'instance_dict'
][
'report-project'
]
=
config
.
get
(
"report-project"
,
""
)
config
[
'instance_dict'
][
'suite-url'
]
=
config
.
get
(
"suite-url"
,
""
)
# XXX: Hack to minimize writes to storage holding MySQL databases.
# Note this is something we want for all test suites, so it would
# not be better to define this parameter on each test suite.
# XXX: Also move here the number of test db to create, so that software
# release stop create ones by default.
config
[
'instance_dict'
][
'_'
]
=
json
.
dumps
({
"mariadb"
:
{
"relaxed-writes"
:
True
,
"mariadb-relaxed-writes"
:
True
,
# BBB
"test-database-amount"
:
30
,
}})
# XXX implicit_erp5_config should not be here but passed by classes
# depending on it
if
implicit_erp5_config
:
# cloudooo-json is required but this is a hack which should be removed
config
[
'instance_dict'
][
'cloudooo-json'
]
=
"{}"
# report-url, report-project and suite-url are required to seleniumrunner
# instance. This is a hack which must be removed.
config
[
'instance_dict'
][
'report-url'
]
=
config
.
get
(
"report-url"
,
""
)
config
[
'instance_dict'
][
'report-project'
]
=
config
.
get
(
"report-project"
,
""
)
config
[
'instance_dict'
][
'suite-url'
]
=
config
.
get
(
"suite-url"
,
""
)
# XXX: Hack to minimize writes to storage holding MySQL databases.
# Note this is something we want for all test suites, so it would
# not be better to define this parameter on each test suite.
# XXX: Also move here the number of test db to create, so that software
# release stop create ones by default.
config
[
'instance_dict'
][
'_'
]
=
json
.
dumps
({
"mariadb"
:
{
"relaxed-writes"
:
True
,
"mariadb-relaxed-writes"
:
True
,
# BBB
"test-database-amount"
:
30
,
}})
for
path
in
self
.
software_path_list
:
try
:
self
.
slap
.
registerOpenOrder
().
request
(
path
,
...
...
erp5/util/testnode/testnode.py
View file @
e9f4d77f
...
...
@@ -46,6 +46,7 @@ from subprocess import CalledProcessError
from
Updater
import
Updater
from
NodeTestSuite
import
NodeTestSuite
,
SlapOSInstance
from
ScalabilityTestRunner
import
ScalabilityTestRunner
from
DREAMSimulationRunner
import
DREAMSimulationRunner
from
UnitTestRunner
import
UnitTestRunner
from
erp5.util
import
taskdistribution
...
...
@@ -137,7 +138,6 @@ class TestNode(object):
node_test_suite
.
reference
)
software_config_path
=
os
.
path
.
relpath
(
software_config_path
,
from_path
)
profile_content_list
.
append
(
"""
[buildout]
extends = %(software_config_path)s
...
...
@@ -188,6 +188,10 @@ branch = %(branch)s
sys
.
path
.
append
(
repository_path
)
def
getAndUpdateFullRevisionList
(
self
,
node_test_suite
):
if
0
:
# already checkout
self
.
log
(
"no getAndUpdateFullRevisionList ..."
)
node_test_suite
.
revision
=
0
,
0
return
[]
full_revision_list
=
[]
config
=
self
.
config
log
=
self
.
log
...
...
@@ -224,6 +228,9 @@ branch = %(branch)s
return
self
.
suite_log
def
_initializeSuiteLog
(
self
,
suite_log_path
):
logger
=
logging
.
getLogger
(
'testsuite'
)
self
.
suite_log
=
logger
.
info
return
# remove previous handlers
logger
=
logging
.
getLogger
(
'testsuite'
)
if
self
.
file_handler
is
not
None
:
...
...
@@ -239,6 +246,9 @@ branch = %(branch)s
self
.
suite_log
=
logger
.
info
def
checkRevision
(
self
,
test_result
,
node_test_suite
):
self
.
log
(
"skipping checkRevision"
)
return
# no check revision
config
=
self
.
config
log
=
self
.
log
if
log
is
None
:
...
...
@@ -349,6 +359,8 @@ from the distributor.")
runner
=
UnitTestRunner
(
self
)
elif
my_test_type
==
'ScalabilityTest'
:
runner
=
ScalabilityTestRunner
(
self
)
elif
my_test_type
==
'DREAMSimulation'
:
runner
=
DREAMSimulationRunner
(
self
)
else
:
log
(
"testnode, Runner type %s not implemented."
,
my_test_type
)
raise
NotImplementedError
...
...
@@ -371,6 +383,8 @@ from the distributor.")
runner
=
UnitTestRunner
(
node_test_suite
)
elif
my_test_type
==
'ScalabilityTest'
:
runner
=
ScalabilityTestRunner
(
node_test_suite
)
elif
my_test_type
==
'DREAMSimulation'
:
runner
=
DREAMSimulationRunner
(
node_test_suite
)
else
:
log
(
"testnode, Runner type %s not implemented."
,
my_test_type
)
raise
NotImplementedError
...
...
@@ -401,7 +415,7 @@ from the distributor.")
# Give some time so computer partitions may start
# as partitions can be of any kind we have and likely will never have
# a reliable way to check if they are up or not ...
time
.
sleep
(
20
)
time
.
sleep
(
1
)
if
my_test_type
==
'UnitTest'
:
runner
.
runTestSuite
(
node_test_suite
,
portal_url
)
elif
my_test_type
==
'ScalabilityTest'
:
...
...
@@ -422,9 +436,11 @@ from the distributor.")
)
self
.
log
(
error_message
)
raise
ValueError
(
error_message
)
elif
my_test_type
==
'DREAMSimulation'
:
runner
.
runSimulationScenario
(
node_test_suite
,
portal_url
,
test_result
)
else
:
raise
NotImplementedError
# break the loop to get latest priorities from master
break
self
.
cleanUp
(
test_result
)
...
...
@@ -457,6 +473,7 @@ from the distributor.")
self
.
cleanUp
(
test_result
)
if
(
now
-
begin
)
<
120
:
sleep_time
=
120
-
(
now
-
begin
)
sleep_time
=
.
1
log
(
"End of processing, going to sleep %s"
%
sleep_time
)
time
.
sleep
(
sleep_time
)
except
:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment