Extend tests to detect shared libraries using system libraries
On some test machines, we saw test failures because openssl version mismatch:
Traceback (most recent call last):
File "/srv/slapgrid/slappart14/srv/testnode/cqg/soft/18d28ecd49e5e0f732e2ce0c00c57519/parts/slapos.core-repository/slapos/testing/testcase.py", line 227, in installSoftwareUrlList
checkSoftware(cls.slap, software_url)
File "/srv/slapgrid/slappart14/srv/testnode/cqg/soft/18d28ecd49e5e0f732e2ce0c00c57519/parts/slapos.core-repository/slapos/testing/testcase.py", line 211, in checkSoftware
raise RuntimeError('\n'.join(error_list))
RuntimeError: ./parts/proftpd/libexec/mod_auth_web.so:
./parts/proftpd/libexec/mod_auth_web.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /srv/slapgrid/slappart14/srv/testnode/cqg/inst/test0-0/tmp/shared/curl/724c785f86fff6993ff24ef745d4dbb9/lib/libcurl.so.4)What we see in this traceback is that the version of openssl used was the one from system ( in /usr/lib/ ), which should not happen because we are using setting rpath to slapos libraries. We had a test checking for missing rpath that could not resolved, but our test did not catch missing rpath that can be resolved on a system library, which is also wrong because this system library might be a different version like here or even be missing.
-
-
Owner
I have also started to fix profiles in a branch somewhere, libzip and zstd had missing rpath to zlib.
-
added 7 commits
-
3146023f...5c733f20 - 5 commits from branch
feat/better_testcase_snapshots - c8333e26 - testing/testcase: cleanups
- a3418d80 - testing/testcase: check executables are not linked with system libraries
Toggle commit list -
3146023f...5c733f20 - 5 commits from branch
-
changed target branch from
feat/better_testcase_snapshotstomasterToggle commit list -
added 41 commits
-
a3418d80...fea3bd23 - 39 commits from branch
master - 73d2fe07 - testing/testcase: cleanups
- 5edb38b1 - testing/testcase: check executables are not linked with system libraries
Toggle commit list -
a3418d80...fea3bd23 - 39 commits from branch
-
-
Owner
I have also started to fix profiles in a branch somewhere, libzip and zstd had missing rpath to zlib.
-
178 'libpthread', 179 'libresolv', 180 'librt', 181 'libstdc++', 182 'libutil', 183 )) 184 185 # we also ignore a few patterns for part that are known to be binary distributions, 186 # for which we generate LD_LIBRARY_PATH wrappers or we don't use directly. 187 ignored_file_patterns = set(( 188 '*/parts/java-re*/*', 189 '*/parts/firefox-*/*', 190 '*/parts/chromium-*/*', 191 '*/parts/chromedriver*/*', 192 '*/node_modules/phantomjs-prebuilt/*', 193 '*/parts/renderjs-repository.git/node_modules/*', -
Owner
In other words, I'm wondering if we need
'*/node_modules/phantomjs-prebuilt/*'if we have'*/node_modules/phantomjs-prebuilt/*' -
Owner
Looking at my test result, you can see that the failing binary location was:
./parts/renderjs-repository.git/node_modules/phantomjs/lib/phantom/bin/phantomjs.So, it seems
*/node_modules/phantomjs-prebuilt/*would not match it.FYI, I'm currently trying to run a first test for some software releases I'm using in order to automatically cache all components.
If you think using
/parts/renderjs-repository.git/node_modules/*was not the best choice, I'll update it to improve. If you prefer, I can submit a merge request next time. -
Owner
Thanks for feedback, I was not saying it's bad :) just wondering if we were trying to whitelist the same thing and in that case we could have only one entry in this list. I'm thinking we can whitelist
*/node_modules/phantomjs*/it would cover both.I see that in your test result
firefoxis also reported as missing rpath, it seems it was a mistake to use/parts/firefox-.*/here, we should use something likeparts/firefox*/*instead ( we also changed from regular expression to fnmatch )I'll make these changes in this MR.
FYI, I'm currently trying to run a first test for some software releases I'm using in order to automatically cache all components.
Not sure what is "cache" in this context, but for reference: the software release tests do not keep softwares between runs, the
shared/folder is also removed. "run" is a test result like this one. For a test result like this one shared folder is used for all tests ( htmlvalidatorserver , jstestnode, ... ) and removed after. After erp5!940 (merged) , this will change a bit, tests will also useshared/from testnode, which is kept. -
Owner
For firefox, I "fix" the issue by freezing the version
Cache was used in the sense: add all components in shacache, as I was told the test bot automatically upload to shacache. This will be really useful for tomcat, which always remove past versions from the public archive.
-
Owner
Ah I see. Maybe we can get it from https://archive.apache.org/dist/tomcat/tomcat-7/ but in any case, it's better to have it in shacache.
-
added 1 commit
- 4b2d89d9 - testing/testcase: check executables are not linked with system libraries
Toggle commit list -
unmarked as a Work In Progress
Toggle commit list -
added 1 commit
- 4656f063 - testing/testcase: check executables are not linked with system libraries
Toggle commit list -
merged
Toggle commit list -
-
-
255 256 def checkExecutableLink(paths_to_check, valid_paths_for_libs): 257 # type: (Iterable[str], Iterable[str]) -> List[str] 258 """Check shared libraries linked with executables in `paths_to_check`. 259 Only libraries from `valid_paths_for_libs` are accepted. 260 Returns a list of error messages. 261 """ 262 executable_link_error_list = [] 263 for path in paths_to_check: 264 for root, dirs, files in os.walk(path): 265 for f in files: 266 f = os.path.join(root, f) 267 if any(fnmatch.fnmatch(f, ignored_pattern) 268 for ignored_pattern in ignored_file_patterns): 269 continue 270 if os.access(f, os.X_OK): -
Owner
Some libraries do not have 'executable' flag...
$ find parts/ -name 'lib*so.*' -type f ! -executable parts/bzip2/lib/libbz2.so.1.0.6 parts/librsync/lib/librsync.so.2.0.0 parts/isl/lib/libisl.so.15.1.1-gdb.py parts/gettext/lib/libintl.so.8.1.5 parts/openldap/lib/libldap-2.4.so.2.10.5 parts/openldap/lib/liblber-2.4.so.2.10.5 parts/openldap/lib/libldap_r-2.4.so.2.10.5 parts/libreoffice-bin/program/libfbembed.so.2.5 parts/serf/lib/libserf-1.so.0.0.0 parts/gcc/lib64/libstdc++.so.6.0.21-gdb.py parts/gcc/lib64/libgcc_s.so.1 parts/mariadb/lib/libmysqlclient.so.18.0.0 -
Owner
True, this was a translation of the
find -type f | xargs ldd | grepwe used before, but I don't see any good reason to only check executables.I'm thinking that probably we don't need this
if os.access(f, os.X_OK):and we can pass all files through this ldd check, since ldd should just outputnot a dynamic executablefor files it does not know (like thisparts/isl/lib/libisl.so.15.1.1-gdb.pyfor example). This is already something slow but it runs only once after spending a few hours installing software, so that should not be an issue either.Do you think we should only check
'lib*so.*'? -
Owner
Including all files will take significantly more time and also there are
*.so(notlib*.so.*) likeparts/mariadb/lib/plugin/*.so. So what we should check are executables and*.so(including symlinks), I believe. -
Owner
Ah yes good idea,
*.soincluding symlinks so that we checklibssl.so.1.1throughlibssl.soin this case:dr-xr-x--- 4 slapuser11 slapuser11 4096 Dec 18 00:32 . dr-xr-x--- 6 slapuser11 slapuser11 4096 Dec 18 00:32 .. dr-xr-x--- 2 slapuser11 slapuser11 4096 Dec 18 00:32 engines-1.1 -r--r--r-- 1 slapuser11 slapuser11 5600374 Dec 18 00:32 libcrypto.a lrwxrwxrwx 1 slapuser11 slapuser11 16 Dec 18 00:32 libcrypto.so -> libcrypto.so.1.1 -r-xr-xr-x 1 slapuser11 slapuser11 3382080 Dec 18 00:32 libcrypto.so.1.1 -r--r--r-- 1 slapuser11 slapuser11 1025640 Dec 18 00:32 libssl.a lrwxrwxrwx 1 slapuser11 slapuser11 13 Dec 18 00:32 libssl.so -> libssl.so.1.1 -r-xr-xr-x 1 slapuser11 slapuser11 688984 Dec 18 00:32 libssl.so.1.1 dr-xr-x--- 2 slapuser11 slapuser11 4096 Dec 18 00:32 pkgconfigThanks, I did that in !180 (merged)
-
-
-
