...
 
Commits (99)
Changes
=======
0.25 (2022-04-05)
-----------------
* generate-signature-key: improve options.
0.24 (2022-03-30)
-----------------
* Fix upload on Python 3 when pyOpenSSL is not available.
* download,upload: fix --url help.
* generate-signature-key: get output path from command line
rather than configuration file.
0.23 (2022-03-28)
-----------------
* Require private key to have a matching certificate.
* Do not require useless metadata.
* New tryDownload/tryUpload methods.
0.22 (2021-12-08)
-----------------
* Move `_verifySignatureInCertificateList` to a public method
0.21 (2021-09-28)
-----------------
* Support python3.9 (remove usage of deprecated functions)
0.20 (2019-12-03)
-----------------
* Drop wrong signature certificates with message and without failing.
0.19 (2019-08-03)
-----------------
* Fix download auto retrying function.
0.18 (2019-08-02)
-----------------
* Add download auto retrying function.
0.17 (2019-06-07)
-----------------
* Fix checksum verification of downloaded data when `.read()` is used.
0.16 (2018-06-15)
-----------------
* Verify checksum of downloaded data.
* Make indexing optional when uploading from command-line.
0.15 (2017-06-09)
-----------------
* Add support for Python 3.
* cli: new --suffix-key option.
0.14.5 (2015-09-25)
-------------------
* Support shacache-ca-file and shadir-ca-file options in networkcachehelper.
0.14.4 (2015-09-24)
-------------------
* Add shacache-ca-file and shadir-ca-file options, that are
required to use a self-signed server certificate in python >= 2.7.9.
0.14.3 (2015-09-07)
-------------------
* Make information dict use str instead of unicode.
0.14.2 (2014-10-09)
-------------------
* Compatibility with pyOpenSSL >= 0.14
0.14.1 (2014-03-17)
-------------------
* Use 'openssl' executable if pyOpenSSL is not available.
0.14 (2013-07-12)
-----------------
* New scripts to download & upload manually from command line.
* Small API changes. `slapos.networkcachehelper` is deprecated.
* Many bugfixes and code cleanup.
* Performance/reliability improvements, by:
- using `pyOpenSSL` instead of spawning `openssl` subprocesses
- reducing the number of created temporary files
0.13.4 (2013-05-13)
-------------------
* Define timeouts for every connection we initiate. May allow not to hang
forever in a hostile environment when connection to networkcache server
can stall / be reset.
0.13.3 (2012-12-11)
-------------------
* Don't use logger.debug() but logger.info() in helpers so that it doesn't.
silent an error.
0.13.2 (2012-09-04)
-------------------
* Fix regression where multiple certificates caused most of them to be
ignored.
0.13.1 (2012-09-04)
-------------------
* Correctly return False if no entry is found while downloading.
0.13 (2012-09-02)
-----------------
* Add high-level helper functions to easily download/upload to networkcache.
* Set timeout in httplib connections.
* networkcache won't stupidly loop for 1000 iterations if "certificate"
parameter is a string instead of a list.
0.12 (2012-02-09)
-----------------
* Binary cache support.
0.11 (2011-12-14)
-----------------
* If given key has multiple *signed* values pick up the first one.
0.10 (2011-09-05)
-----------------
* Create infinite certificates.
0.9 (2011-09-02)
----------------
* Bugfix: Do not trust received content.
0.8 (2011-09-02)
----------------
* Bugfix: Do not try to validate against empty signatures.
0.7 (2011-09-02)
----------------
* Remove M2Crypto dependency and rely on openssl binary presence.
* Fix signing and verification.
* Simplify key generation and use slapos.cfg file by default.
* Internals: Increase test coverage.
0.6 (2011-08-31)
----------------
* Authentication keys are supported.
0.5 (2011-08-25)
----------------
* Re-implemente signature support.
* Follow corrected specification.
* internals: Use urllib2.
0.4 (2011-08-10)
----------------
* Implement signature checking of downloaded content.
0.3 (2011-08-03)
----------------
* Deal with proxy in correct way.
0.2 (2011-07-01)
----------------
* Incompatible change: NetworkcacheClient.download returns opened temporary
file object, which will be deleted on close. This minimises memory footprint.
* Minimise memory footprint during upload.
* Use PUT instead of POST during upload.
0.1 (2011-06-23)
----------------
* Initial version.
Changes
=======
0.12 (unreleased)
-----------------
* No changes yet.
0.11 (2011-12-14)
-----------------
* If given key has multiple *signed* values pick up the first one.
[Łukasz Nowak]
0.10 (2011-09-05)
-----------------
* Create infinite certificates. [Łukasz Nowak]
0.9 (2011-09-02)
----------------
* Bugfix: Do not trust received content. [Łukasz Nowak]
0.8 (2011-09-02)
----------------
* Bugfix: Do not try to validate against empty signatures. [Łukasz Nowak]
0.7 (2011-09-02)
----------------
* Remove M2Crypto dependency and rely on openssl binary presence. [Łukasz
Nowak]
* Fix signing and verification. [Łukasz Nowak]
* Simplify key generation and use slapos.cfg file by default. [Łukasz Nowak]
* Internals: Increase test coverage.
0.6 (2011-08-31)
----------------
* Authentication keys are supported. [Łukasz Nowak]
0.5 (2011-08-25)
----------------
* Re-implemente signature support. [Łukasz Nowak]
* Follow corrected specification. [Łukasz Nowak]
* internals: Use urllib2. [Łukasz Nowak]
0.4 (2011-08-10)
----------------
* Implement signature checking of downloaded content [Lucas Carvalho]
0.3 (2011-08-03)
----------------
* Deal with proxy in correct way [Sebastien Robin]
0.2 (2011-07-01)
----------------
* Incompatible change: NetworkcacheClient.download returns opened temporary
file object, which will be deleted on close. This minimises memory footprint.
[Lucas Carvalho]
* Minimise memory footprint during upload. [Łukasz Nowak]
* Use PUT instead of POST during upload. [Lucas Carvalho]
0.1 (2011-06-23)
----------------
* Initial version.
include CHANGES.txt
include CHANGELOG.rst
......@@ -2,17 +2,17 @@ from setuptools import setup, find_packages
import os
name = "slapos.libnetworkcache"
version = '0.12-dev'
version = '0.25'
def read(*rnames):
return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
with open(os.path.join(os.path.dirname(__file__), *rnames)) as f:
return f.read()
long_description = (
read('README.txt')
read('README.rst')
+ '\n' +
read('CHANGES.txt')
read('CHANGELOG.rst')
)
additional_install_requires = []
......@@ -32,6 +32,7 @@ setup(
keywords="slapos networkcache shadir shacache",
install_requires=[
'setuptools', # for namespace
#'pyOpenSSL',
] + additional_install_requires,
classifiers=[
'Development Status :: 4 - Beta',
......@@ -41,6 +42,8 @@ setup(
entry_points={
'console_scripts': [
'generate-signature-key = slapos.signature:run',
'networkcache-download = slapos.libnetworkcache:cmd_download',
'networkcache-upload = slapos.libnetworkcache:cmd_upload',
]
},
zip_safe=True,
......
# Compatibily code in case that pyOpenSSL is not installed.
import functools, tempfile
from subprocess import Popen, PIPE, STDOUT
_tmpfile = functools.partial(tempfile.NamedTemporaryFile, prefix=__name__+'-')
class Error(Exception): pass
FILETYPE_PEM = 1
class X509(object):
pass
def dump_publickey(type, pkey):
assert type == FILETYPE_PEM, type
pkey.seek(0, 0)
r = pkey.read()
if not r.startswith(b'-----BEGIN PUBLIC KEY-----'):
p = Popen(("openssl", "rsa", "-in", pkey.name, "-pubout"),
stdout=PIPE, stderr=PIPE)
r, err = p.communicate()
if p.poll():
raise Error(err)
return r
def load_privatekey(type, buffer):
assert type == FILETYPE_PEM, type
r = _tmpfile()
r.write(buffer.encode())
r.flush()
return r
def load_certificate(type, buffer):
# extract public key since we only use it to verify signatures
assert type == FILETYPE_PEM, type
r = _tmpfile()
p = Popen(("openssl", "x509", "-pubkey", "-noout"),
stdin=PIPE, stdout=r, stderr=PIPE)
err = p.communicate(buffer.encode())[1]
if p.poll():
raise Error(err)
cert = X509()
cert.get_pubkey = lambda: r
return cert
def sign(pkey, data, digest):
p = Popen(("openssl", digest, "-sign", pkey.name),
stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(data)
if p.poll():
raise Error(err)
return out
def verify(cert, signature, data, digest):
with _tmpfile() as f:
f.write(signature)
f.flush()
p = Popen(("openssl", digest, "-verify", cert.get_pubkey().name,
"-signature", f.name), stdin=PIPE, stdout=PIPE, stderr=STDOUT)
err = p.communicate(data)[0]
if p.poll():
raise Error(err)
This diff is collapsed.
This diff is collapsed.
##############################################################################
#
# Copyright (c) 2012 ViFiB SARL and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
# BBB: Deprecated. This file is ugly and must disappear.
# DO NOT EXTEND IT. Add methods to NetworkcacheClient class instead.
import os
import shutil
from slapos.libnetworkcache import NetworkcacheClient, logger
def __upload_network_cached(dir_url, cache_url,
file_descriptor, directory_key,
signature_private_key_file, shacache_ca_file, shacache_cert_file,
shacache_key_file, shadir_ca_file, shadir_cert_file,
shadir_key_file, metadata_dict={}):
"""
Upload content of a file descriptor to a network cache server using
shacache/shadir API.
It will upload file_descriptor content to server using directory_key as
shacache key, and metadata_dict as shadir metadata if specified.
Return True if successfull, False otherwise.
"""
if not (dir_url and cache_url):
return False
# backward compatibility
metadata_dict.setdefault('file', 'notused')
metadata_dict.setdefault('urlmd5', 'notused')
# convert '' into None in order to call nc nicely
with NetworkcacheClient(cache_url, dir_url,
signature_private_key_file=signature_private_key_file or None,
shacache_ca_file=shacache_ca_file or None,
shacache_cert_file=shacache_cert_file or None,
shacache_key_file=shacache_key_file or None,
shadir_ca_file=shadir_ca_file or None,
shadir_cert_file=shadir_cert_file or None,
shadir_key_file=shadir_key_file or None,
) as nc:
return nc.upload(file_descriptor, directory_key, **metadata_dict)
def helper_upload_network_cached_from_directory(dir_url, cache_url,
path, directory_key, metadata_dict,
signature_private_key_file, shacache_ca_file, shacache_cert_file,
shacache_key_file, shadir_ca_file, shadir_cert_file, shadir_key_file):
"""
Create a tar from a given directory (path) then upload it to networkcache.
"""
return __upload_network_cached(dir_url, cache_url,
NetworkcacheClient.archive(path.rstrip(os.sep)), directory_key,
signature_private_key_file, shacache_ca_file, shacache_cert_file,
shacache_key_file, shadir_ca_file, shadir_cert_file, shadir_key_file,
metadata_dict)
def helper_download_network_cached(dir_url, cache_url,
signature_certificate_list,
directory_key, wanted_metadata_dict={}, required_key_list=[],
strategy=None):
"""
Downloads from a network cache provider.
Select from shadir directory_key entry matching (at least)
wanted_metadata_dict and with all metadata keys in required_key_list defined
and not null.
if a "strategy" function is given as parameter, use it to choose the best
entry between list of matching entries. Otherwise, choose the first.
This strategy function takes a list of entries as parameter, and should
return the best entry.
If something fails (providor be offline, or hash_string fail), we ignore
network cached index.
return (file_descriptor, metadata) if succeeded, False otherwise.
"""
if not (dir_url and cache_url):
return
with NetworkcacheClient(cache_url, dir_url,
signature_certificate_list=signature_certificate_list) as nc:
logger.info('Downloading %s...', directory_key)
result = nc.select(directory_key, wanted_metadata_dict, required_key_list)
if strategy:
entry = None
result = list(result)
if result:
entry = strategy(result)
if not entry: # XXX: this should be the choice of 'strategy' function
logger.info("Can't find best entry matching strategy, selecting "
"random one between acceptable ones.")
entry = result[0]
else:
entry = next(result, None)
if entry:
return nc.download(entry['sha512']), entry
else:
logger.info('No matching entry to download %s', directory_key)
def helper_download_network_cached_to_file(dir_url, cache_url,
signature_certificate_list,
directory_key, path, wanted_metadata_dict={}, required_key_list=[],
strategy=None):
"""
Download a file from network cache. It is the responsibility of caller method
to check md5.
"""
result = helper_download_network_cached(dir_url, cache_url,
signature_certificate_list,
directory_key, wanted_metadata_dict, required_key_list, strategy)
if result:
# XXX check if nc filters signature_certificate_list!
# Creates a file with content to desired path.
file_descriptor, metadata_dict = result
f = open(path, 'w+b')
try:
shutil.copyfileobj(file_descriptor, f)
# XXX method should check MD5.
return metadata_dict
finally:
f.close()
file_descriptor.close()
return False
def helper_download_network_cached_to_directory(dir_url, cache_url,
signature_certificate_list,
directory_key, path, wanted_metadata_dict={}, required_key_list=[],
strategy=None):
"""
Download a tar file from network cache and untar it to specified path.
"""
result = helper_download_network_cached(dir_url, cache_url,
signature_certificate_list,
directory_key, wanted_metadata_dict, required_key_list, strategy)
if result:
file_descriptor, metadata_dict = result
try:
NetworkcacheClient.extract(path.rstrip('/'), file_descriptor)
return metadata_dict
finally:
file_descriptor.close()
......@@ -12,11 +12,14 @@
#
##############################################################################
import ConfigParser
import argparse
import os
import subprocess
import sys
try:
import configparser
except ImportError:
import ConfigParser as configparser
def generateCertificate(certificate_file, key_file, common_name):
if os.path.lexists(certificate_file):
......@@ -26,22 +29,28 @@ def generateCertificate(certificate_file, key_file, common_name):
raise ValueError("Key %r exists, will not overwrite." %
key_file)
print 'Generating certificate for %r (key: %r, certficate: %r)' % (
common_name, key_file, certificate_file)
print('Generating certificate for %r (key: %r, certficate: %r)\n' % (
common_name, key_file, certificate_file))
subj = '/CN=%s' % common_name
subprocess.check_call(["openssl", "req", "-x509", "-nodes", "-days", "36500",
subprocess.check_call(
["openssl", "req", "-x509", "-nodes", "-days", "36500",
"-subj", subj, "-newkey", "rsa:1024", "-keyout", key_file, "-out",
certificate_file])
if certificate_file != '-':
with open(certificate_file, 'r') as f:
print(f.read())
print("\nDon't forget to add the certificate to the "
"signature-certificate-list in your SlapOS configuration file.")
def run(*args):
def run(args=None):
parser = argparse.ArgumentParser()
parser.add_argument('slapos_config', type=argparse.FileType('r'),
help='SlapOS configuration file.')
parser.add_argument('--cert-file', default='-',
help='Path of certificate to generate (by default, only print on stdout)')
parser.add_argument('key_file',
help='Key file to generate.')
parser.add_argument('common_name',
help='Common name to use in the generated certificate.')
args = parser.parse_args(args)
config = ConfigParser.SafeConfigParser()
option = parser.parse_args(list(args) or sys.argv[1:])
config.readfp(option.slapos_config)
generateCertificate(config.get('networkcache', 'signature_certificate_file'),
config.get('networkcache', 'signature_private_key_file'),
config.get('slapos', 'computer_id'))
generateCertificate(args.cert_file, args.key_file, args.common_name)