Skip to content

Z
ZEO
Commit 32c6a136 authored by Éloi Rivard's avatar Éloi Rivard Committed by GitHub
Browse files
Options

Merge pull request #157 from azmeuk/doc 

Documentation draft
parents 881ef452 e6dff300
Hide whitespace changes
Inline Side-by-side
Showing with 1076 additions and 717 deletions
.gitignore
View file @ 32c6a136
...@@ -12,3 +12,4 @@ testing.log ...@@ -12,3 +12,4 @@ testing.log
data.fs* data.fs*
ZEO_forker*.conf ZEO_forker*.conf
.eggs/ .eggs/
doc/_build
MANIFEST.in
View file @ 32c6a136
...@@ -9,5 +9,6 @@ include log.ini ...@@ -9,5 +9,6 @@ include log.ini
recursive-include src * recursive-include src *
recursive-include doc *.txt recursive-include doc *.txt
recursive-include doc *.rst
global-exclude *.pyc global-exclude *.pyc
README.rst
View file @ 32c6a136
...@@ -18,662 +18,4 @@ Some alternatives to ZEO: ...@@ -18,662 +18,4 @@ Some alternatives to ZEO:
- `RelStorage <http://relstorage.readthedocs.io/en/latest/>`_ - `RelStorage <http://relstorage.readthedocs.io/en/latest/>`_
leverages the RDBMS servers to provide a client-server storage. leverages the RDBMS servers to provide a client-server storage.
.. contents:: The documentation is available on `readthedocs <http://zeo.readthedocs.io/en/latest/>`_.
Introduction
============
There are several features that affect the behavior of
ZEO. This section describes how a few of these features
work. Subsequent sections describe how to configure every option.
Client cache
------------
Each ZEO client keeps an on-disk cache of recently used data records
to avoid fetching those records from the server each time they are
requested. It is usually faster to read the objects from disk than it
is to fetch them over the network. The cache can also provide
read-only copies of objects during server outages.
The cache may be persistent or transient. If the cache is persistent,
then the cache files are retained for use after process restarts. A
non-persistent cache uses temporary files that are removed when the
client storage is closed.
The client cache size is configured when the ClientStorage is created.
The default size is 20MB, but the right size depends entirely on the
particular database. Setting the cache size too small can hurt
performance, but in most cases making it too big just wastes disk
space.
ZEO uses invalidations for cache consistency. Every time an object is
modified, the server sends a message to each client informing it of
the change. The client will discard the object from its cache when it
receives an invalidation. (It's actually a little more complicated,
but we won't get into that here.)
Each time a client connects to a server, it must verify that its cache
contents are still valid. (It did not receive any invalidation
messages while it was disconnected.) This involves asking the server
to replay invalidations it missed. If it's been disconnected too long,
it discards its cache.
Invalidation queue
------------------
The ZEO server keeps a queue of recent invalidation messages in
memory. When a client connects to the server, it sends the timestamp
of the most recent invalidation message it has received. If that
message is still in the invalidation queue, then the server sends the
client all the missing invalidations.
The default size of the invalidation queue is 100. If the
invalidation queue is larger, it will be more likely that a client
that reconnects will be able to verify its cache using the queue. On
the other hand, a large queue uses more memory on the server to store
the message. Invalidation messages tend to be small, perhaps a few
hundred bytes each on average; it depends on the number of objects
modified by a transaction.
You can also provide an invalidation age when configuring the
server. In this case, if the invalidation queue is too small, but a
client has been disconnected for a time interval that is less than the
invalidation age, then invalidations are replayed by iterating over
the lower-level storage on the server. If the age is too high, and
clients are disconnected for a long time, then this can put a lot of
load on the server.
Transaction timeouts
--------------------
A ZEO server can be configured to timeout a transaction if it takes
too long to complete. Only a single transaction can commit at a time;
so if one transaction takes too long, all other clients will be
delayed waiting for it. In the extreme, a client can hang during the
commit process. If the client hangs, the server will be unable to
commit other transactions until it restarts. A well-behaved client
will not hang, but the server can be configured with a transaction
timeout to guard against bugs that cause a client to hang.
If any transaction exceeds the timeout threshold, the client's
connection to the server will be closed and the transaction aborted.
Once the transaction is aborted, the server can start processing other
client's requests. Most transactions should take very little time to
commit. The timer begins for a transaction after all the data has
been sent to the server. At this point, the cost of commit should be
dominated by the cost of writing data to disk; it should be unusual
for a commit to take longer than 1 second. A transaction timeout of
30 seconds should tolerate heavy load and slow communications between
client and server, while guarding against hung servers.
When a transaction times out, the client can be left in an awkward
position. If the timeout occurs during the second phase of the two
phase commit, the client will log a panic message. This should only
cause problems if the client transaction involved multiple storages.
If it did, it is possible that some storages committed the client
changes and others did not.
Connection management
---------------------
A ZEO client manages its connection to the ZEO server. If it loses
the connection, it attempts to reconnect. While
it is disconnected, it can satisfy some reads by using its cache.
The client can be configured with multiple server addresses. In this
case, it assumes that each server has identical content and will use
any server that is available. It is possible to configure the client
to accept a read-only connection to one of these servers if no
read-write connection is available. If it has a read-only connection,
it will continue to poll for a read-write connection.
If a single address resolves to multiple IPv4 or IPv6 addresses,
the client will connect to an arbitrary of these addresses.
SSL
---
ZEO supports the use of SSL connections between servers and clients,
including certificate authentication. We're still understanding use
cases for this, so details of operation may change.
Installing software
===================
ZEO is installed like any other Python package using pip, buildout, or
other Python packaging tools.
Running the server
==================
Typically, the ZEO server is run using the ``runzeo`` script that's
installed as part of a ZEO installation. The ``runzeo`` script
accepts command line options, the most important of which is the
``-C`` (``--configure``) option. ZEO servers are best configured
via configuration files. The ``runzeo`` script also accepts some
command-line arguments for ad-hoc configurations, but there's an
easier way to run an ad-hoc server described below. For more on
configuring a ZEO server see `Server configuration`_ below.
Server quick-start/ad-hoc operation
-----------------------------------
You can quickly start a ZEO server from a Python prompt::
import ZEO
address, stop = ZEO.server()
This runs a ZEO server on a dynamic address and using an in-memory
storage.
We can then create a ZEO client connection using the address
returned::
connection = ZEO.connection(address)
This is a ZODB connection for a database opened on a client storage
instance created on the fly. This is a shorthand for::
db = ZEO.DB(address)
connection = db.open()
Which is a short-hand for::
client_storage = ZEO.client(address)
import ZODB
db = ZODB.db(client_storage)
connection = db.open()
If you exit the Python process, the storage exits as well, as it's run
in an in-process thread.
You shut down the server more cleanly by calling the stop function
returned by the ``ZEO.server`` function.
To have data stored persistently, you can specify a file-storage path
name using a ``path`` parameter. If you want blob support, you can
specify a blob-file directory using the ``blob_dir`` directory.
You can also supply a port to listen on, full storage configuration
and ZEO server configuration options to the ``ZEO.server``
function. See it's documentation string for more information.
Server configuration
--------------------
The script ``runzeo`` runs the ZEO server. The server can be
configured using command-line arguments or a configuration file. This
document only describes the configuration file. Run ``runzeo``
-h to see the list of command-line arguments.
The configuration file specifies the underlying storage the server
uses, the address it binds to, and a few other optional parameters.
An example is::
<zeo>
address zeo.example.com:8090
</zeo>
<filestorage>
path /var/tmp/Data.fs
</filestorage>
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
The format is similar to the Apache configuration format. Individual
settings have a name, 1 or more spaces and a value, as in::
address zeo.example.com:8090
Settings are grouped into hierarchical sections.
The example above configures a server to use a file storage from
``/var/tmp/Data.fs``. The server listens on port ``8090`` of
``zeo.example.com``. The ZEO server writes its log file to
``/var/tmp/zeo.log`` and uses a custom format for each line. Assuming the
example configuration it stored in ``zeo.config``, you can run a server by
typing::
runzeo -C zeo.config
A configuration file consists of a ``<zeo>`` section and a storage
section, where the storage section can use any of the valid ZODB
storage types. It may also contain an event log configuration. See
`ZODB documentation <http://www.zodb.org>`_ for information on
configuring storages.
The ``zeo`` section must list the address. All the other keys are
optional.
address
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
read-only
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
invalidation-queue-size
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to support client cache verification when a client
disconnects for a short period of time.
invalidation-age
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
transaction-timeout
The maximum amount of time, in seconds, to wait for a
transaction to commit after acquiring the storage lock,
specified in seconds. If the transaction takes too long, the
client connection will be closed and the transaction aborted.
This defaults to 30 seconds.
client-conflict-resolution
Flag indicating that clients should perform conflict
resolution. This option defaults to false.
msgpack
Use `msgpack <http://msgpack.org/index.html>`_ to serialize
and de-serialize ZEO protocol messages.
An advantage of using msgpack for ZEO communication is that
it's a tiny bit faster and a ZEO server can support Python 2
or Python 3 clients (but not both).
msgpack can also be enabled by setting the ``ZEO_MSGPACK``
environment to a non-empty string.
Server SSL configuration
~~~~~~~~~~~~~~~~~~~~~~~~
A server can optionally support SSL. Do do so, include a `ssl`
subsection of the ZEO section, as in::
<zeo>
address zeo.example.com:8090
<ssl>
certificate server_certificate.pem
key server_certificate_key.pem
</ssl>
</zeo>
<filestorage>
path /var/tmp/Data.fs
</filestorage>
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
The ``ssl`` section has settings:
certificate
The path to an SSL certificate file for the server. (required)
key
The path to the SSL key file for the server certificate (if not
included in certificate file).
password-function
The dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
authenticate
The path to a file or directory containing client certificates
to authenticate. (See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
If this setting is used. then certificate authentication is
used to authenticate clients. A client must be configured
with one of the certificates supplied using this setting.
This option assumes that you're using self-signed certificates.
Running the ZEO server as a daemon
----------------------------------
In an operational setting, you will want to run the ZEO server as a
daemon process that is restarted when it dies. ``runzeo`` makes no
attempt to implement a well behaved daemon. It expects that
functionality to be provided by a wrapper like `zdaemon
<https://pypi.python.org/pypi/zdaemon>`_ or `supervisord
<http://supervisord.org/>`_.
Rotating log files
------------------
``runzeo`` will re-initialize its logging subsystem when it receives a
SIGUSR2 signal. If you are using the standard event logger, you
should first rename the log file and then send the signal to the
server. The server will continue writing to the renamed log file
until it receives the signal. After it receives the signal, the
server will create a new file with the old name and write to it.
ZEO Clients
===========
To use a ZEO server, you need to connect to it using a ZEO client
storage. You create client storages either using a Python API or
using a ZODB storage configuration in a ZODB storage configuration
section.
Python API for creating a ZEO client storage
--------------------------------------------
To create a client storage from Python, use the ``ZEO.client``
function::
import ZEO
client = ZEO.client(8200)
In the example above, we created a client that connected to a storage
listening on port 8200 on local host. The first argument is an
address, or list of addresses to connect to. There are many additional
options, documented below that should be given as keyword arguments.
Addresses can be:
- A host/port tuple
- An integer, which implies that the host is '127.0.0.1'
- A unix domain socket file name.
Options:
cache_size
The cache size in bytes. This defaults to a 20MB.
cache
The ZEO cache to be used. This can be a file name, which will
cause a persistent standard persistent ZEO cache to be used and
stored in the given name. This can also be an object that
implements ``ZEO.interfaces.ICache``.
If not specified, then a non-persistent cache will be used.
blob_dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared_blob_dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob_cache_size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob_cache_size_check
The total size of data to be downloaded to trigger blob cache size
reduction. The default is 10 (percent). This controls how often to
remove blobs from the cache.
ssl
An ``ssl.SSLContext`` object used to make SSL connections.
ssl_server_hostname
Host name to use for SSL host name checks.
If using SSL and if host name checking is enabled in the given SSL
context then use this as the value to check. If an address is a
host/port pair, then this defaults to the host in the address.
read_only
Set to true for a read-only connection.
If false (the default), then request a read/write connection.
This option is ignored if ``read_only_fallback`` is set to a true value.
read_only_fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
server_sync
Flag, false by default, indicating whether the ``sync`` method
should make a server request. The ``sync`` method is called at the
start of explicitly begin transactions. Making a server requests assures
that any invalidations outstanding at the beginning of a
transaction are processed.
Setting this to True is important when application activity is
spread over multiple ZEO clients. The classic example of this is
when a web browser makes a request to an application server (ZEO
client) that makes a change and then makes a request to another
application server that depends on the change.
Setting this to True makes transactions a little slower because of
the added server round trip. For transactions that don't otherwise
need to access the storage server, the impact can be significant.
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
After the initial connection, if the client is disconnected:
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
Configuration strings/files
---------------------------
ZODB databases and storages can be configured using configuration
files, or strings (extracted from configuration files). They use the
same syntax as the server configuration files described above, but
with different sections and options.
An application that used ZODB might configure it's database using a
string like::
<zodb>
cache-size-bytes 1000MB
<filestorage>
path /var/lib/Data.fs
</filestorage>
</zodb>
In this example, we configured a ZODB database with a object cache
size of 1GB. Inside the database, we configured a file storage. The
``filestorage`` section provided file-storage parameters. We saw a
similar section in the storage-server configuration example in `Server
configuration`_.
To configure a client storage, you use a ``clientstorage`` section,
but first you have to import it's definition, because ZEO isn't built
into ZODB. Here's an example::
<zodb>
cache-size-bytes 1000MB
%import ZEO
<clientstorage>
server 8200
</clientstorage>
</zodb>
In this example, we defined a client storage that connected to a
server on port 8200.
The following settings are supported:
cache-size
The cache size in bytes, KB or MB. This defaults to a 20MB.
Optional ``KB`` or ``MB`` suffixes can (and usually are) used to
specify units other than bytes.
cache-path
The file path of a persistent cache file
blob-dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared-blob-dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob-cache-size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob-cache-size-check
The total size of data to be downloaded to trigger blob cache size
reduction. The default is 10 (percent). This controls how often to
remove blobs from the cache.
read-only
Set to true for a read-only connection.
If false (the default), then request a read/write connection.
This option is ignored if ``read_only_fallback`` is set to a true value.
read-only-fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
server-sync
Sets the ``server_sync`` option described above.
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
After the initial connection, if the client is disconnected:
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
Client SSL configuration
~~~~~~~~~~~~~~~~~~~~~~~~
An ``ssl`` subsection can be used to enable and configure SSL, as in::
%import ZEO
<clientstorage>
server zeo.example.com8200
<ssl>
</ssl>
</clientstorage>
In the example above, SSL is enabled in it's simplest form:
- The client expects the server to have a signed certificate, which the
client validates.
- The server server host name ``zeo.example.com`` is checked against
the server's certificate.
A number of settings can be provided to configure SSL:
certificate
The path to an SSL certificate file for the client. This is
needed to allow the server to authenticate the client.
key
The path to the SSL key file for the client certificate (if not
included in the certificate file).
password-function
A dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
authenticate
The path to a file or directory containing server certificates
to authenticate. (See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
If this setting is used. then certificate authentication is
used to authenticate the server. The server must be configured
with one of the certificates supplied using this setting.
check-hostname
This is a boolean setting that defaults to true. Verify the
host name in the server certificate is as expected.
server-hostname
The expected server host name. This defaults to the host name
used in the server address. This option must be used when
``check-hostname`` is true and when a server address has no host
name (localhost, or unix domain socket) or when there is more
than one server and server hostnames differ.
Using this setting implies a true value for the ``check-hostname`` setting.
doc/changelog.rst 0 → 100644
View file @ 32c6a136
.. include:: ../CHANGES.rst
doc/zeo-client-cache-tracing.txt → doc/client-cache-tracing.rst
View file @ 32c6a136
========================
ZEO Client Cache Tracing ZEO Client Cache Tracing
======================== ========================
...@@ -11,7 +12,7 @@ ZEO client cache is only used when an object is not in the Zope object ...@@ -11,7 +12,7 @@ ZEO client cache is only used when an object is not in the Zope object
cache; the ZEO client cache avoids roundtrips to the ZEO server. cache; the ZEO client cache avoids roundtrips to the ZEO server.
Enabling Cache Tracing Enabling Cache Tracing
---------------------- ======================
To enable cache tracing, you must use a persistent cache (specify a ``client`` To enable cache tracing, you must use a persistent cache (specify a ``client``
name), and set the environment variable ZEO_CACHE_TRACE to a non-empty name), and set the environment variable ZEO_CACHE_TRACE to a non-empty
...@@ -32,7 +33,7 @@ transaction comments, access paths, or machine information (such as machine ...@@ -32,7 +33,7 @@ transaction comments, access paths, or machine information (such as machine
name or IP address) are logged. name or IP address) are logged.
Analyzing a Cache Trace Analyzing a Cache Trace
----------------------- =======================
The cache_stats.py command-line tool (``python -m The cache_stats.py command-line tool (``python -m
ZEO.scripts.cache_stats``) is the first-line tool to analyze a cache ZEO.scripts.cache_stats``) is the first-line tool to analyze a cache
...@@ -55,7 +56,7 @@ extension. It will be read from stdin (assuming uncompressed data) if the ...@@ -55,7 +56,7 @@ extension. It will be read from stdin (assuming uncompressed data) if the
tracefile argument is '-'. tracefile argument is '-'.
Simulating Different Cache Sizes Simulating Different Cache Sizes
-------------------------------- ================================
Based on a cache trace file, you can make a prediction of how well the cache Based on a cache trace file, you can make a prediction of how well the cache
might do with a different cache size. The cache_simul.py tool runs a simulation of might do with a different cache size. The cache_simul.py tool runs a simulation of
...@@ -122,7 +123,7 @@ strategy's code has been updated to be aware of MVCC, these are not further ...@@ -122,7 +123,7 @@ strategy's code has been updated to be aware of MVCC, these are not further
documented here. documented here.
Simulation Limitations Simulation Limitations
---------------------- ======================
The cache simulation is an approximation, and actual hit rate may be higher The cache simulation is an approximation, and actual hit rate may be higher
or lower than the simulated result. These are some factors that inhibit or lower than the simulated result. These are some factors that inhibit
......
doc/client-cache.rst 0 → 100644
View file @ 32c6a136
================
ZEO Client Cache
================
The client cache provides a disk based cache for each ZEO client. The
client cache allows reads to be done from local disk rather than by remote
access to the storage server.
The cache may be persistent or transient. If the cache is persistent, then
the cache file is retained for use after process restarts. A non-
persistent cache uses a temporary file.
The client cache is managed in a single file, of the specified size.
The life of the cache is as follows:
- The cache file is opened (if it already exists), or created and set to
the specified size.
- Cache records are written to the cache file, as transactions commit
locally, and as data are loaded from the server.
- Writes are to "the current file position". This is a pointer that
travels around the file, circularly. After a record is written, the
pointer advances to just beyond it. Objects starting at the current
file position are evicted, as needed, to make room for the next record
written.
A distinct index file is not created, although indexing structures are
maintained in memory while a ClientStorage is running. When a persistent
client cache file is reopened, these indexing structures are recreated
by analyzing the file contents.
Persistent cache files are created in the directory named in the ``var``
argument to the ClientStorage, or if ``var`` is None, in the current
working directory. Persistent cache files have names of the form::
client-storage.zec
where:
client -- the client name, as given by the ClientStorage's ``client``
argument
storage -- the storage name, as given by the ClientStorage's ``storage``
argument; this is typically a string denoting a small integer,
"1" by default
For example, the cache file for client '8881' and storage 'spam' is named
"8881-spam.zec".
doc/clients.rst 0 → 100644
View file @ 32c6a136
===========
ZEO Clients
===========
To use a ZEO server, you need to connect to it using a ZEO client
storage. You create client storages either using a Python API or
using a ZODB storage configuration in a ZODB storage configuration
section.
Python API for creating a ZEO client storage
============================================
To create a client storage from Python, use the ``ZEO.client``
function::
import ZEO
client = ZEO.client(8200)
In the example above, we created a client that connected to a storage
listening on port 8200 on local host. The first argument is an
address, or list of addresses to connect to. There are many additional
options, documented below that should be given as keyword arguments.
Addresses can be:
- A host/port tuple
- An integer, which implies that the host is '127.0.0.1'
- A unix domain socket file name.
Options:
cache_size
The cache size in bytes. This defaults to a 20MB.
cache
The ZEO cache to be used. This can be a file name, which will
cause a persistent standard persistent ZEO cache to be used and
stored in the given name. This can also be an object that
implements ``ZEO.interfaces.ICache``.
If not specified, then a non-persistent cache will be used.
blob_dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared_blob_dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob_cache_size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob_cache_size_check
The total size of data to be downloaded to trigger blob cache size
reduction. The default is 10 (percent). This controls how often to
remove blobs from the cache.
ssl
An ``ssl.SSLContext`` object used to make SSL connections.
ssl_server_hostname
Host name to use for SSL host name checks.
If using SSL and if host name checking is enabled in the given SSL
context then use this as the value to check. If an address is a
host/port pair, then this defaults to the host in the address.
read_only
Set to true for a read-only connection.
If false (the default), then request a read/write connection.
This option is ignored if ``read_only_fallback`` is set to a true value.
read_only_fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
server_sync
Flag, false by default, indicating whether the ``sync`` method
should make a server request. The ``sync`` method is called at the
start of explicitly begin transactions. Making a server requests assures
that any invalidations outstanding at the beginning of a
transaction are processed.
Setting this to True is important when application activity is
spread over multiple ZEO clients. The classic example of this is
when a web browser makes a request to an application server (ZEO
client) that makes a change and then makes a request to another
application server that depends on the change.
Setting this to True makes transactions a little slower because of
the added server round trip. For transactions that don't otherwise
need to access the storage server, the impact can be significant.
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
After the initial connection, if the client is disconnected:
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
Configuration strings/files
===========================
ZODB databases and storages can be configured using configuration
files, or strings (extracted from configuration files). They use the
same syntax as the server configuration files described above, but
with different sections and options.
An application that used ZODB might configure it's database using a
string like::
<zodb>
cache-size-bytes 1000MB
<filestorage>
path /var/lib/Data.fs
</filestorage>
</zodb>
In this example, we configured a ZODB database with a object cache
size of 1GB. Inside the database, we configured a file storage. The
``filestorage`` section provided file-storage parameters. We saw a
similar section in the storage-server configuration example in `Server
configuration`_.
To configure a client storage, you use a ``clientstorage`` section,
but first you have to import it's definition, because ZEO isn't built
into ZODB. Here's an example::
<zodb>
cache-size-bytes 1000MB
%import ZEO
<clientstorage>
server 8200
</clientstorage>
</zodb>
In this example, we defined a client storage that connected to a
server on port 8200.
The following settings are supported:
cache-size
The cache size in bytes, KB or MB. This defaults to a 20MB.
Optional ``KB`` or ``MB`` suffixes can (and usually are) used to
specify units other than bytes.
cache-path
The file path of a persistent cache file
blob-dir
The name of a directory to hold/cache blob data downloaded from the
server. This must be provided if blobs are to be used. (Of
course, the server storage must be configured to use blobs as
well.)
shared-blob-dir
A client can use a network files system (or a local directory if
the server runs on the same machine) to share a blob directory with
the server. This allows downloading of blobs (except via a
distributed file system) to be avoided.
blob-cache-size
The size of the blob cache in bytes. IF unset, then blobs will
accumulate. If set, then blobs are removed when the total size
exceeds this amount. Blobs accessed least recently are removed
first.
blob-cache-size-check
The total size of data to be downloaded to trigger blob cache size
reduction. The default is 10 (percent). This controls how often to
remove blobs from the cache.
read-only
Set to true for a read-only connection.
If false (the default), then request a read/write connection.
This option is ignored if ``read_only_fallback`` is set to a true value.
read-only-fallback
Set to true, then prefer a read/write connection, but be willing to
use a read-only connection. This defaults to a false value.
If ``read_only_fallback`` is set, then ``read_only`` is ignored.
server-sync
Sets the ``server_sync`` option described above.
wait_timeout
How long to wait for an initial connection, defaulting to 30
seconds. If an initial connection can't be made within this time
limit, then creation of the client storage will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
After the initial connection, if the client is disconnected:
- In-flight server requests will fail with a
``ZEO.Exceptions.ClientDisconnected`` exception.
- New requests will block for up to ``wait_timeout`` waiting for a
connection to be established before failing with a
``ZEO.Exceptions.ClientDisconnected`` exception.
client_label
A short string to display in *server* logs for an event relating to
this client. This can be helpful when debugging.
disconnect_poll
The delay in seconds between attempts to connect to the
server, in seconds. Defaults to 1 second.
Client SSL configuration
------------------------
An ``ssl`` subsection can be used to enable and configure SSL, as in::
%import ZEO
<clientstorage>
server zeo.example.com8200
<ssl>
</ssl>
</clientstorage>
In the example above, SSL is enabled in it's simplest form:
- The client expects the server to have a signed certificate, which the
client validates.
- The server server host name ``zeo.example.com`` is checked against
the server's certificate.
A number of settings can be provided to configure SSL:
certificate
The path to an SSL certificate file for the client. This is
needed to allow the server to authenticate the client.
key
The path to the SSL key file for the client certificate (if not
included in the certificate file).
password-function
A dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
authenticate
The path to a file or directory containing server certificates
to authenticate. (See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
If this setting is used. then certificate authentication is
used to authenticate the server. The server must be configured
with one of the certificates supplied using this setting.
check-hostname
This is a boolean setting that defaults to true. Verify the
host name in the server certificate is as expected.
server-hostname
The expected server host name. This defaults to the host name
used in the server address. This option must be used when
``check-hostname`` is true and when a server address has no host
name (localhost, or unix domain socket) or when there is more
than one server and server hostnames differ.
Using this setting implies a true value for the ``check-hostname`` setting.
doc/conf.py 0 → 100644
View file @ 32c6a136
# -*- coding: utf-8 -*-
#
# ZEO documentation build configuration file, created by
# sphinx-quickstart on Thu Oct 18 00:16:24 2012.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
import pkg_resources
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath(".."))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
rqmt = pkg_resources.require('ZEO')[0]
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.todo',
'sphinx.ext.viewcode',
'repoze.sphinx.autointerface',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'ZEO'
copyright = u'2012, Zope Foundation Contributors'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '%s.%s' % tuple(map(int, rqmt.version.split('.')[:2]))
# The full version, including alpha/beta/rc tags.
release = version
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'ZEOdoc'
# -- Options for LaTeX output --------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'ZEO.tex', u'ZEO Documentation',
u'Zope Foundation Contributors', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'btrees', u'ZEO Documentation',
[u'Zope Foundation Contributors'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'ZEO', u'ZEO Documentation',
u'Zope Foundation Contributors', 'ZEO', 'Single-server client-server database server for ZODB',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {
"python": ('https://docs.python.org/3/', None),
"persistent": ('https://persistent.readthedocs.io/en/latest/', None),
"ZODB": ("https://zodb-docs.readthedocs.io/en/latest/", None),
}
doc/index.rst 0 → 100644
View file @ 32c6a136
==========================================================
ZEO - Single-server client-server database server for ZODB
==========================================================
ZEO is a client-server storage for `ZODB <http://www.zodb.org>`_ for
sharing a single storage among many clients. When you use ZEO, a
lower-level storage, typically a file storage, is opened in the ZEO
server process. Client programs connect to this process using a ZEO
ClientStorage. ZEO provides a consistent view of the database to all
clients. The ZEO client and server communicate using a custom
protocol layered on top of TCP.
ZEO can be installed with pip like most python packages:
.. code-block:: bash
pip install zeo
Some alternatives to ZEO:
- `NEO <http://www.neoppod.org/>`_ is a distributed-server
client-server storage.
- `RelStorage <http://relstorage.readthedocs.io/en/latest/>`_
leverages the RDBMS servers to provide a client-server storage.
Content
=======
.. toctree::
:maxdepth: 2
introduction
server
clients
protocol
nagios
ordering
client-cache
client-cache-tracing
blob-nfs
changelog
reference
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
doc/introduction.rst 0 → 100644
View file @ 32c6a136
============
Introduction
============
There are several features that affect the behavior of
ZEO. This section describes how a few of these features
work. Subsequent sections describe how to configure every option.
Client cache
============
Each ZEO client keeps an on-disk cache of recently used data records
to avoid fetching those records from the server each time they are
requested. It is usually faster to read the objects from disk than it
is to fetch them over the network. The cache can also provide
read-only copies of objects during server outages.
The cache may be persistent or transient. If the cache is persistent,
then the cache files are retained for use after process restarts. A
non-persistent cache uses temporary files that are removed when the
client storage is closed.
The client cache size is configured when the ClientStorage is created.
The default size is 20MB, but the right size depends entirely on the
particular database. Setting the cache size too small can hurt
performance, but in most cases making it too big just wastes disk
space.
ZEO uses invalidations for cache consistency. Every time an object is
modified, the server sends a message to each client informing it of
the change. The client will discard the object from its cache when it
receives an invalidation. (It's actually a little more complicated,
but we won't get into that here.)
Each time a client connects to a server, it must verify that its cache
contents are still valid. (It did not receive any invalidation
messages while it was disconnected.) This involves asking the server
to replay invalidations it missed. If it's been disconnected too long,
it discards its cache.
Invalidation queue
==================
The ZEO server keeps a queue of recent invalidation messages in
memory. When a client connects to the server, it sends the timestamp
of the most recent invalidation message it has received. If that
message is still in the invalidation queue, then the server sends the
client all the missing invalidations.
The default size of the invalidation queue is 100. If the
invalidation queue is larger, it will be more likely that a client
that reconnects will be able to verify its cache using the queue. On
the other hand, a large queue uses more memory on the server to store
the message. Invalidation messages tend to be small, perhaps a few
hundred bytes each on average; it depends on the number of objects
modified by a transaction.
You can also provide an invalidation age when configuring the
server. In this case, if the invalidation queue is too small, but a
client has been disconnected for a time interval that is less than the
invalidation age, then invalidations are replayed by iterating over
the lower-level storage on the server. If the age is too high, and
clients are disconnected for a long time, then this can put a lot of
load on the server.
Transaction timeouts
====================
A ZEO server can be configured to timeout a transaction if it takes
too long to complete. Only a single transaction can commit at a time;
so if one transaction takes too long, all other clients will be
delayed waiting for it. In the extreme, a client can hang during the
commit process. If the client hangs, the server will be unable to
commit other transactions until it restarts. A well-behaved client
will not hang, but the server can be configured with a transaction
timeout to guard against bugs that cause a client to hang.
If any transaction exceeds the timeout threshold, the client's
connection to the server will be closed and the transaction aborted.
Once the transaction is aborted, the server can start processing other
client's requests. Most transactions should take very little time to
commit. The timer begins for a transaction after all the data has
been sent to the server. At this point, the cost of commit should be
dominated by the cost of writing data to disk; it should be unusual
for a commit to take longer than 1 second. A transaction timeout of
30 seconds should tolerate heavy load and slow communications between
client and server, while guarding against hung servers.
When a transaction times out, the client can be left in an awkward
position. If the timeout occurs during the second phase of the two
phase commit, the client will log a panic message. This should only
cause problems if the client transaction involved multiple storages.
If it did, it is possible that some storages committed the client
changes and others did not.
Connection management
=====================
A ZEO client manages its connection to the ZEO server. If it loses
the connection, it attempts to reconnect. While
it is disconnected, it can satisfy some reads by using its cache.
The client can be configured with multiple server addresses. In this
case, it assumes that each server has identical content and will use
any server that is available. It is possible to configure the client
to accept a read-only connection to one of these servers if no
read-write connection is available. If it has a read-only connection,
it will continue to poll for a read-write connection.
If a single address resolves to multiple IPv4 or IPv6 addresses,
the client will connect to an arbitrary of these addresses.
SSL
===
ZEO supports the use of SSL connections between servers and clients,
including certificate authentication. We're still understanding use
cases for this, so details of operation may change.
doc/nagios.rst 0 → 100644
View file @ 32c6a136
.. include:: ../src/ZEO/nagios.rst
src/ZEO/ordering.rst → doc/ordering.rst
View file @ 32c6a136
...@@ -182,7 +182,7 @@ B3 ...@@ -182,7 +182,7 @@ B3
lastTransaction result in ``sync``. lastTransaction result in ``sync``.
Implementation notes Implementation notes
=================== ====================
ZEO 4 ZEO 4
----- -----
...@@ -195,12 +195,12 @@ and, fact that client requests, for a particular client, are ...@@ -195,12 +195,12 @@ and, fact that client requests, for a particular client, are
handled by a single thread on the server, and that all output for a handled by a single thread on the server, and that all output for a
client goes through a thread-safe queue. client goes through a thread-safe queue.
Invalidations are sent from different threads than clients. Outgoing Invalidations are sent from different threads than clients. Outgoing
data is queued, however, using Python lists, which are protected by data is queued, however, using Python lists, which are protected by
the GIL. This means that the serialization provided though storage the GIL. This means that the serialization provided though storage
locks is preserved by the way that server outputs are queued. **The locks is preserved by the way that server outputs are queued. **The
queueing mechanism is in part a consequence of the way asyncore, used queueing mechanism is in part a consequence of the way asyncore, used
by ZEO4, works. by ZEO4, works.**
In ZEO 4 clients, invalidations and loads are handled by separate In ZEO 4 clients, invalidations and loads are handled by separate
threads. This means that even though data arive in order, they may not threads. This means that even though data arive in order, they may not
......
src/ZEO/protocol.txt → doc/protocol.rst
View file @ 32c6a136
==========================================
ZEO Network Protocol (sans authentication) ZEO Network Protocol (sans authentication)
========================================== ==========================================
......
doc/reference.rst 0 → 100644
View file @ 32c6a136
=========
Reference
=========
Client & server
===============
.. autofunction:: ZEO.server
.. autofunction:: ZEO.client
.. autoclass:: ZEO.ClientStorage.ClientStorage
:members:
:show-inheritance:
.. autofunction:: ZEO.DB
Exceptions
==========
.. automodule:: ZEO.Exceptions
:members:
:show-inheritance:
doc/server.rst 0 → 100644
View file @ 32c6a136
==================
Running the server
==================
Typically, the ZEO server is run using the ``runzeo`` script that's
installed as part of a ZEO installation. The ``runzeo`` script
accepts command line options, the most important of which is the
``-C`` (``--configure``) option. ZEO servers are best configured
via configuration files. The ``runzeo`` script also accepts some
command-line arguments for ad-hoc configurations, but there's an
easier way to run an ad-hoc server described below. For more on
configuring a ZEO server see `Server configuration`_ below.
Server quick-start/ad-hoc operation
===================================
You can quickly start a ZEO server from a Python prompt::
import ZEO
address, stop = ZEO.server()
This runs a ZEO server on a dynamic address and using an in-memory
storage.
We can then create a ZEO client connection using the address
returned::
connection = ZEO.connection(address)
This is a ZODB connection for a database opened on a client storage
instance created on the fly. This is a shorthand for::
db = ZEO.DB(address)
connection = db.open()
Which is a short-hand for::
client_storage = ZEO.client(address)
import ZODB
db = ZODB.db(client_storage)
connection = db.open()
If you exit the Python process, the storage exits as well, as it's run
in an in-process thread.
You shut down the server more cleanly by calling the stop function
returned by the ``ZEO.server`` function.
To have data stored persistently, you can specify a file-storage path
name using a ``path`` parameter. If you want blob support, you can
specify a blob-file directory using the ``blob_dir`` directory.
You can also supply a port to listen on, full storage configuration
and ZEO server configuration options to the ``ZEO.server``
function. See it's documentation string for more information.
Server configuration
====================
The script ``runzeo`` runs the ZEO server. The server can be
configured using command-line arguments or a configuration file. This
document only describes the configuration file. Run ``runzeo``
-h to see the list of command-line arguments.
The configuration file specifies the underlying storage the server
uses, the address it binds to, and a few other optional parameters.
An example is::
<zeo>
address zeo.example.com:8090
</zeo>
<filestorage>
path /var/tmp/Data.fs
</filestorage>
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
The format is similar to the Apache configuration format. Individual
settings have a name, 1 or more spaces and a value, as in::
address zeo.example.com:8090
Settings are grouped into hierarchical sections.
The example above configures a server to use a file storage from
``/var/tmp/Data.fs``. The server listens on port ``8090`` of
``zeo.example.com``. The ZEO server writes its log file to
``/var/tmp/zeo.log`` and uses a custom format for each line. Assuming the
example configuration it stored in ``zeo.config``, you can run a server by
typing::
runzeo -C zeo.config
A configuration file consists of a ``<zeo>`` section and a storage
section, where the storage section can use any of the valid ZODB
storage types. It may also contain an event log configuration. See
`ZODB documentation <http://www.zodb.org>`_ for information on
configuring storages.
The ``zeo`` section must list the address. All the other keys are
optional.
address
The address at which the server should listen. This can be in
the form 'host:port' to signify a TCP/IP connection or a
pathname string to signify a Unix domain socket connection (at
least one '/' is required). A hostname may be a DNS name or a
dotted IP address. If the hostname is omitted, the platform's
default behavior is used when binding the listening socket (''
is passed to socket.bind() as the hostname portion of the
address).
read-only
Flag indicating whether the server should operate in read-only
mode. Defaults to false. Note that even if the server is
operating in writable mode, individual storages may still be
read-only. But if the server is in read-only mode, no write
operations are allowed, even if the storages are writable. Note
that pack() is considered a read-only operation.
invalidation-queue-size
The storage server keeps a queue of the objects modified by the
last N transactions, where N == invalidation_queue_size. This
queue is used to support client cache verification when a client
disconnects for a short period of time.
invalidation-age
The maximum age of a client for which quick-verification
invalidations will be provided by iterating over the served
storage. This option should only be used if the served storage
supports efficient iteration from a starting point near the
end of the transaction history (e.g. end of file).
transaction-timeout
The maximum amount of time, in seconds, to wait for a
transaction to commit after acquiring the storage lock,
specified in seconds. If the transaction takes too long, the
client connection will be closed and the transaction aborted.
This defaults to 30 seconds.
client-conflict-resolution
Flag indicating that clients should perform conflict
resolution. This option defaults to false.
msgpack
Use `msgpack <http://msgpack.org/index.html>`_ to serialize
and de-serialize ZEO protocol messages.
An advantage of using msgpack for ZEO communication is that
it's a tiny bit faster and a ZEO server can support Python 2
or Python 3 clients (but not both).
msgpack can also be enabled by setting the ``ZEO_MSGPACK``
environment to a non-empty string.
Server SSL configuration
------------------------
A server can optionally support SSL. Do do so, include a `ssl`
subsection of the ZEO section, as in::
<zeo>
address zeo.example.com:8090
<ssl>
certificate server_certificate.pem
key server_certificate_key.pem
</ssl>
</zeo>
<filestorage>
path /var/tmp/Data.fs
</filestorage>
<eventlog>
<logfile>
path /var/tmp/zeo.log
format %(asctime)s %(message)s
</logfile>
</eventlog>
The ``ssl`` section has settings:
certificate
The path to an SSL certificate file for the server. (required)
key
The path to the SSL key file for the server certificate (if not
included in certificate file).
password-function
The dotted name if an importable function that, when imported, returns
the password needed to unlock the key (if the key requires a password.)
authenticate
The path to a file or directory containing client certificates
to authenticate. (See the ``cafile`` and ``capath``
parameters in the Python documentation for
``ssl.SSLContext.load_verify_locations``.)
If this setting is used. then certificate authentication is
used to authenticate clients. A client must be configured
with one of the certificates supplied using this setting.
This option assumes that you're using self-signed certificates.
Running the ZEO server as a daemon
==================================
In an operational setting, you will want to run the ZEO server as a
daemon process that is restarted when it dies. ``runzeo`` makes no
attempt to implement a well behaved daemon. It expects that
functionality to be provided by a wrapper like `zdaemon
<https://pypi.python.org/pypi/zdaemon>`_ or `supervisord
<http://supervisord.org/>`_.
Rotating log files
==================
``runzeo`` will re-initialize its logging subsystem when it receives a
SIGUSR2 signal. If you are using the standard event logger, you
should first rename the log file and then send the signal to the
server. The server will continue writing to the renamed log file
until it receives the signal. After it receives the signal, the
server will create a new file with the old name and write to it.
doc/zeo-client-cache.txt deleted 100644 → 0
View file @ 881ef452
ZEO Client Cache
The client cache provides a disk based cache for each ZEO client. The
client cache allows reads to be done from local disk rather than by remote
access to the storage server.
The cache may be persistent or transient. If the cache is persistent, then
the cache file is retained for use after process restarts. A non-
persistent cache uses a temporary file.
The client cache is managed in a single file, of the specified size.
The life of the cache is as follows:
- The cache file is opened (if it already exists), or created and set to
the specified size.
- Cache records are written to the cache file, as transactions commit
locally, and as data are loaded from the server.
- Writes are to "the current file position". This is a pointer that
travels around the file, circularly. After a record is written, the
pointer advances to just beyond it. Objects starting at the current
file position are evicted, as needed, to make room for the next record
written.
A distinct index file is not created, although indexing structures are
maintained in memory while a ClientStorage is running. When a persistent
client cache file is reopened, these indexing structures are recreated
by analyzing the file contents.
Persistent cache files are created in the directory named in the ``var``
argument to the ClientStorage, or if ``var`` is None, in the current
working directory. Persistent cache files have names of the form::
client-storage.zec
where:
client -- the client name, as given by the ClientStorage's ``client``
argument
storage -- the storage name, as given by the ClientStorage's ``storage``
argument; this is typically a string denoting a small integer,
"1" by default
For example, the cache file for client '8881' and storage 'spam' is named
"8881-spam.zec".
setup.py
View file @ 32c6a136
...@@ -141,6 +141,11 @@ setup(name="ZEO", ...@@ -141,6 +141,11 @@ setup(name="ZEO",
'futures', 'futures',
'trollius', 'trollius',
], ],
'docs': [
'Sphinx',
'repoze.sphinx.autointerface',
'sphinx_rtd_theme',
],
}, },
install_requires=install_requires, install_requires=install_requires,
zip_safe=False, zip_safe=False,
......
src/ZEO/__init__.py
View file @ 32c6a136
...@@ -22,10 +22,16 @@ ZEO is now part of ZODB; ZODB's home on the web is ...@@ -22,10 +22,16 @@ ZEO is now part of ZODB; ZODB's home on the web is
""" """
def client(*args, **kw): def client(*args, **kw):
"""
Shortcut for :class:`ZEO.ClientStorage.ClientStorage`.
"""
import ZEO.ClientStorage import ZEO.ClientStorage
return ZEO.ClientStorage.ClientStorage(*args, **kw) return ZEO.ClientStorage.ClientStorage(*args, **kw)
def DB(*args, **kw): def DB(*args, **kw):
"""
Shortcut for creating a :class:`ZODB.DB` using a ZEO :func:`~ZEO.client`.
"""
s = client(*args, **kw) s = client(*args, **kw)
try: try:
import ZODB import ZODB
......
src/ZEO/nagios.rst
View file @ 32c6a136
=================
ZEO Nagios plugin ZEO Nagios plugin
================= =================
...@@ -33,7 +34,7 @@ If we stop the server, we'll error: ...@@ -33,7 +34,7 @@ If we stop the server, we'll error:
2 2
Metrics Metrics
------- =======
The monitor will optionally output server metric data. There are 2 The monitor will optionally output server metric data. There are 2
kinds of metrics it can output, level and rate metric. If we use the kinds of metrics it can output, level and rate metric. If we use the
...@@ -103,7 +104,7 @@ other monitors (e.g. that get address data from ZooKeeper). It takes: ...@@ -103,7 +104,7 @@ other monitors (e.g. that get address data from ZooKeeper). It takes:
>>> stop() >>> stop()
Multi-storage servers Multi-storage servers
--------------------- =====================
A ZEO server can host multiple servers. (This is a feature that will A ZEO server can host multiple servers. (This is a feature that will
likely be dropped in the future.) When this is the case, the monitor likely be dropped in the future.) When this is the case, the monitor
......
tox.ini
View file @ 32c6a136
[tox] [tox]
envlist = envlist =
py27,py34,py35,py36,py37,pypy,pypy3,simple py27,py34,py35,py36,py37,pypy,pypy3,simple,docs
[testenv] [testenv]
commands = commands =
...@@ -31,3 +31,10 @@ basepython = ...@@ -31,3 +31,10 @@ basepython =
commands = commands =
python setup.py -q test -q python setup.py -q test -q
deps = {[testenv]deps} deps = {[testenv]deps}
[testenv:docs]
basepython =
python3.8
commands =
sphinx-build -b html doc doc/_build/html
extras =
docs
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7