Commit 2e27c167 authored by Jim Fulton's avatar Jim Fulton Committed by GitHub

Merge pull request #24 from zopefoundation/setup-and-reference

Setup and reference
parents 73248f68 edbf0628
[buildout]
develop = .
develop = . ../zodb
parts =
stxpy
test
......@@ -13,6 +13,10 @@ recipe = zc.recipe.egg
eggs =
Sphinx
docutils
ZODB
j1m.sphinxautointerface
j1m.sphinxautozconfig
interpreter = stxpy
scripts =
sphinx-build
......
......@@ -24,7 +24,9 @@
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = []
extensions = ['sphinx.ext.autodoc',
'j1m.sphinxautointerface',
'j1m.sphinxautozconfig']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['.templates']
......
......@@ -12,7 +12,11 @@ If you haven't yet, you should read the :ref:`Tutorial <tutorial-label>`.
.. toctree::
:maxdepth: 2
install-and-run
writing-persistent-objects.rst
.. todo:
transaction.rst
storages.rst
configuration.rst
......
==========================
Installing and running ZODB
===========================
This topic discusses some boring nitty-gritty details needed to
actually run ZODB.
Installation
============
Installation of ZODB is pretty straightforward using Python's
packaging system. For example, using pip::
pip install ZODB
You may need additional optional packages, such as `ZEO
<https://pypi.python.org/pypi/ZEO>`_ or `RelStorage
<https://pypi.python.org/pypi/RelStorage>`_, depending your deployment
choices.
Configuration
=============
You can set up ZODB in your application using either Python, or
ZODB's configuration language. For simple database setup, and
especially for exploration, the Python APIs are sufficient.
For more complex configurations, you'll probably find ZODB's
configuration language easier to use.
To understand database setup, it's important to understand ZODB's
architecture. ZODB separates database functionality
from storage concerns. When you create a database object,
you specify a storage object for it to use, as in::
import ZODB, ZODB.FileStorage
storage = ZODB.FileStorage.FileStorage('mydata.fs')
db = ZODB.DB(storage)
So when you define a database, you'll also define a storage. In the
example above, we define a :class:`file storage
<ZODB.FileStorage.FileStorage.FileStorage>` and then use it to define
a database.
Sometimes, storages are created through composition. For example, if
we want to save space, we could layer a ``ZlibStorage``
[#zlibstoragefn]_ over the file storage::
import ZODB, ZODB.FileStorage, zc.zlibstorage
storage = ZODB.FileStorage.FileStorage('mydata.fs')
compressed_storage = zc.zlibstorage.ZlibStorage(storage)
db = ZODB.DB(compressed_storage)
`ZlibStorage <https://pypi.python.org/pypi/zc.zlibstorage>`_
compresses database records [#zlib]_.
Python configuration
--------------------
To set up a database with Python, you'll construct a storage using the
:ref:`storage APIs <included-storages-label>`, and then pass the
storage to the :class:`~ZODB.DB` class to create a database, as shown
in the examples in the previous section.
The :class:`~ZODB.DB` class also accepts a string path name as its
storage argument to automatically create a file storage. You can also
pass ``None`` as the storage to automatically use a
:class:`~ZODB.MappingStorage.MappingStorage`, which is convenient when
exploring ZODB::
db = ZODB.DB(None) # Create an in-memory database.
Text configuration
------------------
ZODB supports a text-based configuration language. It uses a syntax
similar to Apache configuration files. The syntax was chosen to be
familiar to site administrators.
ZODB's text configuration uses `ZConfig
<https://pypi.python.org/pypi/ZConfig>`_. You can use ZConfig to
create your application's configuration, but it's more common to
include ZODB configuration strings in their own files or embedded in
simpler configuration files, such as `configarser
<https://docs.python.org/3/library/configparser.html#module-configparser>`_
files.
A database configuration string has a ``zodb`` section wrapping a
storage section, as in::
<zodb>
cache-size-bytes 100MB
<mappingstorage>
</mappingstorage>
</zodb>
.. -> snippet
In the example above, the :ref:`mappingstorage
<mappingstorage-text-configuration>` section defines the storage used
by the database.
To create a database from a string, use
:func:`ZODB.config.databaseFromString`::
>>> import ZODB.config
>>> db = ZODB.config.databaseFromString(snippet)
To load databases from file names or URLs, use
:func:`ZODB.config.databaseFromURL`.
URI-based configuration
-----------------------
Another database configuration option is provided by the `zodburi
<https://pypi.python.org/pypi/zodburi>`_ package. See:
http://docs.pylonsproject.org/projects/zodburi. It's less powerful
than the Python or text configuration options, but allows
configuration to be reduced to a single URI and handles most cases.
Using databases: connections
============================
Once you have a database, you need to get a database connection to to
much of anything. Connections take care of loading and saving objects
and manage object caches. Each connection has it's own cache
[#caches-are-expensive]_.
Getting connections
-------------------
Amongst [#amongst]_ the common ways of getting a connection:
db.open()
The database :meth:`~ZODB.DB.open` method opens a
connection, returning a connection object::
>>> conn = db.open()
It's up to the application to call
:meth:`~ZODB.Connection.Connection.close` when the application is
done using the connection.
If changes are made, the application :ref:`commits transactions
<commit-transactions>` to make them permanent.
db.transaction()
The database :meth:`~ZODB.DB.transaction` method
returns a context manager that can be used with the `python with
statement
<https://docs.python.org/3/reference/compound_stmts.html#grammar-token-with_stmt>`_
to execute a block of code in a transaction::
with db.transaction() as connection:
connection.root.foo = 1
.. -> src
>>> exec(src)
>>> with db.transaction() as connection:
... print connection.root.foo
1
>>> _ = conn.transaction_manager.begin() # get updates on conn
In the example above, we used ``as connection`` to get the database
connection used in the variable ``connection``.
some_object._p_jar
For code that's already running in the context of an open
connection, you can get the current connection as the ``_p_jar``
attribute of some persistent object that was accessed via the
connection.
Getting objects
---------------
Once you have a connection, you access objects by traversing the
object graph from the root object.
The database root object is a mapping object that holds the top level
objects in the database. There should only be a small number of
top-level objects (often only one). You can get the root object by calling a
connection's ``root`` attribute::
>>> root = conn.root()
>>> root
{'foo': 1}
>>> root['foo']
1
For convenience [#root-convenience]_, you can also get top-level
objects by accessing attributes of the connection root object:
>>> conn.root.foo
1
Once you have a top-level object, you use its methods, attributes, or
operations to access other objects and so on to get the objects you
need. Often indexing data structures like BTrees_ are used to
make it possible to search objects in large collections.
.. [#zlibstoragefn] `zc.zlibstorage
<https://pypi.python.org/pypi/zc.zlibstorage>`_ is an optional
package that you need to install separately.
.. [#zlib] ZlibStorage uses the :mod:`zlib` standard module, which
uses the `zlib library <http://www.zlib.net/>`_.
.. [#caches-are-expensive] ZODB can be very efficient at caching data
in memory, especially if your `working set
<https://en.wikipedia.org/wiki/Working_set>`_ is small enough to
fit in memory, because the cache is simply an object tree and
accessing a cached object typically requires no database
interaction. Because each connection has its own cache,
connections can be expensive, depending on their cache sizes. For
this reason, you'll generally want to limit the number of open
connections you have at any one time. Connections are pooled, so
opening a connection is inexpensive.
.. [#amongst] https://www.youtube.com/watch?v=7WJXHY2OXGE
.. [#root-convenience] The ability to access top-level objects of the
database as root attributes is a recent convenience. Originally,
the ``root()`` method was used to access the root object which was
then accessed as a mapping. It's still potentially useful to
access top-level objects using the mapping interface if their names
aren't valid attribute names.
.. _BTrees: https://pythonhosted.org/BTrees/
......@@ -157,7 +157,7 @@ record from the book record and is managed by ZODB independent of the
management of the book.
In addition to ``PersistentList`` and ``PersistentMapping``, general
persistent data structures are provided by the ``BTrees`` package,
persistent data structures are provided by the BTrees_ package,
most notably ``BTree`` and ``TreeSet`` objects. Unlike
``PersistentList`` and ``PersistentMapping``, ``BTree`` and
``TreeSet`` objects are scalable and can easily hold millions of
......@@ -566,3 +566,5 @@ framework for managing schema-migration scripts.
<https://pypi.python.org/pypi/zope.cachedescriptors>`_ package
provides some descriptors that help implement attributes that cache
data.
.. _BTrees: https://pythonhosted.org/BTrees/
=======================
Reference Documentation
=======================
.. toctree::
:maxdepth: 2
zodb.rst
storages.rst
=============
Storage APIs
=============
.. contents::
Storage interfaces
==================
There are various storage implementations that implement standard
storage interfaces. Thet differ primarily in their constructors.
Application code rarely calls storage methods, and those it calls are
generally called indirectly through databases. There are
interface-defined methods that are called internally by ZODB. These
aren't shown below.
IStorage
--------
.. autointerface:: ZODB.interfaces.IStorage
:members: close, getName, getSize, history, isReadOnly, lastTransaction,
__len__, pack, sortKey
IStorageIteration
-----------------
.. autointerface:: ZODB.interfaces.IStorageIteration
IStorageUndoable
----------------
.. autointerface:: ZODB.interfaces.IStorageUndoable
:members: undoLog, undoInfo
IStorageCurrentRecordIteration
------------------------------
.. autointerface:: ZODB.interfaces.IStorageCurrentRecordIteration
IBlobStorage
------------
.. autointerface:: ZODB.interfaces.IBlobStorage
:members: temporaryDirectory
IStorageRecordInformation
-------------------------
.. autointerface:: ZODB.interfaces.IStorageRecordInformation
IStorageTransactionInformation
------------------------------
.. autointerface:: ZODB.interfaces.IStorageTransactionInformation
.. _included-storages-label:
Included storages
=================
FileStorage
-----------
.. autoclass:: ZODB.FileStorage.FileStorage.FileStorage
:members: __init__
.. autointerface:: ZODB.FileStorage.interfaces.IFileStoragePacker
FileStorage text configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
File storages are configured using the ``filestorage`` section::
<filestorage>
path Data.fs
</filestorage>
which accepts the following options:
.. zconfigsectionkeys:: ZODB component.xml filestorage
MappingStorage
--------------
.. autoclass:: ZODB.MappingStorage.MappingStorage
:members: __init__
.. _mappingstorage-text-configuration:
MappingStorage text configuration
---------------------------------
File storages are configured using the ``mappingstorage`` section::
<mappingstorage>
</mappingstorage>
Options:
.. zconfigsectionkeys:: ZODB component.xml mappingstorage
DemoStorage
-----------
.. autoclass:: ZODB.DemoStorage.DemoStorage
:members: __init__, push, pop
DemoStorage text configuration
------------------------------
Demo storages are configured using the ``demostorage`` section::
<demostorage>
<filestorage base>
path base.fs
</filestorage>
<mappingstorage changes>
name Changes
</mappingstorage>
</demostorage>
.. -> src
>>> import ZODB.config
>>> storage = ZODB.config.storageFromString(src)
>>> storage.base.getName()
'base.fs'
>>> storage.changes.getName()
'Changes'
``demostorage`` sections can contain up to 2 storage subsections,
named ``base`` and ``changes``, specifying the demo storage's base and
changes storages. See :meth:`ZODB.DemoStorage.DemoStorage.__init__`
for more on the base anc changes storages.
Options:
.. zconfigsectionkeys:: ZODB component.xml demostorage
Noteworthy non-included storages
================================
A number of important ZODB storages are distriubuted separately, including:
RelStorage
`RelStorage <http://relstorage.readthedocs.io/en/latest/>`_
stores data in relational databases. This is especially
useful when you have requirements or existing infrastructure for
storing data in relational databases. Unlike the included storages,
multiple processes can share the same database.
For more imformation, see http://relstorage.readthedocs.io/en/latest/.
ZEO
`ZEO <https://github.com/zopefoundation/ZEO>`_ is a client-server
database implementation for ZODB. To use ZEO, you run a ZEO server,
and use ZEO clients in your application. Unlike the included
storages, multiple processes can share the same database.
For more imformation, see https://github.com/zopefoundation/ZEO.
ZRS
`ZRS <https://github.com/zc/zrs>`_
provides replication from one database to another. It's most
commonly used with ZEO. With ZRS, you create a ZRS primary database
around a :class:`~ZODB.FileStorage.FileStorage.FileStorage` and in a
separate process, you creatre a ZRS secondary storage around any
:interface:`storage <ZODB.interfaces.IStorage>`. As transactions are
committed on the primary, they're copied asynchronously to
secondaries.
For more imformation, see https://github.com/zc/zrs.
zlibstorage
`zlibstorage <https://pypi.python.org/pypi/zc.zlibstorage>`_
compresses database records using the compression
algorithm used by `gzip <http://www.gzip.org/>`_.
For more imformation, see https://pypi.python.org/pypi/zc.zlibstorage.
beforestorage
`beforestorage <https://pypi.python.org/pypi/zc.beforestorage>`_
provides a point-in-time view of a database that might
be changing. This can be useful to provide a non-changing view of a
production database for use with a :class:`~ZODB.DemoStorage.DemoStorage`.
For more imformation, see https://pypi.python.org/pypi/zc.beforestorage.
cipher.encryptingstorage
`cipher.encryptingstorage
<https://pypi.python.org/pypi/cipher.encryptingstorage/>`_ provided
compression and encryption of database records.
For more informayion see,
https://pypi.python.org/pypi/cipher.encryptingstorage/.
=========
ZODB APIs
=========
.. contents::
ZODB module functions
=====================
.. method:: DB(storage, *args, **kw)
Create a databse. See :py:class:`ZODB.DB`.
.. autofunction:: ZODB.connection
Databases
=========
.. autoclass:: ZODB.DB
:members: __init__, open, close, pack,
cacheDetail, cacheExtremeDetail, cacheMinimize,
cacheSize, cacheDetailSize, getCacheSize, getCacheSizeBytes,
lastTransaction, getName, getPoolSize, getSize,
getHistoricalCacheSize, getHistoricalCacheSizeBytes,
getHistoricalPoolSize, getHistoricalTimeout,
objectCount, connectionDebugInfo,
setCacheSize, setCacheSizeBytes,
setHistoricalCacheSize, setHistoricalCacheSizeBytes,
setPoolSize, setHistoricalPoolSize, setHistoricalTimeout,
history,
supportsUndo, undoLog, undoInfo, undoMultiple, undo,
transaction, storage
.. _database-text-configuration:
Database text configuration
---------------------------
Databases are configured with ``zodb`` sections::
<zodb>
cache-size-bytes 100MB
<mappingstorage>
</mappingstorage>
</zodb>
A ``zodb`` section must have a storage sub-section specifying a
storage and any of the following options:
.. zconfigsectionkeys:: ZODB component.xml zodb
.. _multidatabase-text-configuration:
For a multi-database configuration, use multiple ``zodb`` sections and
give the sections names::
<zodb first>
cache-size-bytes 100MB
<mappingstorage>
</mappingstorage>
</zodb>
<zodb second>
<mappingstorage>
</mappingstorage>
</zodb>
.. -> src
>>> import ZODB.config
>>> db = ZODB.config.databaseFromString(src)
>>> sorted(db.databases)
['first', 'second']
>>> db._cache_size_bytes
104857600
When the configuration is loaded, a single database will be returned,
but all of the databases will be available through the returned
database's ``databases`` attribute.
Connections
===========
.. autoclass:: ZODB.Connection.Connection
:members: add, cacheGC, cacheMinimize, close, db, get,
getDebugInfo, get_connection, isReadOnly, oldstate,
onCloseCallback, root, setDebugInfo, sync
TimeStamp (transaction ids)
===========================
.. class:: ZODB.TimeStamp.TimeStamp(year, month, day, hour, minute, seconds)
Create a time-stamp object. Time stamps facilitate the computation
of transaction ids, which are based on times. The arguments are
integers, except for seconds, which may be a floating-point
number. Time stamps have microsecond precision. Time stamps are
implicitly UTC based.
Time stamps are orderable and hashable.
.. method:: day()
Return the time stamp's day.
.. method:: hour()
Return the time stamp's hour.
.. method:: laterThan(other)
Return a timestamp instance which is later than 'other'.
If self already qualifies, return self.
Otherwise, return a new instance one moment later than 'other'.
.. method:: minute()
Return the time stamp's minute.
.. method:: month()
Return the time stamp's month.
.. method:: raw()
Get an 8-byte representatin of the time stamp for use in APIs
that require a time stamp.
.. method:: second()
Return the time stamp's second.
.. method:: timeTime()
Return the time stamp as seconds since the epoc, as used by the
``time`` module.
.. method:: year()
Return the time stamp's year.
Loading configuration
=====================
.. automodule:: ZODB.config
:members: databaseFromString, databaseFromFile, databaseFromURL,
storageFromString, storageFromFile, storageFromURL
......@@ -198,6 +198,7 @@ Learning more
documentation/tutorial
documentation/guide/index
documentation/reference/index
documentation/articles/index
* `The ZODB Book (in progress) <http://zodb.readthedocs.org/en/latest/>`_
......
......@@ -34,11 +34,15 @@ def tearDown(test):
def test_suite():
here = os.path.dirname(__file__)
guide = join(here, '..', 'documentation', 'guide')
reference = join(here, '..', 'documentation', 'reference')
return unittest.TestSuite((
manuel.testing.TestSuite(
manuel.doctest.Manuel() + manuel.capture.Manuel(),
join(guide, 'writing-persistent-objects.rst'),
join(guide, 'install-and-run.rst'),
join(reference, 'zodb.rst'),
join(reference, 'storages.rst'),
setUp=setUp, tearDown=tearDown,
),
))
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment