Commit 224433cd by Julien Muchembled

Update documentation

1 parent 2c170443
Although NEO is considered ready for production use in most cases, there are
a few bugs to know because they concern basic features of ZODB, or promised
features of NEO.
All the listed bugs will be fixed with high priority.
Conflict resolution not fully implemented
-----------------------------------------
When a cluster has several storage nodes, so-called 'deadlock avoidance' may
happen to in order to resolve conflicts. In such cases, conflicts will not be
resolved even if your _p_resolveConflict() method would succeed, leading to a
normal ConflictError.
Although this should happen rarely enough not to affect performance, this can
be an issue if your application can't afford restarting the transaction,
e.g. because it interacted with external environment.
Master failure not recovered by clients
---------------------------------------
After a master failure, clients are stuck in a bad state. In particular, caches
are not flushed, which could lead to data corruption if they reconnect a master.
This makes secondary masters mostly useless for the moment.
Storage failure or update may lead to POSException or break undoLog()
---------------------------------------------------------------------
Storage nodes are only queried once at most and if all (for the requested
partition) failed, the client raises instead of asking the master whether it
had an up-to-date partition table (and retry if useful).
In the case of undoLog(), incomplete results may be returned.
......@@ -14,9 +14,9 @@ A NEO cluster is composed of the following types of nodes:
- "storage" nodes (mandatory, 1 or more)
Stores data in a MySQL database, preserving history. All available storage
nodes are in use simultaneously. This offers redundancy and data distribution.
Other storage backends than MySQL are considered for future release.
Stores data, preserving history. All available storage nodes are in use
simultaneously. This offers redundancy and data distribution.
Available backends: MySQL, SQLite
- "admin" nodes (mandatory for startup, optional after)
......@@ -56,29 +56,43 @@ Requirements
- Python 2.6.x or 2.7.x
- For storage nodes:
- For storage nodes using MySQL backend:
- MySQLdb: http://sourceforge.net/projects/mysql-python
- For client nodes: ZODB 3.10.x but it should work with ZODB 3.9.x
- For client nodes: ZODB 3.9.x or later
Installation
============
a. Make neo directory available for python to import (for example, by
a. NEO can be installed like any other egg (see setup.py). Or you can simply
make `neo` directory available for Python to import (for example, by
adding its container directory to the PYTHONPATH environment variable).
b. Choose a cluster name and setup a MySQL database
b. Write a neo.conf file like the example provided. If you use MySQL,
you'll also need create 1 database per storage node.
c. Start all required nodes::
neomaster --cluster=<cluster name>
neostorage --cluster=<cluster name> --database=user:passwd@db
neoadmin --cluster=<cluster name>
$ neomaster -f neo.conf
$ neostorage -f neo.conf -s storage1
$ neostorage -f neo.conf -s storage2
$ neoadmin -f neo.conf
d. Tell the cluster it can provide service::
d. Tell the cluster to initialize storage nodes::
neoctl -a <admin> start
$ neoctl -a <admin> start
e. Clients can connect when the cluster is in RUNNING state::
$ neoctl -a <admin> print cluster
RUNNING
f. See `neomigrate` command to import an existing FileStorage database,
or `neoctl` command for more administrative tasks.
Alternatively, you can use `neosimple` command to quickly setup a cluster for
testing.
How to use
==========
......@@ -134,6 +148,23 @@ This can also be done manually, which helps if your cluster is in bad state:
don't become in OUT_OF_DATE state.
- Next stop remaining nodes with a SIGINT or SIGTERM.
Master-slave asynchronous replication
-------------------------------------
This is the recommanded way to backup a NEO cluster.
Once a cluster with appropriate `upstream_cluster` & `upstream_masters`
configuration is started, you can switch it into backup mode
using::
neoctl -a <admin> set cluster STARTING_BACKUP
It remembers it is in such mode when it is stopped, and it can be put back into
normal mode (RUNNING) by setting it into STOPPING_BACKUP state.
Packs are currently not replicated, which means packing should always be done
up to a TID that is already fully replicated, so that the backup cluster has a
full history (and not random holes).
Deployment
==========
......@@ -145,15 +176,15 @@ NEO has no built-in deployment features such as process daemonization. We use
[program:storage_01]
priority=10
command=neostorage -c neo -s storage_01 -f /neo/neo.conf
command=neostorage -s storage_01 -f /neo/neo.conf
[program:master_01]
priority=20
command=neomaster -c neo -s master_01 -f /neo/neo.conf
command=neomaster -s master_01 -f /neo/neo.conf
[program:admin]
priority=20
command=neoadmin -c neo -s admin -f /neo/neo.conf
command=neoadmin -s admin -f /neo/neo.conf
Developers
==========
......
# Note: Unless otherwise noted, all parameters in this configuration file
# must be identical for all nodes in a given cluster.
# This file is optionnal: parameters can be given at the command line.
# Default parameters.
# Common parameters.
[DEFAULT]
# The cluster name
# This must be set.
# It must be a name unique to a given cluster, to prevent foreign
# IMPORTANT: It must be a name unique to a given cluster, to prevent foreign
# misconfigured nodes from interfering.
cluster:
cluster: test
# The list of master nodes
# Master nodes not not in this list will be rejected by the cluster.
......@@ -22,18 +23,25 @@ masters: 127.0.0.1:10000
# 0 means no redundancy
# 1 means there is a spare copy of all partitions
replicas: 1
# Partitions: How data spreads among storage nodes. This number must be at
# least equal to the number of storage nodes the cluster contains.
# IMPORTANT: This must not be changed once the cluster contains data.
partitions: 20
# Partitions: How data spreads among storage nodes.
# IMPORTANT: This can not be changed once the cluster contains data.
partitions: 12
# The maximum number of usable storage nodes is: partitions * (replicas + 1)
# Master-slave asynchronous replication
# The following 2 options are only required if you want to backup another
# NEO cluster.
;upstream_cluster: test2
;upstream_masters: 127.0.0.1:30000
# Individual nodes parameters
# Some parameters makes no sense to be defined in [DEFAULT] section.
# They are:
# bind: The ip:port the node will listen on.
# database: Storage nodes only.
# Syntax for MySQL database is [user[:password]@]database[unix_socket]
# Those database must be created manualy.
# database: Storage nodes only. Syntax for:
# - MySQL: [user[:password]@]database[unix_socket]
# Database must be created manually.
# - SQLite: path
# Admin node
[admin]
......
......@@ -195,7 +195,7 @@ class TerminalNeoCTL(object):
def checkReplicas(self, params):
"""
Test whether partitions have corrupted meta
Test whether partitions have corrupted metadata
Any corrupted cell is put in CORRUPTED state, possibly make the
cluster non operational.
......
Styling with Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!