Commit 224433cd authored by Julien Muchembled's avatar Julien Muchembled

Update documentation

parent 2c170443
Although NEO is considered ready for production use in most cases, there are
a few bugs to know because they concern basic features of ZODB, or promised
features of NEO.
All the listed bugs will be fixed with high priority.
Conflict resolution not fully implemented
-----------------------------------------
When a cluster has several storage nodes, so-called 'deadlock avoidance' may
happen to in order to resolve conflicts. In such cases, conflicts will not be
resolved even if your _p_resolveConflict() method would succeed, leading to a
normal ConflictError.
Although this should happen rarely enough not to affect performance, this can
be an issue if your application can't afford restarting the transaction,
e.g. because it interacted with external environment.
Master failure not recovered by clients
---------------------------------------
After a master failure, clients are stuck in a bad state. In particular, caches
are not flushed, which could lead to data corruption if they reconnect a master.
This makes secondary masters mostly useless for the moment.
Storage failure or update may lead to POSException or break undoLog()
---------------------------------------------------------------------
Storage nodes are only queried once at most and if all (for the requested
partition) failed, the client raises instead of asking the master whether it
had an up-to-date partition table (and retry if useful).
In the case of undoLog(), incomplete results may be returned.
...@@ -14,9 +14,9 @@ A NEO cluster is composed of the following types of nodes: ...@@ -14,9 +14,9 @@ A NEO cluster is composed of the following types of nodes:
- "storage" nodes (mandatory, 1 or more) - "storage" nodes (mandatory, 1 or more)
Stores data in a MySQL database, preserving history. All available storage Stores data, preserving history. All available storage nodes are in use
nodes are in use simultaneously. This offers redundancy and data distribution. simultaneously. This offers redundancy and data distribution.
Other storage backends than MySQL are considered for future release. Available backends: MySQL, SQLite
- "admin" nodes (mandatory for startup, optional after) - "admin" nodes (mandatory for startup, optional after)
...@@ -56,29 +56,43 @@ Requirements ...@@ -56,29 +56,43 @@ Requirements
- Python 2.6.x or 2.7.x - Python 2.6.x or 2.7.x
- For storage nodes: - For storage nodes using MySQL backend:
- MySQLdb: http://sourceforge.net/projects/mysql-python - MySQLdb: http://sourceforge.net/projects/mysql-python
- For client nodes: ZODB 3.10.x but it should work with ZODB 3.9.x - For client nodes: ZODB 3.9.x or later
Installation Installation
============ ============
a. Make neo directory available for python to import (for example, by a. NEO can be installed like any other egg (see setup.py). Or you can simply
make `neo` directory available for Python to import (for example, by
adding its container directory to the PYTHONPATH environment variable). adding its container directory to the PYTHONPATH environment variable).
b. Choose a cluster name and setup a MySQL database b. Write a neo.conf file like the example provided. If you use MySQL,
you'll also need create 1 database per storage node.
c. Start all required nodes:: c. Start all required nodes::
neomaster --cluster=<cluster name> $ neomaster -f neo.conf
neostorage --cluster=<cluster name> --database=user:passwd@db $ neostorage -f neo.conf -s storage1
neoadmin --cluster=<cluster name> $ neostorage -f neo.conf -s storage2
$ neoadmin -f neo.conf
d. Tell the cluster it can provide service:: d. Tell the cluster to initialize storage nodes::
neoctl -a <admin> start $ neoctl -a <admin> start
e. Clients can connect when the cluster is in RUNNING state::
$ neoctl -a <admin> print cluster
RUNNING
f. See `neomigrate` command to import an existing FileStorage database,
or `neoctl` command for more administrative tasks.
Alternatively, you can use `neosimple` command to quickly setup a cluster for
testing.
How to use How to use
========== ==========
...@@ -134,6 +148,23 @@ This can also be done manually, which helps if your cluster is in bad state: ...@@ -134,6 +148,23 @@ This can also be done manually, which helps if your cluster is in bad state:
don't become in OUT_OF_DATE state. don't become in OUT_OF_DATE state.
- Next stop remaining nodes with a SIGINT or SIGTERM. - Next stop remaining nodes with a SIGINT or SIGTERM.
Master-slave asynchronous replication
-------------------------------------
This is the recommanded way to backup a NEO cluster.
Once a cluster with appropriate `upstream_cluster` & `upstream_masters`
configuration is started, you can switch it into backup mode
using::
neoctl -a <admin> set cluster STARTING_BACKUP
It remembers it is in such mode when it is stopped, and it can be put back into
normal mode (RUNNING) by setting it into STOPPING_BACKUP state.
Packs are currently not replicated, which means packing should always be done
up to a TID that is already fully replicated, so that the backup cluster has a
full history (and not random holes).
Deployment Deployment
========== ==========
...@@ -145,15 +176,15 @@ NEO has no built-in deployment features such as process daemonization. We use ...@@ -145,15 +176,15 @@ NEO has no built-in deployment features such as process daemonization. We use
[program:storage_01] [program:storage_01]
priority=10 priority=10
command=neostorage -c neo -s storage_01 -f /neo/neo.conf command=neostorage -s storage_01 -f /neo/neo.conf
[program:master_01] [program:master_01]
priority=20 priority=20
command=neomaster -c neo -s master_01 -f /neo/neo.conf command=neomaster -s master_01 -f /neo/neo.conf
[program:admin] [program:admin]
priority=20 priority=20
command=neoadmin -c neo -s admin -f /neo/neo.conf command=neoadmin -s admin -f /neo/neo.conf
Developers Developers
========== ==========
......
# Note: Unless otherwise noted, all parameters in this configuration file # Note: Unless otherwise noted, all parameters in this configuration file
# must be identical for all nodes in a given cluster. # must be identical for all nodes in a given cluster.
# This file is optionnal: parameters can be given at the command line.
# Default parameters. # Common parameters.
[DEFAULT] [DEFAULT]
# The cluster name # The cluster name
# This must be set. # This must be set.
# It must be a name unique to a given cluster, to prevent foreign # IMPORTANT: It must be a name unique to a given cluster, to prevent foreign
# misconfigured nodes from interfering. # misconfigured nodes from interfering.
cluster: cluster: test
# The list of master nodes # The list of master nodes
# Master nodes not not in this list will be rejected by the cluster. # Master nodes not not in this list will be rejected by the cluster.
...@@ -22,18 +23,25 @@ masters: 127.0.0.1:10000 ...@@ -22,18 +23,25 @@ masters: 127.0.0.1:10000
# 0 means no redundancy # 0 means no redundancy
# 1 means there is a spare copy of all partitions # 1 means there is a spare copy of all partitions
replicas: 1 replicas: 1
# Partitions: How data spreads among storage nodes. This number must be at # Partitions: How data spreads among storage nodes.
# least equal to the number of storage nodes the cluster contains. # IMPORTANT: This can not be changed once the cluster contains data.
# IMPORTANT: This must not be changed once the cluster contains data. partitions: 12
partitions: 20 # The maximum number of usable storage nodes is: partitions * (replicas + 1)
# Master-slave asynchronous replication
# The following 2 options are only required if you want to backup another
# NEO cluster.
;upstream_cluster: test2
;upstream_masters: 127.0.0.1:30000
# Individual nodes parameters # Individual nodes parameters
# Some parameters makes no sense to be defined in [DEFAULT] section. # Some parameters makes no sense to be defined in [DEFAULT] section.
# They are: # They are:
# bind: The ip:port the node will listen on. # bind: The ip:port the node will listen on.
# database: Storage nodes only. # database: Storage nodes only. Syntax for:
# Syntax for MySQL database is [user[:password]@]database[unix_socket] # - MySQL: [user[:password]@]database[unix_socket]
# Those database must be created manualy. # Database must be created manually.
# - SQLite: path
# Admin node # Admin node
[admin] [admin]
......
...@@ -195,7 +195,7 @@ class TerminalNeoCTL(object): ...@@ -195,7 +195,7 @@ class TerminalNeoCTL(object):
def checkReplicas(self, params): def checkReplicas(self, params):
""" """
Test whether partitions have corrupted meta Test whether partitions have corrupted metadata
Any corrupted cell is put in CORRUPTED state, possibly make the Any corrupted cell is put in CORRUPTED state, possibly make the
cluster non operational. cluster non operational.
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment