Blame view

Vincent Pelletier committed
1 2
NEO is a distributed, redundant and scalable implementation of ZODB API.
NEO stands for Nexedi Enterprise Object.
Yoshinori Okuji committed
3

Julien Muchembled committed
4 5
Overview
========
Yoshinori Okuji committed
6

Julien Muchembled committed
7
A NEO cluster is composed of the following types of nodes:
Vincent Pelletier committed
8

Julien Muchembled committed
9
- "master" nodes (mandatory, 1 or more)
Yoshinori Okuji committed
10

Julien Muchembled committed
11 12 13
  Takes care of transactionality. Only one master node is really active
  (the active master node is called "primary master") at any given time,
  extra masters are spares (they are called "secondary masters").
Yoshinori Okuji committed
14

Julien Muchembled committed
15
- "storage" nodes (mandatory, 1 or more)
Yoshinori Okuji committed
16

Julien Muchembled committed
17 18 19
  Stores data, preserving history. All available storage nodes are in use
  simultaneously. This offers redundancy and data distribution.
  Available backends: MySQL, SQLite
Yoshinori Okuji committed
20

Julien Muchembled committed
21
- "admin" nodes (mandatory for startup, optional after)
Vincent Pelletier committed
22

Julien Muchembled committed
23 24
  Accepts commands from neoctl tool and transmits them to the
  primary master, and monitors cluster state.
Vincent Pelletier committed
25

Julien Muchembled committed
26
- "client" nodes
Vincent Pelletier committed
27

Julien Muchembled committed
28
  Well... Something needing to store/load data in a NEO cluster.
Vincent Pelletier committed
29

Julien Muchembled committed
30 31 32 33 34 35
ZODB API is fully implemented except:

- pack: only old revisions of objects are removed for the moment
        (full implementation is considered)
- blobs: not implemented (not considered yet)

Julien Muchembled committed
36 37 38
Any ZODB like FileStorage can be converted to NEO instanteously,
which means the database is operational before all data are imported.
There's also a tool to convert back to FileStorage.
Julien Muchembled committed
39 40 41 42

See also http://www.neoppod.org/links for more detailed information about
features related to scalability.

Julien Muchembled committed
43 44
Disclaimer
==========
Vincent Pelletier committed
45

Julien Muchembled committed
46 47
In addition of the disclaimer contained in the licence this code is
released under, please consider the following.
Vincent Pelletier committed
48

Julien Muchembled committed
49 50 51 52
NEO does not implement any authentication mechanism between its nodes, and
does not encrypt data exchanged between nodes either.
If you want to protect your cluster from malicious nodes, or your data from
being snooped, please consider encrypted tunelling (such as openvpn).
Vincent Pelletier committed
53

Julien Muchembled committed
54 55
Requirements
============
Vincent Pelletier committed
56

Julien Muchembled committed
57 58
- Linux 2.6 or later

Julien Muchembled committed
59
- Python 2.7.x
Julien Muchembled committed
60

Julien Muchembled committed
61
- For storage nodes using MySQL backend:
Julien Muchembled committed
62 63

  - MySQLdb: http://sourceforge.net/projects/mysql-python
Julien Muchembled committed
64

Julien Muchembled committed
65
- For client nodes: ZODB 3.10.x
Vincent Pelletier committed
66

Aurel committed
67
Installation
Julien Muchembled committed
68
============
Aurel committed
69

Julien Muchembled committed
70 71
a. NEO can be installed like any other egg (see setup.py). Or you can simply
   make `neo` directory available for Python to import (for example, by
Julien Muchembled committed
72
   adding its container directory to the PYTHONPATH environment variable).
Vincent Pelletier committed
73

Julien Muchembled committed
74 75
b. Write a neo.conf file like the example provided. If you use MySQL,
   you'll also need create 1 database per storage node.
Vincent Pelletier committed
76

Julien Muchembled committed
77
c. Start all required nodes::
Vincent Pelletier committed
78

Julien Muchembled committed
79 80 81 82
    $ neomaster -f neo.conf
    $ neostorage -f neo.conf -s storage1
    $ neostorage -f neo.conf -s storage2
    $ neoadmin -f neo.conf
Vincent Pelletier committed
83

Julien Muchembled committed
84
d. Tell the cluster to initialize storage nodes::
Vincent Pelletier committed
85

Julien Muchembled committed
86 87 88 89 90 91 92
    $ neoctl -a <admin> start

e. Clients can connect when the cluster is in RUNNING state::

    $ neoctl -a <admin> print cluster
    RUNNING

Julien Muchembled committed
93
f. See `importer.conf` file to import an existing database,
Julien Muchembled committed
94 95 96 97
   or `neoctl` command for more administrative tasks.

Alternatively, you can use `neosimple` command to quickly setup a cluster for
testing.
Vincent Pelletier committed
98 99

How to use
Julien Muchembled committed
100
==========
Vincent Pelletier committed
101

Julien Muchembled committed
102 103
First make sure Python can import 'neo.client' package.

Julien Muchembled committed
104 105
In zope
-------
Aurel committed
106

Julien Muchembled committed
107
a. Edit your zope.conf, add a neo import and edit the `zodb_db` section by
Julien Muchembled committed
108 109
   replacing its filestorage subsection by a NEOStorage one.
   It should look like::
Aurel committed
110

Julien Muchembled committed
111 112 113 114 115 116 117 118
    %import neo.client
    <zodb_db main>
        <NEOStorage>
            master_nodes 127.0.0.1:10000
            name <cluster name>
        </NEOStorage>
        mount-point /
    </zodb_db>
Aurel committed
119

Julien Muchembled committed
120
b. Start zope
Aurel committed
121

Julien Muchembled committed
122 123
In a Python script
------------------
Aurel committed
124

Julien Muchembled committed
125 126 127 128 129
Just create the storage object and play with it::

  from neo.client.Storage import Storage
  s = Storage(master_nodes="127.0.0.1:10010", name="main")
  ...
Aurel committed
130

Julien Muchembled committed
131 132
"name" and "master_nodes" parameters have the same meaning as in
configuration file.
Aurel committed
133

Julien Muchembled committed
134 135
Shutting down
-------------
Aurel committed
136

Julien Muchembled committed
137 138 139
Before shutting down NEO, all clients like Zope instances should be stopped,
so that cluster become idle. This is required for multi-DB setups, to prevent
critical failures in second phase of TPC.
Vincent Pelletier committed
140

Julien Muchembled committed
141 142 143 144 145 146 147 148
A cluster (i.e. masters+storages+admin) can be stopped gracefully by putting it
in STOPPING state using neoctl::

  neoctl -a <admin> set cluster STOPPING

This can also be done manually, which helps if your cluster is in bad state:

- Stop all master nodes first with a SIGINT or SIGTERM, so that storage nodes
Julien Muchembled committed
149
   don't become in OUT_OF_DATE state.
Julien Muchembled committed
150
- Next stop remaining nodes with a SIGINT or SIGTERM.
Grégory Wisniewski committed
151

Julien Muchembled committed
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168
Master-slave asynchronous replication
-------------------------------------

This is the recommanded way to backup a NEO cluster.
Once a cluster with appropriate `upstream_cluster` & `upstream_masters`
configuration is started, you can switch it into backup mode
using::

  neoctl -a <admin> set cluster STARTING_BACKUP

It remembers it is in such mode when it is stopped, and it can be put back into
normal mode (RUNNING)  by setting it into STOPPING_BACKUP state.

Packs are currently not replicated, which means packing should always be done
up to a TID that is already fully replicated, so that the backup cluster has a
full history (and not random holes).

Julien Muchembled committed
169 170
Deployment
==========
Grégory Wisniewski committed
171 172

NEO has no built-in deployment features such as process daemonization. We use
Julien Muchembled committed
173
`supervisor <http://supervisord.org/>`_ with configuration like below::
Grégory Wisniewski committed
174

Julien Muchembled committed
175 176
  [group:neo]
  programs=master_01,storage_01,admin
Grégory Wisniewski committed
177

Julien Muchembled committed
178
  [program:storage_01]
Julien Muchembled committed
179
  priority=10
Julien Muchembled committed
180
  command=neostorage -s storage_01 -f /neo/neo.conf
Julien Muchembled committed
181 182 183

  [program:master_01]
  priority=20
Julien Muchembled committed
184
  command=neomaster -s master_01 -f /neo/neo.conf
Grégory Wisniewski committed
185

Julien Muchembled committed
186
  [program:admin]
Julien Muchembled committed
187
  priority=20
Julien Muchembled committed
188
  command=neoadmin -s admin -f /neo/neo.conf
Julien Muchembled committed
189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207

Developers
==========

Developers interested in NEO may refer to
`NEO Web site <http://www.neoppod.org/>`_ and subscribe to following mailing
lists:

- `neo-users <http://mail.tiolive.com/mailman/listinfo/neo-users>`_:
  users discussion
- `neo-dev <http://mail.tiolive.com/mailman/listinfo/neo-dev>`_:
  developers discussion
- `neo-report <http://mail.tiolive.com/mailman/listinfo/neo-report>`_:
  automated test results (read-only list)

Commercial Support
==================

Nexedi provides commercial support for NEO: http://www.nexedi.com/