Skip to content

jio
jio
Commit 2d5c99d2 authored by lucas.parsy's avatar lucas.parsy Committed by Romain Courteaud
1 Browse files
Options

Update documentation. 

Squashed commit of the following:

commit 3ddc5e522969d8e5270dc408eec71fc0847c40ce
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Thu Dec 17 16:44:19 2015 +0100

    corrected misspellings and ambiguous examples in manage_document file.
    deleted gid and revision storage documentation.

commit 85d30999afdd227c8d4085fcd2df329f0d95333e
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Tue Dec 15 15:13:23 2015 +0100

    updated documentation
    renamed "revision_storage.rst" "replicate_storage.rst"
    you need to "compile" the files (go in doc/ directory and type "make")
    to have the html files.
parent 0a7b8a91
Expand all Show whitespace changes
Inline Side-by-side
Showing with 472 additions and 688 deletions
docs/available_storages.rst
View file @ 2d5c99d2
This diff is collapsed.
docs/developers.rst
View file @ 2d5c99d2
......@@ -12,7 +12,7 @@ If you want to modify or build jIO yourself, you need to
* Clone from a repository
``$ git clone https://github.com/nexedi/jio.git``
``$ git clone https://lab.nexedi.com/nexedi/jio.git``
* Install `NodeJS <http://nodejs.org/>`_ (including ``npm``)
......@@ -53,8 +53,8 @@ Create a constructor:
}
}
Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``, ``check`` and ``repair``.
Create 9 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``. ``repair`` method is optional.
.. code-block:: javascript
......@@ -81,66 +81,26 @@ Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachmen
(To help you design your methods, some tools are provided by jIO.util.)
The first parameter command provides some methods to act on the jIO job:
* ``success``, to tell jIO that the job is successfully terminated
``command.success(status[Text], [{custom key to add to the response}]);``
* ``resolve``, is equal to success
* ``error``, to tell jIO that the job cannot be done
``command.error(status[Text], [reason], [message], [{custom key to add to the response}])``
* ``retry``, to tell jIO that the job cannot be done now, but can be retried later. (same API than error)
* ``reject``, to tell jIO that the job cannot be done, let jIO to decide whether to retry or not. (same API than error)
The second parameter ``metadata`` or ``param`` is the first parameter provided by the jIO user.
The third parameter ``option`` is the option parameter provided by the jIO user.
Methods should return the following objects:
Methods should return:
* **post()** --> ``success("created", {"id": new_generated_id})``
* **post()**, **put()**, **remove()** --> id of the document affected (string)
* **put()**, **remove()**, **putAttachment()** or **removeAttachment()** --> ``success(204)``
* **putAttachment()** or **removeAttachment()** --> no specific value
* **get()** --> ``success("ok", {"data": document_metadata})``
* **get()** --> document_metadata (object)
* **getAttachment()** -->
.. code-block:: javascript
success("ok", {"data": binary_string, "content_type": content_type})
// or
success("ok", {"data": new Blob([data], {"type": content_type})})
* **allDocs()** --> ``success("ok", {"data": row_object})``
* **check()** -->
.. code-block:: javascript
// if metadata provides "_id" -> check document state
// if metadata doesn't promides "_id" -> check storage state
success("no_content")
// or
error("conflict", "corrupted", "incoherent document or storage")
* **repair()** -->
new Blob([data], {"type": content_type})
.. code-block:: javascript
* **allDocs()** --> list of all documents (restricted by a query, if given). (object)
// if metadata provides "_id" -> repair document state
// if metadata doesn't promides "_id" -> repair storage state
success("no_content")
// or
error("conflict", "corrupted", "impossible to repair document or storage")
// DON'T DESIGN STORAGES IF THERE IS NO WAY
// TO REPAIR INCOHERENT STATES
After creating all methods, your storage must be added to jIO. This is done
with the ``jIO.addStorage()`` method, which requires two parameters: the storage
......@@ -154,160 +114,3 @@ type (string) and a constructor (function). It is called like this:
Please refer to *localstorage.js* implementation for a good example on how to
setup a storage and what methods are required.
Also keep in mind that jIO is a job-based library: whenever you trigger a method,
a job is created, which will later return a response, after being processed.
Job rules
---------
The jIO job manager follows several rules set at the creation of a new jIO
instance. When you try to call a method, jIO will create a job and will make
sure the job is really necessary and will be executed. Thanks to these job
rules, jIO knows what to do with the new job before adding it to the queue. You
can also add your own rules, as we're going to see now.
These are the jIO **default rules**:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"sameParameters",
"sameOptions"
],
"action": "update"
}, {
"code_name": "metadata writers update",
"conditions": [
"sameStorageDescription",
"areWriters",
"useMetadataOnly",
"sameMethod",
"haveDocumentIds",
"sameParameters"
],
"action": "update"
}, {
"code_name": "writers wait",
"conditions": [
"sameStorageDescription",
"areWriters",
"haveDocumentIds",
"sameDocumentId"
],
"action": "wait"
}]
});
The following actions can be used:
* ``ok`` - accept the job
* ``wait`` - wait until the end of the selected job
* ``update`` - bind the selected job to this one
* ``deny`` - reject the job
The following condition functions can be used:
* ``sameStorageDescription`` - check if the storage descriptions are different.
* ``areWriters`` - check if the commands are ``post``, ``put``, ``putAttachment``, ``remove``, ``removeAttachment``, or ``repair``.
* ``areReaders`` - check if the commands are ``get``, ``getAttachment``, ``allDocs`` or ``check``.
* ``useMetadataOnly`` - check if the commands are ``post``, ``put``, ``get``, ``remove`` or ``allDocs``.
* ``sameMethod`` - check if the commands are equal.
* ``sameDocumentId`` - check if the document ids are equal.
* ``sameParameters`` - check if the metadata or param are equal (deep comparison).
* ``sameOptions`` - check if the command options are equal.
* ``haveDocumentIds`` - test if the two commands contain document ids.
Create Job Condition
--------------------
You can create two types of function: job condition, and job comparison.
.. code-block:: javascript
// Job Condition
// Check if the job is a get command
jIO.addJobRuleCondition("isGetMethod", function (job) {
return job.method === 'get';
});
// Job Comparison
// Check if the jobs have the same 'title' property
// only if they are strings
jIO.addJobRuleCondition("sameTitleIfString",
function (job, selected_job) {
if (typeof job.kwargs.title === 'string' &&
typeof selected_job.kwargs.title === 'string') {
return job.kwargs.title === selected_job.kwargs.title;
}
return false;
});
Add job rules
-------------
You just have to define job rules in the jIO options:
.. code-block:: javascript
// Do not accept to post or put a document which title is equal
// to another already running post or put document title
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "avoid similar title",
"conditions": [
"sameStorageDescription",
"areWriters",
"sameTitleIfString"
],
"action": "deny",
"before": "writers update" // optional
// "after": also exists
}]
});
Clear/Replace default job rules
-------------------------------
If a job's ``code_name`` is equal to ``readers update``, then adding this rule will replace it:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"haveDocumentIds"
"sameParameters"
// sameOptions is removed
],
"action": "update"
}]
});
Or you can just clear all rules before adding new ones:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"clear_job_rules": true,
"job_rules": [{
// ...
}]
});
docs/getting_started.rst
View file @ 2d5c99d2
......@@ -79,8 +79,6 @@ Getting started
| Deletes a document's attachment
``.allDocs()`` | ``my_jio.allDocs([options]);``
| Retrieves a list of existing documents
``.check()`` | ``my_jio.check(document, [options]);``
| Checks the document state
``.repair()`` | ``my_jio.repair(document, [options]);``
| Repairs the document
======================= ======================================================
......@@ -122,29 +120,29 @@ Storage dependencies
Storage connectors
^^^^^^^^^^^^^^^^^^
* localstorage.js
* davstorage.js
* searchableencryptionstorage.js (depends on sjcl) (WIP)
* s3storage.js (depends on sha1, jQuery) (WIP)
* Localstorage
* MemoryStorage
* IndexedDB
* WebSQL
* DavStorage
* Dropbox
* Google Drive
* ERP5Storage
* s3storage.js (WIP)
* xwikistorage.js (depends on jQuery) (WIP)
* erp5storage.js (WIP)
* restsqlstorage.js (depends on jQuery) (WIP)
* mioga2storage.js (depends on jQuery) (WIP)
Storage handlers
^^^^^^^^^^^^^^^^
* indexstorage.js
* gidstorage.js
* splitstorage.js (WIP)
* replicatestorage.js (WIP)
Revision based storage handlers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* revisionstorage.js (depends on sha256)
* replicaterevisionstorage.js
* Zipstorage
* ShaStorage
* UUIDStorage
* QueryStorage
* CryptStorage
* UnionStorage
* FileSystemBridgeStorage
* Document Storage
* Replicate Storage
Unit tests
^^^^^^^^^^
......@@ -155,9 +153,5 @@ the test suite with each release.
Fork jIO
^^^^^^^^
The same source code is kept in three synchronized repositories.
Feel free to use any of them.
* `GitHub <https://github.com/nexedi/jio>`_: ``git clone https://github.com/nexedi/jio.git``
* `Gitorius <https://gitorious.org/nexedi/jio>`_: ``git clone https://git.gitorious.org/nexedi/jio.git``
* `Git Erp5 <http://git.erp5.org/gitweb/jio.git>`_ (read only): ``git clone http://git.erp5.org/repos/jio.git``
| Feel free to use the Gitlab repository:
| `GitLab <https://lab.nexedi.com/nexedi/jio.git>`_: ``git clone https://lab.nexedi.com/nexedi/jio.git``
docs/gid_storage.rst deleted 100644 → 0
View file @ 0a7b8a91
.. _gid-storage:
jIO GIDStorage
==============
A storage to enable interoperability between all kind of storages.
A global ID (GID) is a document id which represents a unique document. This ID
is then used to find this unique document on all types of backends.
This storage uses sub storage ``.allDocs()`` and queries to find unique
documents, and converts their ids to gids.
Where it can be used
--------------------
When you want to duplicate / synchronize / split / edit data in different kind of storages (ERP5 + XWiki + Dav + ...).
Storage Description
-------------------
* ``type`` - ``"gid"``
* ``sub_storage`` - the sub storage description.
* ``constraints`` - the constraints to use to generate a gid by defining metadata types for some kind of document.
Example:
.. code-block:: javascript
{
"type": "gid",
"sub_storage": {<storage description>},
"constraints": {
"default": { // constraints for all kind of documents
// "document metadata": "type of metadata"
"type": "list"
"title": "string"
},
"Text": { // document of type 'Text' additional constraints
"language": "string"
}
}
}
This description tells the *GIDStorage* to use 2 metadata attributes (``type``, ``title``) to define a
document as unique in the default case. If the document is of type ``Text``, then
the handler will use 3 metadata (``type``, ``title``, ``language``).
If these constraints are not respected, then the storage returns an error telling us to
review the document metadata. Here are samples of document respecting the above
constraints:
.. code-block:: javascript
{
"type": "Text",
"title": "Hello World!",
"language": "en"
}
{
"type": ["Text", "Web Page"],
"title": "My Web Page Title",
"language": "en-US",
"format": "text/html"
}
{
"type": "Image",
"title": "My Image Title"
}
Available metadata types are:
* ``"json"`` - The json value of the metadata.
* ``"string"`` - The value as string if it is not a list.
* ``"list"`` - The value as list.
* ``"date"`` - The value if it can be converted to a date (as string).
* ``"DCMIType"`` - A value matching one of the DCMIType Vocabulary (as string).
* ``"contentType"`` - A value which is a content type (as string).
* ``["DCMIType", "list"]`` - The value which contains a DCMIType (as list).
* ``[...]`` - make your own combination.
Document Requirements
---------------------
A metadata value must be a string. This string can be placed in an attribute within
a ``"content"`` key. The object can contains custom keys with string values. A
metadata object can contain several values. Example:
.. code-block:: javascript
{
"key": "value",
// or
"key": ["value1", "value2"],
// or
"key": {
"attribute name": "attribute value",
"content": "value"
},
// or
"key": [
{"scheme": "DCTERMS.URI", "content": "http://foo.com/bar"},
"value2",
"value3",
...
],
...
}
Metadata attributes which names begin with an underscore can contain anything.
.. code-block:: javascript
{
"_key": {"whatever": ["blue", []], "a": null}
}
Storage Requirements
--------------------
* This storage is not compatible with *RevisionStorage* and *ReplicateRevisionStorage*.
* Sub storages have to support options for queries and ``include_docs``.
Dependencies
------------
No dependency.
Suggested storage tree
----------------------
Replication between storages::
Replicate Storage
+-- GID Storage
| `-- Local Storage
+-- GID Storage
| `-- Remote Storage 1
`-- GID Storage
`-- Remote Storage 2
**CAUTION: All gid storage must have the same description!**
Offline application usage::
Replicate Storage
+-- Index Storage with DB in Local Storage
| `-- GID Storage
| `-- ERP5 Storage
`-- GID Storage
`-- Local Storage
**CAUTION: All gid storage must have the same description!**
docs/index.rst
View file @ 2d5c99d2
......@@ -38,9 +38,8 @@ jIO documentation
getting_started
manage_documents
revision_storages
replicate_storage
available_storages
gid_storage
query
keys
metadata
......
docs/manage_documents.rst
View file @ 2d5c99d2
This diff is collapsed.
docs/metadata.rst
View file @ 2d5c99d2
......@@ -58,11 +58,6 @@ List of metadata to use
Identification
^^^^^^^^^^^^^^
* **_id**
A specific jIO metadata which helps the storage to find a document
(can be a real path name, a dc:identifier, a uuid, ...). **String Only**
* **identifier**
| ``{"identifier": "http://domain/jio_home_page"}``
......
docs/revision_storages.rst → docs/replicate_storage.rst
View file @ 2d5c99d2
.. _revision-storages-conflicts-and-resolution:
.. _replicate-storage-conflicts-and-resolution:
Revision Storages: Conflicts and Resolution
Replicate Storage: Conflicts and Resolution
===========================================
......@@ -14,25 +14,29 @@ defines a conflict as multiple versions of a document existing in a storage
tree and a user trying to save on a version that does not match the latest
version of the document.
To keep track of document versions a revision storage must be used. When doing
To keep track of document versions a replicate storage must be used. When doing
so, jIO creates a document tree file for every document. This file contains all
existing versions and their status and is modified whenever a version is
added/updated/removed or when storages are being synchronized.
How to solve conflicts
----------------------
How conflicts are handled
-------------------------
The RemoteStorage takes in parameter two substorages, one "local" and one "remote".
The "local" storage can be remote, but it will be used for all the requests
like **get()**, **getAttachment()**, **allDocs()**...
Using the document tree, jIO tries to make every version of a document
available on every storage. When multiple versions of a document exist, jIO
will select the **latest**, **left-most** version on the document tree, along with the
conflicting versions (when option **conflicts: true** is set in order for
developers to setup a routine to solve conflicts.
available on the two storages. When multiple versions of a document exist,
Jio will follow the rule set by the conflict_handling option, given at storage creation.
This option can one of the following numbers:
* 0: no conflict resolution (throws an error when conflict is occuring)
* 1: keep the local state. (overwrites the remote document with local content)
* 2: keep the remote state. (overwrites the local document with remote content)
* 3: keep both copies (leave documents untouched, no signature update)
Technically, a conflict is solved by deleting alternative versions of a document
("cutting leaves off from the document tree"). When a user decides to keep a
version of a document and manually deletes all conflicting versions, the
storage tree is updated accordingly and the document is available in a single
version on all storages.
Simple Conflict Example
-----------------------
......@@ -45,123 +49,85 @@ your PC with your new email adress. Someone else changes this email from your PC
and once your smartphone is recharged, you go back online and the previous
update is executed.
#. Set up the storage tree:
#. Set up the replicate storage:
.. code-block:: javascript
var jio_instance = jIO.createJIO({
// replicate revision storage
type: 'replicaterevision',
storagelist:[{
type: 'revision',
sub_storage: {
type: 'dav',
// replicate storage
type: 'replicate',
local_sub_storage : {
type: 'local',
...
}
}, {
type: 'revision',
sub_storage: {
type: 'local',
remote_sub_storage: {
type: 'dav',
...
}
}]
conflict_handling: ...
});
#. Create the namecard on your smartphone:
#. 1) Create the namecard on your smartphone:
.. code-block:: javascript
jio_instance.post({
_id: 'myNameCard',
email: 'me@web.com'
jio_instance.put("myNameCard", {
email: 'jb@td.com'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
// response -> 'myNameCard'
});
This will create the document on your WebDAV and local storage
#. Someone else updates your shared namecard on WebDAV:
.. code-block:: javascript
jio_instance.put({
email: 'my_new_me@web.com',
_id: 'myNameCard'
_rev: '1-5782E71F1E4BF698FA3793D9D5A96393'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-068E73F5B44FEC987B51354DFC772891'
});
Your smartphone is offline, so now you will have one version (1-578...) on
your smartphone and another version on WebDAV (2-068...) on your PC.
#. You modify the namecard while being offline:
#. 2) Someone else updates your shared namecard on WebDAV:
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
// response.data.email -> 'me@web.com'
return jio_instance.put({
_id: 'myNameCard',
email: 'me_again@web.com'
});
jio_instance.put(myNameCard, {
email: 'kyle@td.com',
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C'
// response -> 'myNameCard'
});
Your smartphone is offline, so now you will have one version on
your smartphone and another version on WebDAV on your PC.
#. Later, your smartphone is online and you retrieve the other version of the namecard:
#. 3) Later, your smartphone is online and you modify your email:
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C'
// response.data.email -> 'me_again@web.com'
});
When multiple versions of a document are available, jIO returns the latest,
left-most version on the document tree (2-375... and labels all other
versions as conflicting 2-068...).
#. Retrieve conflicts by setting option:
jio_instance.get('myNameCard').then(function (response) {
// response.email -> 'jb@td.com'
// the get() method checks only on your local storage
// and doesn't warn you about remote modifications.
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}, {
conflicts: true
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C',
// response.conflicts -> ['2-068E73F5B44FEC987B51354DFC772891']
return jio_instance.put('myNameCard', {
email: 'jack@td.com'
})
.then(function (response) {
// response -> 'myNameCard'
});
The conflicting version (*2-068E...*) is displayed, because **{conflicts: true}** was
specified in the GET call. Deleting either version will solve the conflict.
| Your latest modification of the email is: "jack@td.com"
| The modification from the other user is: "kyle@td.com"
#. Delete the conflicting version:
If your conflict_handling option was:
.. code-block:: javascript
* | 0: the email is:
| -"kyle@td.com" on WebDAV
| -"jack@td.com" on your local storage
| The storage rejects your latest modification,
| you get an error because local and remote documents are desynchronized.
| The documents in local and remote state are left untouched.
jio_instance.remove({
_id: 'myNameCard',
_rev: '2-068E73F5B44FEC987B51354DFC772891'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '3-28910A4937537B5168E772896B70EC98'
});
* | 1: the email is: "jack@td.com" on both storages
| The storage pushes the local modification, which is yours.
When deleting the conflicting version of your namecard, jIO removed it
from all storages and set the document tree leaf of that version to
*deleted*. All storages now contain just a single version of the namecard
(2-3753...). Note that, on the document tree, removing a revison will
create a new revision with status set to *deleted*.
* | 2: the email is: "kyle@td.com" on both storages
| The storage keeps the remote modification, which is from the other user.
| Your local storage is modified to fit the state of the remote storage.
* | 3: the email is: "jack@td.com" on both storages
| The storage doesn't do synchronization, and pushes your modification
| without checking if the remote storage has been changed or not
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7