The recent API change was not enough.
A grouping method may need more information: in particular, the dummy grouping
method must be fixed to change user.

When grouped messages fail, ActivityTool must distinguish 3 groups,
in order to reexecute them separately, as follows:
- first, those that succeeded
- then, those that were skipped
- at last, failed ones

Grouping methods are updated to handle partial failures, and stop doing
anything when something goes wrong.

Without this, we would have the following pathological cases.

1. Let's suppose first that skipped messages are marked as succeeded.

The problem is that each skipped message that will fail causes the reexecution
of those that didn't fail.

Exemple: A:ok B:ok C:err D:err E:err F:err
  1: A:ok, B:ok, C:err, D:skipped, E:skipped, F:skipped
  2: A:ok, B:ok, D:err, E:skipped, F:skipped
  3: A:ok, B:ok, E:err, F:skipped
  4: A:ok, B:ok, F:err
  5: A:ok, B:ok -> commit

And worst, the first failed (C) may be processable again before 5, entering
a failing loop if it is executed again in the same group as A & B.

2. Another implementation is to mark all skipped as failed.

Example:
  1: A:ok, B:ok, C:err, D:skipped, E:skipped, F:skipped
  2: A:ok, B:ok -> commit
  3: C:err, D:skipped, E:skipped, F:skipped
 >3: same as 3

=> D, E or F are never tried.

For even more refactoring between SQLDict & SQLQueue,
which now uses SQL tables with the same schema.

The action to recreate activity tables while preserving existing messages
was unsafe for 2 reasons:
- if any error happened, messages could be lost
- it relied on Message.reactivate

Which this patch, any instance created after commit d881edd1 (Aug 2010) will
upgrade successfully. For older instances, make sure you have no activity left.

For cases where 'ALTER TABLE' would not work, a better way to implement repair
functionality would be:
- one action to backup all messages in ZODB
- and another to restore them
And maybe a security so that during the backup-clear-restore sequence,
activities can't be created nor processed.

If any column is added in the future, it would still be possible to write code
that fills them by inspecting messages.

see http://docs.python.org/2/library/unittest.html#deprecated-aliases

CMFActivity: Drop messages when document is removed but another document can be acquired with the same path.
Signed-off-by: Julien Muchembled <jm@nexedi.com>

So that creating an ActiveWrapper (or Method) once and reusing it to spawn
several activities gets a larger speed-up.
Message class API is not supposed to be used outside this module, so
drop failing test rather than fixing it.

With this change, code like the following will use much less memory.

  for i in portal_catalog(...):
    i.activate().method()
Signed-off-by: Vincent Pelletier <vincent@nexedi.com>

so that we can call ActivityTool.activateObject() with not only by
object but also by it's path.
Signed-off-by: Vincent Pelletier <vincent@nexedi.com>

When an object is deleted, higher level code used to flush its messages (without
invoking them). However, a concurrent and very long transaction may be about to
activate such an object, without conflict. We already experienced false -3
errors that could prevent other messages to be validated.

Because there is no efficient and reliable way to flush absolutely all messages,
messages on deleted objects are now ignored and deleted without any email
notification. There's only a WARNING in logs. But for performance reasons,
there's still a flush on object deletion.

To simplify code, messages that went to -3 for other reasons, like a
non-existing method, now go to -2. In fact, this was already the case for
grouped messages.

In case that a path is recycled, it may still be possible for a message to be
executed on a wrong object (the new one), instead of being ignored (because the
activated object was deleted). So in such scenario, developer should make sure
not to delete an object that may be activated in a concurrent transaction.
If the original object has an OID at the moment it is activated, an assertion
will make sure the message is not executed on another object.

Because flush() relies on this method, it becomes also possible to do
  ob.flushActivity(method_id=('immediateReindexObject',
                              'recursiveImmediateReindexObject'))

This makes code simpler, faster and easier to understand.
It is easy to forget that ZODB connections can be reused by different threads,
which led to bug such as the one fixed by commit 2c11b76a.

ZODB already maintains a pool of connections to reuse
so we don't need anymore to have one.

When a ZODB connection is closed, it usually returns to a ZODB pool and may be
reused by another thread. If the SQL connection was open and is still in ZODB
cache, the _v_database_connection attribute is still there:
ActivityConnection.connect() is not called and a new instance of ZMySQLDA.db.DB
is created for the new thread without initializing its sort key.

Messages passed to a grouping method are not sorted.

New SQLBase._getMessageList method has a very simple ZSQLCatalog-like API,
and replaces SQL{Dict,Queue}_readMessageList DTML scripts.

'include_processing' parameter disappears (in favor of 'processing')
and default behaviour is to include messages being processed.

If specified, <path> must be the path of an ancestor and SQLDict will merge all
similar messages under this path and process only the root message.

This will be used in ERP5 simulation to drop 'expand' activities that are
children of other 'expand' activities.

Because it can be used with grouped messages, it may be interesting to used it
for recursiveImmediateReindexObject.

This is a followup to commit 8995b645
("always start transactions with begin() and make tic() commit automatically"):
there is now no need to use self.stepTic() instead of self.tic()

Update all known callers to use the new method.

- reduce probability and cost of ConflictError by posting the
  activity result by activity
- do not post None (to force it, it's still possible to forge an
  ActiveResult object containing the None value)
- allow grouped messages to post results (previous implementation was not
  working, and this required to change the API of grouping methods)

This is required for workflow scripts that are run at the end of the transaction.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@42257 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@42230 20353a03-c40f-0410-a6d1-a30d3c3de9de

Fix test failures on testBusinessTemplate and spurious error messages on the console when trying to set the sort_key on an unconnected Connection object. Also fix the manual creation of the ActivityConnection object.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@41865 20353a03-c40f-0410-a6d1-a30d3c3de9de

ZMySQLDA connector for the activities database would finish its
commit procedure before ZODB, making the description of an activity
message in MySQL available before its respective data in the ZODB.

The fix consisted in replacing the ZMySQLDA connector with another
one based on ZMySQLDA but with a “sortKey()” method that forced it
to be sorted after both the ZODB connection and the ZMySQLDA
connection for ZSQLCatalog.

Analysis of issue was done by Sebastien and Julien.

This patch itself was done by Leonardo.

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@41598 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@39430 20353a03-c40f-0410-a6d1-a30d3c3de9de

A lockdown can happen here when the activity can't find
the waitingActivity method.


git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@39416 20353a03-c40f-0410-a6d1-a30d3c3de9de

git-svn-id: https://svn.erp5.org/repos/public/erp5/trunk@38120 20353a03-c40f-0410-a6d1-a30d3c3de9de