nexedi / erp5

Hello @nexedi,

As this fix could impact other ERP5 projects (mainly the ones using captcha), I share to every one.

This merge requests fixes the bug we have with captcha on public forums (tested on clone environment). I changed the type of cache to use for portal_sessions from Ram Cache to Distributed Ram Cache (volatile memcached).

This was not an issue before because Haproxy was coupling anonymous users to one zope node by sending SERVERID cookie, which is no longer the case.

Tests are running [1].

[1] https://nexedi.erp5.net/test_result_module/20160823-59314844

Here are what I know :

• Ram Cache is much faster.
• Ram Cache can store unpickable object but Distributed Ram Cache cannot.
• erp5_commerce puts a newTempOrder() result in session that is unpickable.
• flush cache is supported in Ram Cache, not in Distributed Ram Cache, but anyway 'do not flush cache' is our policy.

IMHO, non-distributed session cache has anyway no meaning at all if we don't use SERVERID cookie. So I agree with this change, that I already did for my projects. But then we need to re-implement erp5_commerce in a different way.

erp5_commerce puts a newTempOrder() result in session that is unpickable.

oh ... that's a very good point. I see that we did 41ed1033 to comply with that test, but if I remember correctly, what we want is not to prevent pickling temp objects, but just to prevent storing them in the ZODB by mistake. Maybe we coud implement this at a level that would only prevent it to be saved in ZODB but won't prevent temp objects to be pickled ? ( one way, seem to be setting a _p_jar to something so that this fails, but there might be a more elegant way )

Anyway we cannot pickle a Temporary Order :

File "/path/to/eggs/python_memcached-1.58-py2.7.egg/memcache.py", line 998, in _val_to_store_info
    pickler.dump(val)
PicklingError: Can't pickle <class 'erp5.temp_portal_type.Temporary Order'>: it's not the same object as erp5.temp_portal_type.Temporary Order

testERP5Commerce fails as expected. I am afraid we cannot make a change that would prevent portal_sessions to store temp objects.

Does it mean that we should keep using RAM Cache and that we have to force haproxy cookie stickyness in a way similar to fa6ceb99 when portal_sessions is used ?

Haproxy cannot handle 100% cases with SERVERID, for instance if a zope is busy, the client will use another node and captcha would fail.

I think this is inconsistent to have a individual session for each zope, it should be shared.

Haproxy cannot handle 100% cases with SERVERID, for instance if a zope is busy, the client will use another node and captcha would fail.

I think this is inconsistent to have a individual session for each zope, it should be shared.

+1

We must never rely on a user accessing the same zope for two queries.

We may do our best to to direct a user to the same zope as often as practical, to improve cache hit, but it is nothing more than optimisation (and I am interested by measures of the actual impact of this optimisation, by the way). Functionality must not suffer when such opportunistic optimisation fails.

From an algorithmic point of view, RAM cache in ERP5 only makes sense if user is redirected as often as possible to the same zope (or to the few same zopes). The same could be true for some local ZODB caches. If you want to improve this feature, it would be for example OK to redirect users to one of X (ex. X=3) zopes selected randomly once for each user out of M (ex. M=100) zopes in large cluster. This means rebuilding cache 3 times in average for each user on a system with 100 zope backends. But rebuilding cache 100 times for each user would be really too much.

In both cases, sessions should be shared to all zopes, so we should use a kind of Distributed Ram Cache. As we don't use ecommerce in nexedi erp5, this patch can be applied on production.

I'm going to add a bug in bug_module, close (reject) this merge request and then wait for someone to fix it properly.

On the note considering cache on 100 zope. This is true if there is only one anonymous user. In the case of a 100 anonymous users in parallel, they will be considered by zope as the same anonymous user, so the cache will only be rebuild once.

For session mechanism JSON Web Token which are currently implement for authentication can be used as session storage and remove the needs for a cache. But no secret can be stored in it.

I'm going to add a bug in bug_module

https://nexedi.erp5.net/bug_module/20160824-1634FFA

Anyway we cannot pickle a Temporary Order

PicklingError: Can't pickle <class 'erp5.temp_portal_type.Temporary Order'>: it's not the same object as erp5.temp_portal_type.Temporary Order

True, but isn't it a bug ?

About erp5_commerce implementation, we want to keep it implemented as using a temporary order where we can easily use price lookup and tax / discount calculations, so the easiest for us would be to be able to put a temp order in memcached.

Yes it's a bug. But we cannot only resolve it by simply putting temp order in memcached.

Distributed Ram Cache is like the SERVERID here, it works at 99%. As it's not transactionnal cache, a client that point to random zope the key in the session MIGHT not be available everytime (or with a old version of it).

A solution could be to let to check (with cookie support) the version of the session entry. If the version does not match, for instance, then we can decide to do something like wait for cache to be filled or warn the client to wait or something else.

A better solution could be to use a different kind of cache like NEO purgeable object (not implemented) OR SQLCache (also not implemented) which both would be transactionnal.

... it's not transactionnal cache ...

ERP5 writes to memcached only when transaction is commited which I believe makes it transactional enough for the use cases of erp5_commerce.

If the issue comes from Captcha Field, I see one way to solve this issue at CaptchaField level.

When ERP5 calculate the CaptchaField HTML (at Formulator level), calculate an HMAC signature of the capta result (based on browser AgentID, IP), and return this value as an hidden input field.

When Formulator validate the Captcha result, it will receive the HMAC signature and the user proposal. It can accept the user proposal only if the signature matches.

In order to prevent infinite reusability of the same captcha, the HMAC signature should have a short time to live (1 min?).

The form HTML should also not be cached with HTTP, to prevent sending the same outdated Captcha to all users.

With this, no need for any Cookie/Session. I believe this is what Cedric already proposed.

mentioned in merge request !791