• Julien Muchembled's avatar
    Safer DB truncation, new 'truncate' ctl command · d3c8b76d
    Julien Muchembled authored
    With the previous commit, the request to truncate the DB was not stored
    persistently, which means that this operation was still vulnerable to the case
    where the master is restarted after some nodes, but not all, have already
    truncated. The master didn't have the information to fix this and the result
    was a DB partially truncated.
    
    -> On a Truncate packet, a storage node only stores the tid somewhere, to send
       it back to the master, which stays in RECOVERING state as long as any node
       has a different value than that of the node with the latest partition table.
    
    We also want to make sure that there is no unfinished data, because a user may
    truncate at a tid higher than a locked one.
    
    -> Truncation is now effective at the end on the VERIFYING phase, just before
       returning the last ids to the master.
    
    At last all nodes should be truncated, to avoid that an offline node comes back
    with a different history. Currently, this would not be an issue since
    replication is always restart from the beginning, but later we'd like they
    remember where they stopped to replicate.
    
    -> If a truncation is requested, the master waits for all nodes to be pending,
       even if it was previously started (the user can still force the cluster to
       start with neoctl). And any lost node during verification also causes the
       master to go back to recovery.
    
    Obviously, the protocol has been changed to split the LastIDs packet and
    introduce a new Recovery, since it does not make sense anymore to ask last ids
    during recovery.
    d3c8b76d
app.py 22.3 KB