With the previous commit, the request to truncate the DB was not stored
persistently, which means that this operation was still vulnerable to the case
where the master is restarted after some nodes, but not all, have already
truncated. The master didn't have the information to fix this and the result
was a DB partially truncated.
-> On a Truncate packet, a storage node only stores the tid somewhere, to send
it back to the master, which stays in RECOVERING state as long as any node
has a different value than that of the node with the latest partition table.
We also want to make sure that there is no unfinished data, because a user may
truncate at a tid higher than a locked one.
-> Truncation is now effective at the end on the VERIFYING phase, just before
returning the last ids to the master.
At last all nodes should be truncated, to avoid that an offline node comes back
with a different history. Currently, this would not be an issue since
replication is always restart from the beginning, but later we'd like they
remember where they stopped to replicate.
-> If a truncation is requested, the master waits for all nodes to be pending,
even if it was previously started (the user can still force the cluster to
start with neoctl). And any lost node during verification also causes the
master to go back to recovery.
Obviously, the protocol has been changed to split the LastIDs packet and
introduce a new Recovery, since it does not make sense anymore to ask last ids