• Julien Muchembled's avatar
    New feature: monitoring · e434c253
    Julien Muchembled authored
    This task is done by the admin node, in 2 possible ways:
    - email notifications, as soon as some state change;
    - new 'neoctl print summary' command that can be used periodically
      to check the health of the database.
    They report the same information.
    
    About backup clusters:
    
    The admin of the main cluster also monitors selected backup clusters,
    with the help of their admin nodes.
    
    Internally, when a backup master node connects to the upstream master node,
    it receives the address of the upstream admin node and forwards it to its
    admin node, which is therefore able to connect to the upstream admin node.
    So the 2 admin nodes remain connected and communicate in 2 ways:
    - the backup node notifies upstream about the health of the backup cluster;
    - the upstream node queries the backup node periodically to check whether
      replication is not too late.
    
    TODO:
    
    A few things are hard-coded and we may want to configure them:
    - backup lateness is checked every 10 min;
    - backup is expected to never be late.
    
    There's also no delay to prevent 2 consecutive emails from having the same
    Date: (unfortunately, the RFC 5322 does not allow sub-second precision),
    in which case the MUA can display them in random order. This is mostly
    confusing when one notification is OK and the other is not, because one
    may wonder if there's a new problem.
    e434c253
backup.py 3.35 KB