Otherwise when `removeNonMembers` gets called as we reconnect, we get a
long list of erroneous leave notifications.