Commit 7989e6ce authored by Kirill Smelkov's avatar Kirill Smelkov

software/ors-amarisoft: Do not recreate slaptapX-* on every idempotent `slapos node instance` run

To run tapsplit we use plone.recipe.command with both command and
update-command set to `tapsplit ...`. But tapsplit, when run, currently fully
recreates and reinitializes subtap interfaces, which leads to interfering with
running enb because subtap interfaces, that enb started to use, are removed.
This is not desirable behaviour.

What we need:

1) create subtap interfaces only once and keep them stable
2) until configuration changes which should lead to
   * subtaps recreated, and
   * enb restarted
3) if subtap interfaces disappear for any reason, recreate it

-> Rework tapsplit to keep its promise, that it "brings tap interface into state
   with several children interfaces each covering part of original interface
   address space", without recreating those children on every run and instead
   doing any action only if their state is not what is desired.

In other words those interfaces now are only created when they do not exist
before. Addresses and routes are added only if they are not there before
tapsplit is run, etc.

After the patch the first run of tapsplit to split by 2 looks like

    # ./pythonwitheggs ru/tapsplit slaptap16 2
    slaptap16: split 2401:5180:0:66:a200::/71 by 2
    preserve         2401:5180:0:66:a200::/73
    -> slaptap16-1   2401:5180:0:66:a280::/73
     # ip tuntap add dev slaptap16-1 mode tap user slapuser16
     # ip link set slaptap16-1 up
     # ip addr add 2401:5180:0:66:a280::/73 dev slaptap16-1 noprefixroute
     # ip route add 2401:5180:0:66:a280::1 dev slaptap16-1
     # ip route add 2401:5180:0:66:a280::/73 dev slaptap16-1 via 2401:5180:0:66:a280::1
    -> slaptap16-2   2401:5180:0:66:a300::/73
     # ip tuntap add dev slaptap16-2 mode tap user slapuser16
     # ip link set slaptap16-2 up
     # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
     # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
     # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1

The second run with the same arguments looks as

    # ./pythonwitheggs ru/tapsplit slaptap16 2
    slaptap16: split 2401:5180:0:66:a200::/71 by 2
    preserve         2401:5180:0:66:a200::/73
    -> slaptap16-1   2401:5180:0:66:a280::/73
     # slaptap16-1: already exists
     # slaptap16-1: already up
     # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
     # slaptap16-1: already has 2401:5180:0:66:a280::1 route
     # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
    -> slaptap16-2   2401:5180:0:66:a300::/73
     # slaptap16-2: already exists
     # slaptap16-2: already up
     # slaptap16-2: already has 2401:5180:0:66:a300::/73 addr
     # slaptap16-2: already has 2401:5180:0:66:a300::1 route
     # slaptap16-2: already has 2401:5180:0:66:a300::/73 route

where it could be seen that no actions had been taken.

And if, for example, the user manipulates slaptap16-2 and manually sets it
down, the third run restores it to desired 'UP' state and readds the address
and routes because the kernel removed them when link went down:

    # ip -6 addr show dev slaptap16-2
    157: slaptap16-2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
        inet6 2401:5180:0:66:a300::/73 scope global tentative noprefixroute
           valid_lft forever preferred_lft forever
    # ip -6 route show dev slaptap16-2
    2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
    2401:5180:0:66:a300::/73 via 2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
    # ip link set slaptap16-2 down
    # ip -6 addr show dev slaptap16-2
    # ip -6 route show dev slaptap16-2
    # ./pythonwitheggs ru/tapsplit slaptap16 2
    slaptap16: split 2401:5180:0:66:a200::/71 by 2
    preserve         2401:5180:0:66:a200::/73
    -> slaptap16-1   2401:5180:0:66:a280::/73
     # slaptap16-1: already exists
     # slaptap16-1: already up
     # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
     # slaptap16-1: already has 2401:5180:0:66:a280::1 route
     # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
    -> slaptap16-2   2401:5180:0:66:a300::/73
     # slaptap16-2: already exists
     # ip link set slaptap16-2 up
     # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
     # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
     # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1

The first version of this patch tried to solve the problem by setting
update-command to be noop instead of reworking tapsplit itself. But as Thomas
noted this does not satisfy requirement "3".

Amends 49ce8ef5 (software/ors-amarisoft: Provide dedicated TAP interface for each Radio Unit)

/helped-by @tomo
/cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
/reviewed-on nexedi/slapos!1508
parent 7ff43824
...@@ -76,7 +76,7 @@ md5sum = 2b08bb666c5f3ab287cdddbfdb4c9249 ...@@ -76,7 +76,7 @@ md5sum = 2b08bb666c5f3ab287cdddbfdb4c9249
[ru_tapsplit] [ru_tapsplit]
_update_hash_filename_ = ru/tapsplit _update_hash_filename_ = ru/tapsplit
md5sum = 2b8b57c5771b2a2203c0e7767e629e55 md5sum = 700aab566289619fb83ac6f3b085d983
[ru_capdo.c] [ru_capdo.c]
_update_hash_filename_ = ru/capdo.c _update_hash_filename_ = ru/capdo.c
......
#!/usr/bin/env python #!/usr/bin/env python
# Copyright (C) 2023 Nexedi SA and Contributors. # Copyright (C) 2023-2024 Nexedi SA and Contributors.
# #
# This program is free software: you can Use, Study, Modify and Redistribute # This program is free software: you can Use, Study, Modify and Redistribute
# it under the terms of the GNU General Public License version 3, or (at your # it under the terms of the GNU General Public License version 3, or (at your
...@@ -34,38 +34,98 @@ from math import log2, ceil ...@@ -34,38 +34,98 @@ from math import log2, ceil
import sys import sys
import subprocess import subprocess
import json
# LinkDB represents snapshot of state of all network interfaces.
class LinkDB:
def __init__(db):
db.linkv = ip('link', 'show')
# ifget returns information about interface with specified name.
def ifget(db, ifname):
for link in db.linkv:
if link['ifname'] == ifname:
return link
raise KeyError('interface %r not found' % ifname)
def main(): def main():
tap = sys.argv[1] tap = sys.argv[1]
n = int(sys.argv[2]) n = int(sys.argv[2])
assert n >= 0, n assert n >= 0, n
# determine tap's network address and owner # determine tap's network address and owner
owner = readfile(sysnet(tap) + '/owner') .strip() ldb = LinkDB()
_ = ldb.ifget(tap)
owner = _['linkinfo']['info_data']['user']
net = ifnet6(tap) net = ifnet6(tap)
print('%s: split %s by %d' % (tap, net, n)) print('%s: split %s by %d' % (tap, net, n))
# cleanup existing children
for ifname in netifaces.interfaces():
if ifname.startswith('%s-' % tap):
run('ip', 'link', 'del', ifname)
# do the split # do the split
# with leaving first range for the original tap # with leaving first range for the original tap
subtap_set = set()
for i, subnet in enumerate(netsplit(net, 1+n)): for i, subnet in enumerate(netsplit(net, 1+n)):
if i == 0: if i == 0:
print('preserve %s' % subnet) print('preserve %s' % subnet)
continue # leave this range for original tap continue # leave this range for original tap
subtap = '%s-%d' % (tap, i) subtap = '%s-%d' % (tap, i)
subtap_set.add(subtap)
print('-> %s %s' % (subtap, subnet)) print('-> %s %s' % (subtap, subnet))
def note(msg):
print(' # %s: %s' % (subtap, msg))
# create subtap
try:
link = ldb.ifget(subtap)
except KeyError:
run('ip', 'tuntap', 'add', 'dev', subtap, 'mode', 'tap', 'user', owner) run('ip', 'tuntap', 'add', 'dev', subtap, 'mode', 'tap', 'user', owner)
link = ip('link', 'show', 'dev', subtap)[0]
else:
note('already exists')
# set it up
if 'UP' not in link['flags']:
run('ip', 'link', 'set', subtap, 'up') run('ip', 'link', 'set', subtap, 'up')
else:
note('already up')
# add subnet address
addrv = []
for _ in ip('-6', 'addr', 'show', 'dev', subtap):
addrv.extend(_['addr_info'])
for addr in addrv:
_ = netaddr.IPNetwork('%s/%s' % (addr['local'], addr['prefixlen']))
if _ == subnet and addr['noprefixroute']:
note('already has %s addr' % str(subnet))
break
else:
run('ip', 'addr', 'add', str(subnet), 'dev', subtap, 'noprefixroute') run('ip', 'addr', 'add', str(subnet), 'dev', subtap, 'noprefixroute')
# add /128 route to subnet::1
rtv = ip('-6', 'route', 'show', 'dev', subtap)
for rt in rtv:
if rt['dst'] == str(subnet[1]) and 'gateway' not in rt:
note('already has %s route' % str(subnet[1]))
break
else:
run('ip', 'route', 'add', str(subnet[1]), 'dev', subtap) run('ip', 'route', 'add', str(subnet[1]), 'dev', subtap)
# add route to subnet via subnet::1
for rt in rtv:
if rt['dst'] == str(subnet) and rt.get('gateway') == str(subnet[1]):
note('already has %s route' % str(subnet))
break
else:
run('ip', 'route', 'add', str(subnet), 'dev', subtap, 'via', str(subnet[1])) run('ip', 'route', 'add', str(subnet), 'dev', subtap, 'via', str(subnet[1]))
# remove other existing children
for ifname in netifaces.interfaces():
if ifname.startswith('%s-' % tap) and (ifname not in subtap_set):
print('-> del %s' % ifname)
run('ip', 'link', 'del', ifname)
# netsplit splits network into n subnetworks. # netsplit splits network into n subnetworks.
def netsplit(net, n): # -> []subnet def netsplit(net, n): # -> []subnet
...@@ -102,17 +162,15 @@ def ifnet6(ifname): ...@@ -102,17 +162,15 @@ def ifnet6(ifname):
net = net.cidr net = net.cidr
return net return net
# sysnet returns path on /sys corresponding to given interface. # run executes `*argv` as action.
def sysnet(ifname):
return '/sys/devices/virtual/net/%s' % ifname
def run(*argv): def run(*argv):
print(' # %s' % ' '.join(argv)) print(' # %s' % ' '.join(argv))
subprocess.check_call(argv) subprocess.check_call(argv)
def readfile(path): # ip returns decoded output of `ip -details *argv`
with open(path) as f: def ip(*argv):
return f.read() _ = subprocess.check_output(['ip', '-json', '-details'] + list(argv))
return json.loads(_)
if __name__ == '__main__': if __name__ == '__main__':
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment