1. 19 Mar, 2024 2 commits
  2. 18 Mar, 2024 6 commits
  3. 15 Mar, 2024 2 commits
  4. 13 Mar, 2024 1 commit
    • Léo-Paul Géneau's avatar
      component/qjs-wrapper: v↑ qjs-wrapper (1.3 -> 2.0) · 649249ca
      Léo-Paul Géneau authored
      component/mavsdk: v↑ mavsdk (0.39.0 -> 1.4.13)
      component/gwsocket: add gwsocket websocket server
      software/js-drone: add frontend for subscriber
      software/js-drone: use WebSocket for subscriber
      slapos/software: run quickjs as a service
      649249ca
  5. 12 Mar, 2024 2 commits
    • Łukasz Nowak's avatar
      rapid-cdn: Fix re6st-verification-url testing · be75d825
      Łukasz Nowak authored
       * the default is already tested so drop TestRe6stVerificationUrlDefaultSlave
       * use locally provided URL to check re6st-verification-url, to not depend
         on external resources
      
      !1501 did a good job with making default disabled, but during the code
      review tests changes were not well checked.
      be75d825
    • Titouan Soulard's avatar
      rapid-cdn: fix failing test · 00586554
      Titouan Soulard authored
      logrotate test for the rapid-cdn SR have been failing for quite some
      time because of a squid.log file created (and henceforth rotated) on
      any slave instance of the CDN:
      
      ```
      Traceback (most recent call last):
          self.assertEqual(
      AssertionError: Items in the second set but not the first:
      'squid.log_10.0.160.212.20240306.23h57m09s-20240307.00h00m02s.old.xz'
      ```
      
      Since we do not want to test this file in that specific test, this
      commit instead loosely check the content of the rotated files
      directory.
      In other words, the test now checks for the presence of the two
      files to be tested, but avoid failing if other files exist.
      
      This goes in line with the two following lines of that same test,
      loosely checking for file absence instead of an empty directory.
      
      /cc @tomo @luke
      /approved-by @luke
      /reviewed-on !1541
      00586554
  6. 08 Mar, 2024 3 commits
  7. 05 Mar, 2024 1 commit
  8. 01 Mar, 2024 1 commit
    • Jérome Perrin's avatar
      stack/erp5: patch RestrictedPython to compile with print_function · ccbe9a26
      Jérome Perrin authored
      Every restricted python code on python2 will be compiled as if it had
      `from __future__ import print_function`, to ease transition away from
      python2.
      
      To update project code, 2to3 from python2.7 seems to do a good job.
      Invoking like from the root of a repository rewrite all scripts:
      
          2to3  --write --nobackups --no-diffs --fix=print .
      ccbe9a26
  9. 29 Feb, 2024 3 commits
  10. 27 Feb, 2024 2 commits
  11. 26 Feb, 2024 2 commits
  12. 22 Feb, 2024 2 commits
    • Jérome Perrin's avatar
      stack/erp5,software/slapos-master: remove unused traces of wsgi parameter · 31c5f124
      Jérome Perrin authored
      This parameter no longer exists, this was not removed correctly
      31c5f124
    • Nicolas Wavrant's avatar
      erp5: restore ZODB using the --with-verify option of "repozo --recover" · a21d8d03
      Nicolas Wavrant authored
      "repozo --verify" is not working as this code expects it to: it simply
      prints errors in stdout, and doesn't return an error code in case of
      error. Thus, running it had absolutely no effect, except wasting IO
      and CPU time.
      
      This commit introduces the use of "repozo --recover --with-verify",
      which runs the verify and the recover in a same step, and has the
      advantage to raise (it doesn't exit with 0) in case of error. Also, as
      it does the verification and the recovery at the same time, it uses
      half the IO for the read. On a production server using SSDs, with a
      ZODB of 1Tb, runner-import-restore now takes 14h instead of 26h, iow a
      performance increase of 46%.
      a21d8d03
  13. 21 Feb, 2024 1 commit
  14. 20 Feb, 2024 1 commit
  15. 19 Feb, 2024 1 commit
  16. 18 Feb, 2024 2 commits
    • Kirill Smelkov's avatar
      software/ors-amarisoft: MultiRU · ed138c6e
      Kirill Smelkov authored
      Hello up there. This merge-request brings in major update to ors-amarisoft
      software release: first eNB is significantly restructured to prepare base for
      further changes, and then we add support for working with multiple radio units
      and multiple cells with all LTE/NR and FDD/TDD simultaneously. All kinds of
      Carrier Aggregation - LTE+LTE, NR+NR and LTE+NR are now supported. All kinds of
      Handover - Intra-ENB, Inter-ENB with LTE→NR and NR→LTE are now supported as
      well. UE simulator is also updated to support multiple radio units, cells and
      UEs. In the new system configuration of RU, CELL, PEERCELL, PEER and UE objects
      are done via shared instances attached to the main eNB or UEsim instance.
      
      Most of the parameters become runtime settings instead of being static choice
      of particular software template. There is no longer multiple rendered
      softwares - all that remain is
      
      1. `software.cfg` for generic software, and
      2. `software-ors.cfg` for ORS.
      
      Switching to configuring things at runtime became possible because SlapOS Master
      recently switched to new JSON-editor with support for `oneOf`, arrays and
      conditionals - bits that make it possible to configure settings in the WEB UI
      with multiple choices for e.g. RF mode, cell or radio unit type.
      
      For ORS full backward compatibility is preserved via special proxy which
      translates ORS input schema to configuration objects of the new generic eNB.
      Since most our current ORS deployments are TDD, `software-tdd-ors.cfg` link to
      `software-ors.cfg` is also provided to preserve backward compatibility at
      software-release URL level for those instances.
      
      eNB and gNB are merged along the way. Unittests are improved. JSON schemas
      become primary source for defaults(*). Unnecessary parameters are removed and
      are now computed automatically. For example it is no longer needed to
      explicitly specify SSB NR-ARFCN for peer NR cell, or `txa0cc00_center_frequency`
      for Lopcomm RU. `tx_gain` and `rx_gain` become generic parameters that
      semantically apply uniformly to all Radio Units.
      
      A protection against buildout code injection via specially-crafted references
      of shared instances is installed. The problem was noticed because instantiation
      was failing with spaces in the references - a condition that is present by
      default on the testnodes. Solving the problem generally via custom "buildout
      encoding" was not hard and probably the solution might be useful not only for
      ors-amarisoft software release. Please see the patch `"Protect from buildout
      code injection"` for details.
      
      There are more minor enhancements and bug fixes in there.
      
      Please see individual patches for details.
      
      Kirill
      
      /cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
      /approved-by @tomo
      /reviewed-on !1533
      
      (*) this goes in line with similar design choice to make JSON schemas primary
          source of defaults in Rapid-CDN: !1380 .
      ed138c6e
    • Kirill Smelkov's avatar
      software/ors-amarisoft: Do not recreate slaptapX-* on every idempotent `slapos node instance` run · 7989e6ce
      Kirill Smelkov authored
      To run tapsplit we use plone.recipe.command with both command and
      update-command set to `tapsplit ...`. But tapsplit, when run, currently fully
      recreates and reinitializes subtap interfaces, which leads to interfering with
      running enb because subtap interfaces, that enb started to use, are removed.
      This is not desirable behaviour.
      
      What we need:
      
      1) create subtap interfaces only once and keep them stable
      2) until configuration changes which should lead to
         * subtaps recreated, and
         * enb restarted
      3) if subtap interfaces disappear for any reason, recreate it
      
      -> Rework tapsplit to keep its promise, that it "brings tap interface into state
         with several children interfaces each covering part of original interface
         address space", without recreating those children on every run and instead
         doing any action only if their state is not what is desired.
      
      In other words those interfaces now are only created when they do not exist
      before. Addresses and routes are added only if they are not there before
      tapsplit is run, etc.
      
      After the patch the first run of tapsplit to split by 2 looks like
      
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # ip tuntap add dev slaptap16-1 mode tap user slapuser16
           # ip link set slaptap16-1 up
           # ip addr add 2401:5180:0:66:a280::/73 dev slaptap16-1 noprefixroute
           # ip route add 2401:5180:0:66:a280::1 dev slaptap16-1
           # ip route add 2401:5180:0:66:a280::/73 dev slaptap16-1 via 2401:5180:0:66:a280::1
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # ip tuntap add dev slaptap16-2 mode tap user slapuser16
           # ip link set slaptap16-2 up
           # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
           # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
           # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1
      
      The second run with the same arguments looks as
      
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # slaptap16-1: already exists
           # slaptap16-1: already up
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
           # slaptap16-1: already has 2401:5180:0:66:a280::1 route
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # slaptap16-2: already exists
           # slaptap16-2: already up
           # slaptap16-2: already has 2401:5180:0:66:a300::/73 addr
           # slaptap16-2: already has 2401:5180:0:66:a300::1 route
           # slaptap16-2: already has 2401:5180:0:66:a300::/73 route
      
      where it could be seen that no actions had been taken.
      
      And if, for example, the user manipulates slaptap16-2 and manually sets it
      down, the third run restores it to desired 'UP' state and readds the address
      and routes because the kernel removed them when link went down:
      
          # ip -6 addr show dev slaptap16-2
          157: slaptap16-2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
              inet6 2401:5180:0:66:a300::/73 scope global tentative noprefixroute
                 valid_lft forever preferred_lft forever
          # ip -6 route show dev slaptap16-2
          2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
          2401:5180:0:66:a300::/73 via 2401:5180:0:66:a300::1 metric 1024 linkdown pref medium
          # ip link set slaptap16-2 down
          # ip -6 addr show dev slaptap16-2
          # ip -6 route show dev slaptap16-2
          # ./pythonwitheggs ru/tapsplit slaptap16 2
          slaptap16: split 2401:5180:0:66:a200::/71 by 2
          preserve         2401:5180:0:66:a200::/73
          -> slaptap16-1   2401:5180:0:66:a280::/73
           # slaptap16-1: already exists
           # slaptap16-1: already up
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 addr
           # slaptap16-1: already has 2401:5180:0:66:a280::1 route
           # slaptap16-1: already has 2401:5180:0:66:a280::/73 route
          -> slaptap16-2   2401:5180:0:66:a300::/73
           # slaptap16-2: already exists
           # ip link set slaptap16-2 up
           # ip addr add 2401:5180:0:66:a300::/73 dev slaptap16-2 noprefixroute
           # ip route add 2401:5180:0:66:a300::1 dev slaptap16-2
           # ip route add 2401:5180:0:66:a300::/73 dev slaptap16-2 via 2401:5180:0:66:a300::1
      
      The first version of this patch tried to solve the problem by setting
      update-command to be noop instead of reworking tapsplit itself. But as Thomas
      noted this does not satisfy requirement "3".
      
      Amends 49ce8ef5 (software/ors-amarisoft: Provide dedicated TAP interface for each Radio Unit)
      
      /helped-by @tomo
      /cc @jhuge, @lu.xu, @xavier_thompson, @Daetalus
      /reviewed-on !1508
      7989e6ce
  17. 16 Feb, 2024 8 commits
    • Jérome Perrin's avatar
      ERP5: Move frontend virtualhost logic on backend · 7ff43824
      Jérome Perrin authored
       - use caucase for balancer certificate
       - move virtual host logic on the backend
       - change "frontend" parameter to request "" type (and no longer "zope")
      
      See merge request !1504
      7ff43824
    • Jérome Perrin's avatar
      stack/erp5: implement Zope's rewrite rules in ERP5 balancer partition · 6e735808
      Jérome Perrin authored
      The strategy for compatibility is that:
       - haproxy still listen on the same port as before, without rewrite rule.
         This is called "legacy" port.
       - for each frontend from request parameters, we introduce an haproxy
         frontend with a rewrite for the corresponding `internal-path`
         parameter.
       - the shared frontend instance is updated to use this new frontend
         entry from haproxy. This will cause a small downtime until the shared
         frontend is updated to the new URL on ERP5, but since this feature
         was not used, it's OK.
      
      Technical details are that we:
       - split haproxy config to have frontends and backends.
       - introduce one frontend in haproxy for each frontend from request
         parameters.
       - routing-rule-list argument is still honored the same way, globally
         and after path from frontend.
       - change the shared frontend requests to use "" type, no longer "zope"
         type.
       - we don't do automatic detection of /VirtualHostRoot in URL but always
         add it, because it could be used to trick zope into thinking it
         serves requests for an arbitrary host and do open redirects
       - before using the request's host header in virtualhost path, we check
         that it does not contain /, to prevent injection of virutalhost path
         elements through the host header.
       - we don't use the "path" parameter from shared frontend, because we
         want the frontend to be simple, so we don't want it to rewrite the
         request path (which is also the reason why we deprecated "zope" type)
       - the tests have changed a lot, because they were using what's now the
         "legacy" URL types, so we updated it to use the new URL types with
         all the /VirtualHostRoot/../ in path and also because they use IPv6
         URL, no longer IPv4
      6e735808
    • Jérome Perrin's avatar
      5b3fc1f2
    • Jérome Perrin's avatar
      stack/erp5: use slapos.recipe.build to manage haproxy parameters · 2fc522bf
      Jérome Perrin authored
      and save the already allocated ports in a state file, so that requesting
      new families does not change already allocated ports.
      2fc522bf
    • Jérome Perrin's avatar
      stack/erp5: use caucase managed certificate for balancer · d49914a6
      Jérome Perrin authored
      This reverts commit 620c9332 (stack/erp5: stop using caucase managed
      certificate for balancer, 2020-11-10) with an updated design. We add a
      caucase service for balancer in the balancer partition. The caucase
      service from the root partition (that was not used) is removed.
      
      The underlying idea is that the default configuration should use multiple
      caucases with limited scope, here we have one caucase to manage the
      certificate used by haproxy server in the balancer partition, so we put
      one caucase to manage this certificate and the caucase is configured to
      auto-accept one certificate only. The plan is that when we will add a
      certificate for mariadb server, we'll add another caucase inside this
      mariadb server.
      
      For more advanced usage and also to support the cases where a new
      certificate needs to be re-emitted for some reason, users can request
      with an existing caucase URL. In that case, they will have to accept
      the certificate requests.
      
      Notable changes:
      
      balancer/ssl/caucase-url is no longer documented in parameters, this is
      an internal parameter, users can pass one global caucase service to
      manage all partition
      
      CAUCASE environment variable is no longer set when running zope. There
      was no identified use case and with this new approach of multiple
      caucases, the term "caucase" alone became ambiguous.
      d49914a6
    • Jérome Perrin's avatar
      stack/erp5: remove not used "backend-path" · 16c9df39
      Jérome Perrin authored
      This is not documented in schema and has no effect in erp5 (but this is
      still used for slapos-master)
      16c9df39
    • Jérome Perrin's avatar
    • Jérome Perrin's avatar
      ERP5: rework frontend instance parameter · cb78214e
      Jérome Perrin authored
      This change the format or the (mostly) unused frontend parameter to
      support requesting more than one frontend and also enable the request of
      a frontend by default, so that requesting a frontend separately is no
      longer needed.
      
      The `frontend` parameter now also supports requesting frontends for
      specific paths on the ERP5 backend, the example below requests a
      frontend serving directly a web site, with the necessary rewrite rules:
      
      ```js
      {
        "frontend": {
          "default": {
            "internal-path": "/erp5/web_site_module/renderjs_runner/"
          }
        }
      }
      ```
      
      The example below requests a default frontend to the erp5 root, to
      access the ZMI or erp5_xhtml_style interface and two web sites:
      
      ```js
      {
        "frontend": {
          "default": {},
          "erp5js": {
            "internal-path": "/erp5/web_site_module/renderjs_runner/"
          },
          "crm": {
            "internal-path": "/erp5/web_site_module/erp5_officejs_support_request_ui/"
          }
        }
      }
      ```
      
      The example below has an explicit definition of the zope families using
      `zope-partition-dict` parameter, because there is more than one zope
      family, no frontend is requested by default:
      
      ```js
      {
        "zope-partition-dict": {
          "backoffice": {
            "family": "backoffice"
          },
          "web": {
            "family": "web"
          },
          "activities": {
            "family": "activities"
          }
        }
      }
      ```
      
      Continuing this example, to have frontends for backoffice and web
      families, the frontend request can specify the families, like it is
      demonstrated in the example below. In this example, we don't specify an
      entry for "activities" family, so no frontend will be requested for
      this family.
      
      ```js
      {
        "frontend": {
          "backoffice": {
            "zope-family": "backoffice"
          },
          "web": {
            "zope-family": "web",
            "internal-path": "/erp5/web_site_module/web_site/"
          }
        }
        "zope-partition-dict": {
          "backoffice": {
            "family": "backoffice"
          },
          "web": {
            "family": "web"
          },
          "activities": {
            "family": "activities"
          }
        }
      }
      ```
      cb78214e