Minimal working examples for RDMA: standalone server and client and library for a TRX driver.
Monorepo containing some experimental projects around *R*emote *D*irect *M*emory *A*ccess and *S*oftware *D*efined *R*adio.
## Configuring an RDMA interface
## Prerequisites
In order to run this RDMA example, you will need to create a RDMA interface on two virtual machines.
For instance, to use RoCE (RDMA over Converged Ethernet) on interface `enp1s0`:
### Required packages
Many libraries and executables are needed in order to make RDMA work. For now, the following packages seem to be needed, although it has not been tried to shorten the list, so some of them might not be required:
-`rdma-core`
-`libibmad`
-`libibumad`
-`infiniband-diags`
-`libfabric`
-`libibverbs`
-`ibverbs-providers`
-`ibverbs-utils`
-`librdmacm`
-`rdmacm-utils`
On Ubuntu 22.04, all the required packages can be installed by running the following command:
In order to use RDMA, two different computers with an RDMA interface are needed. Without any specific hardware, it is possible to use RoCE (RDMA over Converged Ethernet).
For instance, to create a RoCE interface on Ethernet interface `enp1s0`, run the following commands:
```bash
modprobe rdma_rxe
...
...
@@ -13,27 +36,32 @@ rdma link add rxe0 type rxe netdev enp1s0
mkdir-p /sys/kernel/config/rdma_cm/rxe0
```
## Standalone server and client
## Contents of the repository
### libcapulet
`libcapulet` is an opinionated wrapper library around `libibverbs`. While the latter makes almost everything possible, the former intends at being more user-friendly, while restricting RDMA’s possibilities to a small (but very useful) subset.
### RDMA example program
The provided `rdma_standalone` example allows reading the memory of one computer from another one.
On one machine, run the server program, which will read content to be put in the shared memory from STDIN:
An example program demonstrating RDMA and `libcapulet` can be found in `example/rdma_test.c`. Two main functionalities are implemented: a simple example of RDMA Read/Write send a file over RDMA, and a stress test showing the latency between two computers (using Send/Recv).
In order to exchange RDMA parameters, an UDP context is used, so one machine will act as a server, and the other as a client. For instance, to run the example program send the content of a file run the server program, on one machine, which will read content to be put in the shared memory from STDIN:
```bash
make install
rdma_standalone -d rxe0 < README.md
make out/rdma-test
out/rdma-test file -d rxe0 < README.md
```
The other machine will act as a client, so we need to pass it the IP address of the server:
The client should (almost) immediately display the content of `README.md`, and the server the content of `Makefile`.
## TRX driver
In order to compile the TRX driver, you will need to provide your own `trx_driver.h` file from Amarisoft.
Copy the file in `include/amarisoft` and run `make`, the library driver will be located at `out/libtrx_rdma.so`.
### TRX drivers
Two drivers, in the `libtrx` directory are meant to be used with the [Amarisoft software stack](https://www.amarisoft.com/technology). In order to compile these, you will need to provide your own `trx_driver.h` file from Amarisoft. Copy the file in `include/amarisoft` and run `make trx`, the driver will be located in the `out` directory.