Commit 8a678d25 authored by Titouan Soulard's avatar Titouan Soulard

meta: refactor README

parent c6b05feb
......@@ -4,7 +4,8 @@ LIB_FLAGS =-lc -ldl -lm -libverbs -lpthread -L./out
# Libraries after `--no-as-needed` are forcefully loaded: they are required by Amarisoft libraries
FORCE_LIB_FLAGS = -Wl,--no-as-needed -lm -lpthread
all: out/libcapulet.so out/libpotoml.so out/libtrx_rdma.so out/libtrx_play.so out/rdma-test out/generate-iq-samples out/trx-bridge
all: trx out/rdma-test out/generate-iq-samples
trx: out/libtrx_rdma.so out/libtrx_play.so out/trx-bridge
install: all
ln -sfr out/libcapulet.so /usr/local/lib/
......@@ -39,8 +40,8 @@ out/libcapulet.so: common/hashtable.lo libcapulet/net_udp.lo libcapulet/rdma_ib.
out/libpotoml.so: common/hashtable.lo libpotoml/toml.lo
gcc -shared -fPIC -DPIC -o $@ $^ $(LIB_FLAGS)
out/libtrx_rdma.so: common/circular_buffer.lo libtrx/trx_rdma.lo
gcc -shared -fPIC -DPIC -o $@ $^ $(LIB_FLAGS) -lcapulet
out/libtrx_rdma.so: out/libcapulet.so common/circular_buffer.lo libtrx/trx_rdma.lo
gcc -shared -fPIC -DPIC -o $@ common/circular_buffer.lo libtrx/trx_rdma.lo $(LIB_FLAGS) -lcapulet
out/libtrx_play.so: common/circular_buffer.lo libtrx/trx_play.lo
gcc -shared -fPIC -DPIC -o $@ $^ $(LIB_FLAGS)
......
# libcapulet and RDMA minimal working examples
# RDMA TRX driver and various utilities
Minimal working examples for RDMA: standalone server and client and library for a TRX driver.
Monorepo containing some experimental projects around *R*emote *D*irect *M*emory *A*ccess and *S*oftware *D*efined *R*adio.
## Configuring an RDMA interface
## Prerequisites
In order to run this RDMA example, you will need to create a RDMA interface on two virtual machines.
For instance, to use RoCE (RDMA over Converged Ethernet) on interface `enp1s0`:
### Required packages
Many libraries and executables are needed in order to make RDMA work. For now, the following packages seem to be needed, although it has not been tried to shorten the list, so some of them might not be required:
- `rdma-core`
- `libibmad`
- `libibumad`
- `infiniband-diags`
- `libfabric`
- `libibverbs`
- `ibverbs-providers`
- `ibverbs-utils`
- `librdmacm`
- `rdmacm-utils`
On Ubuntu 22.04, all the required packages can be installed by running the following command:
```bash
apt install rdma-core libibverbs1 libibverbs-dev librdmacm1 libibmad5 libibumad3 ibverbs-providers rdmacm-utils infiniband-diags libfabric1 ibverbs-utils
```
### Configuring an RDMA interface
In order to use RDMA, two different computers with an RDMA interface are needed. Without any specific hardware, it is possible to use RoCE (RDMA over Converged Ethernet).
For instance, to create a RoCE interface on Ethernet interface `enp1s0`, run the following commands:
```bash
modprobe rdma_rxe
......@@ -13,27 +36,32 @@ rdma link add rxe0 type rxe netdev enp1s0
mkdir -p /sys/kernel/config/rdma_cm/rxe0
```
## Standalone server and client
## Contents of the repository
### libcapulet
`libcapulet` is an opinionated wrapper library around `libibverbs`. While the latter makes almost everything possible, the former intends at being more user-friendly, while restricting RDMA’s possibilities to a small (but very useful) subset.
### RDMA example program
The provided `rdma_standalone` example allows reading the memory of one computer from another one.
On one machine, run the server program, which will read content to be put in the shared memory from STDIN:
An example program demonstrating RDMA and `libcapulet` can be found in `example/rdma_test.c`. Two main functionalities are implemented: a simple example of RDMA Read/Write send a file over RDMA, and a stress test showing the latency between two computers (using Send/Recv).
In order to exchange RDMA parameters, an UDP context is used, so one machine will act as a server, and the other as a client. For instance, to run the example program send the content of a file run the server program, on one machine, which will read content to be put in the shared memory from STDIN:
```bash
make install
rdma_standalone -d rxe0 < README.md
make out/rdma-test
out/rdma-test file -d rxe0 < README.md
```
The other machine will act as a client, so we need to pass it the IP address of the server:
```bash
make install
rdma_standalone -d rxe0 -c 192.168.16.10 < Makefile
out/rdma-test file -d rxe0 -c 192.168.16.10 < Makefile
```
The client should (almost) immediately display the content of `README.md`, and the server the content of `Makefile`.
## TRX driver
In order to compile the TRX driver, you will need to provide your own `trx_driver.h` file from Amarisoft.
Copy the file in `include/amarisoft` and run `make`, the library driver will be located at `out/libtrx_rdma.so`.
### TRX drivers
Two drivers, in the `libtrx` directory are meant to be used with the [Amarisoft software stack](https://www.amarisoft.com/technology). In order to compile these, you will need to provide your own `trx_driver.h` file from Amarisoft. Copy the file in `include/amarisoft` and run `make trx`, the driver will be located in the `out` directory.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment