Skip to content

L
linux
Commit f438ee21 authored by Daniel Borkmann's avatar Daniel Borkmann Committed by Andrii Nakryiko
Browse files
Options

Merge branch 'bpf-mips-jit' 

Johan Almbladh says:

====================
This is an implementation of an eBPF JIT for MIPS I-V and MIPS32/64 r1-r6.
The new JIT is written from scratch, but uses the same overall structure
as other eBPF JITs.

Before, the MIPS JIT situation looked like this.

  - 32-bit: MIPS32, cBPF-only, tests fail
  - 64-bit: MIPS64r2-r6, eBPF, tests fail, incomplete eBPF ISA support

The new JIT implementation raises the bar to the following level.

  - 32/64-bit: all MIPS ISA, eBPF, all tests pass, full eBPF ISA support

Overview
--------
The implementation supports all 32-bit and 64-bit eBPF instructions
defined as of this writing, including the recently-added atomics. It is
intended to provide good performance for native word size operations,
while also being complete so the JIT never has to fall back to the
interpreter. The new JIT replaces the current cBPF and eBPF JITs for MIPS.

The implementation is divided into separate files as follows. The source
files contains comments describing internal mechanisms and details on
things like eBPF-to-CPU register mappings, so I won't repeat that here.

  - jit_comp.[ch]    code shared between 32-bit and 64-bit JITs
  - jit_comp32.c     32-bit JIT implementation
  - jit_comp64.c     64-bit JIT implementation

Both the 32-bit and 64-bit versions map all eBPF registers to native MIPS
CPU registers. There are also enough unmapped CPU registers available to
allow all eBPF operations implemented natively by the JIT to use only CPU
registers without having to resort to stack scratch space.

Some operations are deemed too complex to implement natively in the JIT.
Those are instead implemented as a function call to a helper that performs
the operation. This is done in the following cases.

  - 64-bit div and mod on a 32-bit CPU
  - 64-bit atomics on a 32-bit CPU
  - 32-bit atomics on a 32-bit CPU that lacks ll/sc instructions

CPU errata workarounds
----------------------
The JIT implements workarounds for R10000, Loongson-2F and Loongson-3 CPU
errata. For the Loongson workarounds, I have used the public information
available on the matter.

Link: https://sourceware.org/legacy-ml/binutils/2009-11/msg00387.html

Testing
-------
During the development of the JIT, I have added a number of new test cases
to the test_bpf.ko test suite to be able to verify correctness of JIT
implementations in a more systematic way. The new additions increase the
test suite roughly three-fold, with many of the new tests being very
extensive and even exhaustive when feasible.

Link: https://lore.kernel.org/bpf/20211001130348.3670534-1-johan.almbladh@anyfinetworks.com/
Link: https://lore.kernel.org/bpf/20210914091842.4186267-1-johan.almbladh@anyfinetworks.com/
Link: https://lore.kernel.org/bpf/20210809091829.810076-1-johan.almbladh@anyfinetworks.com/

The JIT has been tested by running the test_bpf.ko test suite in QEMU with
the following MIPS ISAs, in both big and little endian mode, with and
without JIT hardening enabled.

  MIPS32r2, MIPS32r6, MIPS64r2, MIPS64r6

For the QEMU r2 targets, the correctness of pre-r2 code emitted has been
tested by manually overriding each of the following macros with 0.

  cpu_has_llsc, cpu_has_mips_2, cpu_has_mips_r1, cpu_has_mips_r2

Similarly, CPU errata workaround code has been tested by enabling the
each of the following configurations for the MIPS64r2 targets.

  CONFIG_WAR_R10000
  CONFIG_CPU_LOONGSON3_WORKAROUNDS
  CONFIG_CPU_NOP_WORKAROUNDS
  CONFIG_CPU_JUMP_WORKAROUNDS

The JIT passes all tests in all configurations. Below is the summary for
MIPS32r2 in little endian mode.

  test_bpf: Summary: 1006 PASSED, 0 FAILED, [994/994 JIT'ed]
  test_bpf: test_tail_calls: Summary: 8 PASSED, 0 FAILED, [8/8 JIT'ed]
  test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED

According to MIPS ISA reference documentation, the result of a 32-bit ALU
arithmetic operation on a 64-bit CPU is unpredictable if an operand
register value is not properly sign-extended to 64 bits. To verify the
code emitted by the JIT, the code generation engine in QEMU was modifed to
flip all low 32 bits if the above condition was not met. With this
trip-wire installed, the kernel booted properly in qemu-system-mips64el
and all test_bpf.ko tests passed.

Remaining features
------------------
While the JIT is complete is terms of eBPF ISA support, this series does
not include support for BPF-to-BPF calls and BPF trampolines. Those
features are planned to be added in another patch series.

The BPF_ST | BPF_NOSPEC instruction currently emits nothing. This is
consistent with the behavior if the MIPS interpreter and the existing
eBPF JIT.

Why not build on the existing eBPF JIT?
---------------------------------------
The existing eBPF JIT was originally written for MIPS64. An effort was
made to add MIPS32 support to it in commit 716850ab ("MIPS: eBPF:
Initial eBPF support for MIPS32 architecture."). That turned out to
contain a number of flaws, so eBPF support for MIPS32 was disabled in
commit 36366e36 ("MIPS: BPF: Restore MIPS32 cBPF JIT").

Link: https://lore.kernel.org/bpf/5deaa994.1c69fb81.97561.647e@mx.google.com/

The current eBPF JIT for MIPS64 lacks a lot of functionality regarding
ALU32, JMP32 and atomic operations. It also lacks 32-bit CPU support on a
fundamental level, for example 32-bit CPU register mappings and o32 ABI
calling conventions. For optimization purposes, it tracks register usage
through the program control flow in order to do zero-extension and sign-
extension only when necessary, a static analysis of sorts. In my opinion,
having this kind of complexity in JITs, and for which there is not
adequate test coverage, is a problem. Such analysis should be done by the
verifier, if needed at all. Finally, when I run the BPF test suite
test_bpf.ko on the current JIT, there are errors and warnings.

I believe that an eBPF JIT should strive to be correct, complete and
optimized, and in that order. The JIT runs after the verifer has audited
the program and given its approval. If the JIT then emits code that does
something else, it will undermine the eBPF security model. A simple
implementation is easier to get correct than a complex one. Furthermore,
the real performance hit is not an extra CPU instruction here and there,
but when the JIT bails on an unimplemented eBPF instruction and cause the
whole program to fall back to the interpreter. My reasoning here boils
down to the following.

* The JIT should not contain a static analyzer that tracks branches.

* It is acceptable to emit possibly superfluous sign-/zero-extensions for
  ALU32 and JMP32 operations on a 64-bit MIPS to guarantee correctness.

* The JIT should handle all eBPF instructions on all MIPS CPUs.

I conclude that the current eBPF MIPS JIT is complex, incomplete and
incorrect. For the reasons stated above, I decided to not use the existing
JIT implementation.
====================
Signed-off-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
Signed-off-by: default avatarAndrii Nakryiko <andrii@kernel.org>
parents 9d057872 ebcbacfa
Hide whitespace changes
Inline Side-by-side
Showing with 4244 additions and 3608 deletions
MAINTAINERS
View file @ f438ee21
......@@ -3422,6 +3422,7 @@ S: Supported
F: arch/arm64/net/
BPF JIT for MIPS (32-BIT AND 64-BIT)
M: Johan Almbladh <johan.almbladh@anyfinetworks.com>
M: Paul Burton <paulburton@kernel.org>
L: netdev@vger.kernel.org
L: bpf@vger.kernel.org
......
arch/mips/Kconfig
View file @ f438ee21
......@@ -57,7 +57,6 @@ config MIPS
select HAVE_ARCH_TRACEHOOK
select HAVE_ARCH_TRANSPARENT_HUGEPAGE if CPU_SUPPORTS_HUGEPAGES
select HAVE_ASM_MODVERSIONS
select HAVE_CBPF_JIT if !64BIT && !CPU_MICROMIPS
select HAVE_CONTEXT_TRACKING
select HAVE_TIF_NOHZ
select HAVE_C_RECORDMCOUNT
......@@ -65,7 +64,10 @@ config MIPS
select HAVE_DEBUG_STACKOVERFLOW
select HAVE_DMA_CONTIGUOUS
select HAVE_DYNAMIC_FTRACE
select HAVE_EBPF_JIT if 64BIT && !CPU_MICROMIPS && TARGET_ISA_REV >= 2
select HAVE_EBPF_JIT if !CPU_MICROMIPS && \
!CPU_DADDI_WORKAROUNDS && \
!CPU_R4000_WORKAROUNDS && \
!CPU_R4400_WORKAROUNDS
select HAVE_EXIT_THREAD
select HAVE_FAST_GUP
select HAVE_FTRACE_MCOUNT_RECORD
......
arch/mips/include/asm/uasm.h
View file @ f438ee21
......@@ -145,6 +145,7 @@ Ip_u1(_mtlo);
Ip_u3u1u2(_mul);
Ip_u1u2(_multu);
Ip_u3u1u2(_mulu);
Ip_u3u1u2(_muhu);
Ip_u3u1u2(_nor);
Ip_u3u1u2(_or);
Ip_u2u1u3(_ori);
......@@ -248,7 +249,11 @@ static inline void uasm_l##lb(struct uasm_label **lab, u32 *addr) \
#define uasm_i_bnezl(buf, rs, off) uasm_i_bnel(buf, rs, 0, off)
#define uasm_i_ehb(buf) uasm_i_sll(buf, 0, 0, 3)
#define uasm_i_move(buf, a, b) UASM_i_ADDU(buf, a, 0, b)
#ifdef CONFIG_CPU_NOP_WORKAROUNDS
#define uasm_i_nop(buf) uasm_i_or(buf, 1, 1, 0)
#else
#define uasm_i_nop(buf) uasm_i_sll(buf, 0, 0, 0)
#endif
#define uasm_i_ssnop(buf) uasm_i_sll(buf, 0, 0, 1)
static inline void uasm_i_drotr_safe(u32 **p, unsigned int a1,
......
arch/mips/mm/uasm-mips.c
View file @ f438ee21
......@@ -90,7 +90,7 @@ static const struct insn insn_table[insn_invalid] = {
RS | RT | RD},
[insn_dmtc0] = {M(cop0_op, dmtc_op, 0, 0, 0, 0), RT | RD | SET},
[insn_dmultu] = {M(spec_op, 0, 0, 0, 0, dmultu_op), RS | RT},
[insn_dmulu] = {M(spec_op, 0, 0, 0, dmult_dmul_op, dmultu_op),
[insn_dmulu] = {M(spec_op, 0, 0, 0, dmultu_dmulu_op, dmultu_op),
RS | RT | RD},
[insn_drotr] = {M(spec_op, 1, 0, 0, 0, dsrl_op), RT | RD | RE},
[insn_drotr32] = {M(spec_op, 1, 0, 0, 0, dsrl32_op), RT | RD | RE},
......@@ -150,6 +150,8 @@ static const struct insn insn_table[insn_invalid] = {
[insn_mtlo] = {M(spec_op, 0, 0, 0, 0, mtlo_op), RS},
[insn_mulu] = {M(spec_op, 0, 0, 0, multu_mulu_op, multu_op),
RS | RT | RD},
[insn_muhu] = {M(spec_op, 0, 0, 0, multu_muhu_op, multu_op),
RS | RT | RD},
#ifndef CONFIG_CPU_MIPSR6
[insn_mul] = {M(spec2_op, 0, 0, 0, 0, mul_op), RS | RT | RD},
#else
......
arch/mips/mm/uasm.c
View file @ f438ee21
......@@ -59,7 +59,7 @@ enum opcode {
insn_lddir, insn_ldpte, insn_ldx, insn_lh, insn_lhu, insn_ll, insn_lld,
insn_lui, insn_lw, insn_lwu, insn_lwx, insn_mfc0, insn_mfhc0, insn_mfhi,
insn_mflo, insn_modu, insn_movn, insn_movz, insn_mtc0, insn_mthc0,
insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu, insn_nor,
insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu, insn_muhu, insn_nor,
insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sb, insn_sc,
insn_scd, insn_seleqz, insn_selnez, insn_sd, insn_sh, insn_sll,
insn_sllv, insn_slt, insn_slti, insn_sltiu, insn_sltu, insn_sra,
......@@ -344,6 +344,7 @@ I_u1(_mtlo)
I_u3u1u2(_mul)
I_u1u2(_multu)
I_u3u1u2(_mulu)
I_u3u1u2(_muhu)
I_u3u1u2(_nor)
I_u3u1u2(_or)
I_u2u1u3(_ori)
......
arch/mips/net/Makefile
View file @ f438ee21
......@@ -2,4 +2,10 @@
# MIPS networking code
obj-$(CONFIG_MIPS_CBPF_JIT) += bpf_jit.o bpf_jit_asm.o
obj-$(CONFIG_MIPS_EBPF_JIT) += ebpf_jit.o
obj-$(CONFIG_MIPS_EBPF_JIT) += bpf_jit_comp.o
ifeq ($(CONFIG_32BIT),y)
obj-$(CONFIG_MIPS_EBPF_JIT) += bpf_jit_comp32.o
else
obj-$(CONFIG_MIPS_EBPF_JIT) += bpf_jit_comp64.o
endif
arch/mips/net/bpf_jit.c deleted 100644 → 0
View file @ 9d057872
/*
* Just-In-Time compiler for BPF filters on MIPS
*
* Copyright (c) 2014 Imagination Technologies Ltd.
* Author: Markos Chandras <markos.chandras@imgtec.com>
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; version 2 of the License.
*/
#include <linux/bitops.h>
#include <linux/compiler.h>
#include <linux/errno.h>
#include <linux/filter.h>
#include <linux/if_vlan.h>
#include <linux/moduleloader.h>
#include <linux/netdevice.h>
#include <linux/string.h>
#include <linux/slab.h>
#include <linux/types.h>
#include <asm/asm.h>
#include <asm/bitops.h>
#include <asm/cacheflush.h>
#include <asm/cpu-features.h>
#include <asm/uasm.h>
#include "bpf_jit.h"
/* ABI
* r_skb_hl SKB header length
* r_data SKB data pointer
* r_off Offset
* r_A BPF register A
* r_X BPF register X
* r_skb *skb
* r_M *scratch memory
* r_skb_len SKB length
*
* On entry (*bpf_func)(*skb, *filter)
* a0 = MIPS_R_A0 = skb;
* a1 = MIPS_R_A1 = filter;
*
* Stack
* ...
* M[15]
* M[14]
* M[13]
* ...
* M[0] <-- r_M
* saved reg k-1
* saved reg k-2
* ...
* saved reg 0 <-- r_sp
* <no argument area>
*
* Packet layout
*
* <--------------------- len ------------------------>
* <--skb-len(r_skb_hl)-->< ----- skb->data_len ------>
* ----------------------------------------------------
* | skb->data |
* ----------------------------------------------------
*/
#define ptr typeof(unsigned long)
#define SCRATCH_OFF(k) (4 * (k))
/* JIT flags */
#define SEEN_CALL (1 << BPF_MEMWORDS)
#define SEEN_SREG_SFT (BPF_MEMWORDS + 1)
#define SEEN_SREG_BASE (1 << SEEN_SREG_SFT)
#define SEEN_SREG(x) (SEEN_SREG_BASE << (x))
#define SEEN_OFF SEEN_SREG(2)
#define SEEN_A SEEN_SREG(3)
#define SEEN_X SEEN_SREG(4)
#define SEEN_SKB SEEN_SREG(5)
#define SEEN_MEM SEEN_SREG(6)
/* SEEN_SK_DATA also implies skb_hl an skb_len */
#define SEEN_SKB_DATA (SEEN_SREG(7) | SEEN_SREG(1) | SEEN_SREG(0))
/* Arguments used by JIT */
#define ARGS_USED_BY_JIT 2 /* only applicable to 64-bit */
#define SBIT(x) (1 << (x)) /* Signed version of BIT() */
/**
* struct jit_ctx - JIT context
* @skf: The sk_filter
* @prologue_bytes: Number of bytes for prologue
* @idx: Instruction index
* @flags: JIT flags
* @offsets: Instruction offsets
* @target: Memory location for the compiled filter
*/
struct jit_ctx {
const struct bpf_prog *skf;
unsigned int prologue_bytes;
u32 idx;
u32 flags;
u32 *offsets;
u32 *target;
};
static inline int optimize_div(u32 *k)
{
/* power of 2 divides can be implemented with right shift */
if (!(*k & (*k-1))) {
*k = ilog2(*k);
return 1;
}
return 0;
}
static inline void emit_jit_reg_move(ptr dst, ptr src, struct jit_ctx *ctx);
/* Simply emit the instruction if the JIT memory space has been allocated */
#define emit_instr(ctx, func, ...) \
do { \
if ((ctx)->target != NULL) { \
u32 *p = &(ctx)->target[ctx->idx]; \
uasm_i_##func(&p, ##__VA_ARGS__); \
} \
(ctx)->idx++; \
} while (0)
/*
* Similar to emit_instr but it must be used when we need to emit
* 32-bit or 64-bit instructions
*/
#define emit_long_instr(ctx, func, ...) \
do { \
if ((ctx)->target != NULL) { \
u32 *p = &(ctx)->target[ctx->idx]; \
UASM_i_##func(&p, ##__VA_ARGS__); \
} \
(ctx)->idx++; \
} while (0)
/* Determine if immediate is within the 16-bit signed range */
static inline bool is_range16(s32 imm)
{
return !(imm >= SBIT(15) || imm < -SBIT(15));
}
static inline void emit_addu(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, addu, dst, src1, src2);
}
static inline void emit_nop(struct jit_ctx *ctx)
{
emit_instr(ctx, nop);
}
/* Load a u32 immediate to a register */
static inline void emit_load_imm(unsigned int dst, u32 imm, struct jit_ctx *ctx)
{
if (ctx->target != NULL) {
/* addiu can only handle s16 */
if (!is_range16(imm)) {
u32 *p = &ctx->target[ctx->idx];
uasm_i_lui(&p, r_tmp_imm, (s32)imm >> 16);
p = &ctx->target[ctx->idx + 1];
uasm_i_ori(&p, dst, r_tmp_imm, imm & 0xffff);
} else {
u32 *p = &ctx->target[ctx->idx];
uasm_i_addiu(&p, dst, r_zero, imm);
}
}
ctx->idx++;
if (!is_range16(imm))
ctx->idx++;
}
static inline void emit_or(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, or, dst, src1, src2);
}
static inline void emit_ori(unsigned int dst, unsigned src, u32 imm,
struct jit_ctx *ctx)
{
if (imm >= BIT(16)) {
emit_load_imm(r_tmp, imm, ctx);
emit_or(dst, src, r_tmp, ctx);
} else {
emit_instr(ctx, ori, dst, src, imm);
}
}
static inline void emit_daddiu(unsigned int dst, unsigned int src,
int imm, struct jit_ctx *ctx)
{
/*
* Only used for stack, so the imm is relatively small
* and it fits in 15-bits
*/
emit_instr(ctx, daddiu, dst, src, imm);
}
static inline void emit_addiu(unsigned int dst, unsigned int src,
u32 imm, struct jit_ctx *ctx)
{
if (!is_range16(imm)) {
emit_load_imm(r_tmp, imm, ctx);
emit_addu(dst, r_tmp, src, ctx);
} else {
emit_instr(ctx, addiu, dst, src, imm);
}
}
static inline void emit_and(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, and, dst, src1, src2);
}
static inline void emit_andi(unsigned int dst, unsigned int src,
u32 imm, struct jit_ctx *ctx)
{
/* If imm does not fit in u16 then load it to register */
if (imm >= BIT(16)) {
emit_load_imm(r_tmp, imm, ctx);
emit_and(dst, src, r_tmp, ctx);
} else {
emit_instr(ctx, andi, dst, src, imm);
}
}
static inline void emit_xor(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, xor, dst, src1, src2);
}
static inline void emit_xori(ptr dst, ptr src, u32 imm, struct jit_ctx *ctx)
{
/* If imm does not fit in u16 then load it to register */
if (imm >= BIT(16)) {
emit_load_imm(r_tmp, imm, ctx);
emit_xor(dst, src, r_tmp, ctx);
} else {
emit_instr(ctx, xori, dst, src, imm);
}
}
static inline void emit_stack_offset(int offset, struct jit_ctx *ctx)
{
emit_long_instr(ctx, ADDIU, r_sp, r_sp, offset);
}
static inline void emit_subu(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, subu, dst, src1, src2);
}
static inline void emit_neg(unsigned int reg, struct jit_ctx *ctx)
{
emit_subu(reg, r_zero, reg, ctx);
}
static inline void emit_sllv(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
emit_instr(ctx, sllv, dst, src, sa);
}
static inline void emit_sll(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
/* sa is 5-bits long */
if (sa >= BIT(5))
/* Shifting >= 32 results in zero */
emit_jit_reg_move(dst, r_zero, ctx);
else
emit_instr(ctx, sll, dst, src, sa);
}
static inline void emit_srlv(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
emit_instr(ctx, srlv, dst, src, sa);
}
static inline void emit_srl(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
/* sa is 5-bits long */
if (sa >= BIT(5))
/* Shifting >= 32 results in zero */
emit_jit_reg_move(dst, r_zero, ctx);
else
emit_instr(ctx, srl, dst, src, sa);
}
static inline void emit_slt(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, slt, dst, src1, src2);
}
static inline void emit_sltu(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, sltu, dst, src1, src2);
}
static inline void emit_sltiu(unsigned dst, unsigned int src,
unsigned int imm, struct jit_ctx *ctx)
{
/* 16 bit immediate */
if (!is_range16((s32)imm)) {
emit_load_imm(r_tmp, imm, ctx);
emit_sltu(dst, src, r_tmp, ctx);
} else {
emit_instr(ctx, sltiu, dst, src, imm);
}
}
/* Store register on the stack */
static inline void emit_store_stack_reg(ptr reg, ptr base,
unsigned int offset,
struct jit_ctx *ctx)
{
emit_long_instr(ctx, SW, reg, offset, base);
}
static inline void emit_store(ptr reg, ptr base, unsigned int offset,
struct jit_ctx *ctx)
{
emit_instr(ctx, sw, reg, offset, base);
}
static inline void emit_load_stack_reg(ptr reg, ptr base,
unsigned int offset,
struct jit_ctx *ctx)
{
emit_long_instr(ctx, LW, reg, offset, base);
}
static inline void emit_load(unsigned int reg, unsigned int base,
unsigned int offset, struct jit_ctx *ctx)
{
emit_instr(ctx, lw, reg, offset, base);
}
static inline void emit_load_byte(unsigned int reg, unsigned int base,
unsigned int offset, struct jit_ctx *ctx)
{
emit_instr(ctx, lb, reg, offset, base);
}
static inline void emit_half_load(unsigned int reg, unsigned int base,
unsigned int offset, struct jit_ctx *ctx)
{
emit_instr(ctx, lh, reg, offset, base);
}
static inline void emit_half_load_unsigned(unsigned int reg, unsigned int base,
unsigned int offset, struct jit_ctx *ctx)
{
emit_instr(ctx, lhu, reg, offset, base);
}
static inline void emit_mul(unsigned int dst, unsigned int src1,
unsigned int src2, struct jit_ctx *ctx)
{
emit_instr(ctx, mul, dst, src1, src2);
}
static inline void emit_div(unsigned int dst, unsigned int src,
struct jit_ctx *ctx)
{
if (ctx->target != NULL) {
u32 *p = &ctx->target[ctx->idx];
uasm_i_divu(&p, dst, src);
p = &ctx->target[ctx->idx + 1];
uasm_i_mflo(&p, dst);
}
ctx->idx += 2; /* 2 insts */
}
static inline void emit_mod(unsigned int dst, unsigned int src,
struct jit_ctx *ctx)
{
if (ctx->target != NULL) {
u32 *p = &ctx->target[ctx->idx];
uasm_i_divu(&p, dst, src);
p = &ctx->target[ctx->idx + 1];
uasm_i_mfhi(&p, dst);
}
ctx->idx += 2; /* 2 insts */
}
static inline void emit_dsll(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
emit_instr(ctx, dsll, dst, src, sa);
}
static inline void emit_dsrl32(unsigned int dst, unsigned int src,
unsigned int sa, struct jit_ctx *ctx)
{
emit_instr(ctx, dsrl32, dst, src, sa);
}
static inline void emit_wsbh(unsigned int dst, unsigned int src,
struct jit_ctx *ctx)
{
emit_instr(ctx, wsbh, dst, src);
}
/* load pointer to register */
static inline void emit_load_ptr(unsigned int dst, unsigned int src,
int imm, struct jit_ctx *ctx)
{
/* src contains the base addr of the 32/64-pointer */
emit_long_instr(ctx, LW, dst, imm, src);
}
/* load a function pointer to register */
static inline void emit_load_func(unsigned int reg, ptr imm,
struct jit_ctx *ctx)
{
if (IS_ENABLED(CONFIG_64BIT)) {
/* At this point imm is always 64-bit */
emit_load_imm(r_tmp, (u64)imm >> 32, ctx);
emit_dsll(r_tmp_imm, r_tmp, 16, ctx); /* left shift by 16 */
emit_ori(r_tmp, r_tmp_imm, (imm >> 16) & 0xffff, ctx);
emit_dsll(r_tmp_imm, r_tmp, 16, ctx); /* left shift by 16 */
emit_ori(reg, r_tmp_imm, imm & 0xffff, ctx);
} else {
emit_load_imm(reg, imm, ctx);
}
}
/* Move to real MIPS register */
static inline void emit_reg_move(ptr dst, ptr src, struct jit_ctx *ctx)
{
emit_long_instr(ctx, ADDU, dst, src, r_zero);
}
/* Move to JIT (32-bit) register */
static inline void emit_jit_reg_move(ptr dst, ptr src, struct jit_ctx *ctx)
{
emit_addu(dst, src, r_zero, ctx);
}
/* Compute the immediate value for PC-relative branches. */
static inline u32 b_imm(unsigned int tgt, struct jit_ctx *ctx)
{
if (ctx->target == NULL)
return 0;
/*
* We want a pc-relative branch. We only do forward branches
* so tgt is always after pc. tgt is the instruction offset
* we want to jump to.
* Branch on MIPS:
* I: target_offset <- sign_extend(offset)
* I+1: PC += target_offset (delay slot)
*
* ctx->idx currently points to the branch instruction
* but the offset is added to the delay slot so we need
* to subtract 4.
*/
return ctx->offsets[tgt] -
(ctx->idx * 4 - ctx->prologue_bytes) - 4;
}
static inline void emit_bcond(int cond, unsigned int reg1, unsigned int reg2,
unsigned int imm, struct jit_ctx *ctx)
{
if (ctx->target != NULL) {
u32 *p = &ctx->target[ctx->idx];
switch (cond) {
case MIPS_COND_EQ:
uasm_i_beq(&p, reg1, reg2, imm);
break;
case MIPS_COND_NE:
uasm_i_bne(&p, reg1, reg2, imm);
break;
case MIPS_COND_ALL:
uasm_i_b(&p, imm);
break;
default:
pr_warn("%s: Unhandled branch conditional: %d\n",
__func__, cond);
}
}
ctx->idx++;
}
static inline void emit_b(unsigned int imm, struct jit_ctx *ctx)
{
emit_bcond(MIPS_COND_ALL, r_zero, r_zero, imm, ctx);
}
static inline void emit_jalr(unsigned int link, unsigned int reg,
struct jit_ctx *ctx)
{
emit_instr(ctx, jalr, link, reg);
}
static inline void emit_jr(unsigned int reg, struct jit_ctx *ctx)
{
emit_instr(ctx, jr, reg);
}
static inline u16 align_sp(unsigned int num)
{
/* Double word alignment for 32-bit, quadword for 64-bit */
unsigned int align = IS_ENABLED(CONFIG_64BIT) ? 16 : 8;
num = (num + (align - 1)) & -align;
return num;
}
static void save_bpf_jit_regs(struct jit_ctx *ctx, unsigned offset)
{
int i = 0, real_off = 0;
u32 sflags, tmp_flags;
/* Adjust the stack pointer */
if (offset)
emit_stack_offset(-align_sp(offset), ctx);
tmp_flags = sflags = ctx->flags >> SEEN_SREG_SFT;
/* sflags is essentially a bitmap */
while (tmp_flags) {
if ((sflags >> i) & 0x1) {
emit_store_stack_reg(MIPS_R_S0 + i, r_sp, real_off,
ctx);
real_off += SZREG;
}
i++;
tmp_flags >>= 1;
}
/* save return address */
if (ctx->flags & SEEN_CALL) {
emit_store_stack_reg(r_ra, r_sp, real_off, ctx);
real_off += SZREG;
}
/* Setup r_M leaving the alignment gap if necessary */
if (ctx->flags & SEEN_MEM) {
if (real_off % (SZREG * 2))
real_off += SZREG;
emit_long_instr(ctx, ADDIU, r_M, r_sp, real_off);
}
}
static void restore_bpf_jit_regs(struct jit_ctx *ctx,
unsigned int offset)
{
int i, real_off = 0;
u32 sflags, tmp_flags;
tmp_flags = sflags = ctx->flags >> SEEN_SREG_SFT;
/* sflags is a bitmap */
i = 0;
while (tmp_flags) {
if ((sflags >> i) & 0x1) {
emit_load_stack_reg(MIPS_R_S0 + i, r_sp, real_off,
ctx);
real_off += SZREG;
}
i++;
tmp_flags >>= 1;
}
/* restore return address */
if (ctx->flags & SEEN_CALL)
emit_load_stack_reg(r_ra, r_sp, real_off, ctx);
/* Restore the sp and discard the scrach memory */
if (offset)
emit_stack_offset(align_sp(offset), ctx);
}
static unsigned int get_stack_depth(struct jit_ctx *ctx)
{
int sp_off = 0;
/* How may s* regs do we need to preserved? */
sp_off += hweight32(ctx->flags >> SEEN_SREG_SFT) * SZREG;
if (ctx->flags & SEEN_MEM)
sp_off += 4 * BPF_MEMWORDS; /* BPF_MEMWORDS are 32-bit */
if (ctx->flags & SEEN_CALL)
sp_off += SZREG; /* Space for our ra register */
return sp_off;
}
static void build_prologue(struct jit_ctx *ctx)
{
int sp_off;
/* Calculate the total offset for the stack pointer */
sp_off = get_stack_depth(ctx);
save_bpf_jit_regs(ctx, sp_off);
if (ctx->flags & SEEN_SKB)
emit_reg_move(r_skb, MIPS_R_A0, ctx);
if (ctx->flags & SEEN_SKB_DATA) {
/* Load packet length */
emit_load(r_skb_len, r_skb, offsetof(struct sk_buff, len),
ctx);
emit_load(r_tmp, r_skb, offsetof(struct sk_buff, data_len),
ctx);
/* Load the data pointer */
emit_load_ptr(r_skb_data, r_skb,
offsetof(struct sk_buff, data), ctx);
/* Load the header length */
emit_subu(r_skb_hl, r_skb_len, r_tmp, ctx);
}
if (ctx->flags & SEEN_X)
emit_jit_reg_move(r_X, r_zero, ctx);
/*
* Do not leak kernel data to userspace, we only need to clear
* r_A if it is ever used. In fact if it is never used, we
* will not save/restore it, so clearing it in this case would
* corrupt the state of the caller.
*/
if (bpf_needs_clear_a(&ctx->skf->insns[0]) &&
(ctx->flags & SEEN_A))
emit_jit_reg_move(r_A, r_zero, ctx);
}
static void build_epilogue(struct jit_ctx *ctx)
{
unsigned int sp_off;
/* Calculate the total offset for the stack pointer */
sp_off = get_stack_depth(ctx);
restore_bpf_jit_regs(ctx, sp_off);
/* Return */
emit_jr(r_ra, ctx);
emit_nop(ctx);
}
#define CHOOSE_LOAD_FUNC(K, func) \
((int)K < 0 ? ((int)K >= SKF_LL_OFF ? func##_negative : func) : \
func##_positive)
static bool is_bad_offset(int b_off)
{
return b_off > 0x1ffff || b_off < -0x20000;
}
static int build_body(struct jit_ctx *ctx)
{
const struct bpf_prog *prog = ctx->skf;
const struct sock_filter *inst;
unsigned int i, off, condt;
u32 k, b_off __maybe_unused;
u8 (*sk_load_func)(unsigned long *skb, int offset);
for (i = 0; i < prog->len; i++) {
u16 code;
inst = &(prog->insns[i]);
pr_debug("%s: code->0x%02x, jt->0x%x, jf->0x%x, k->0x%x\n",
__func__, inst->code, inst->jt, inst->jf, inst->k);
k = inst->k;
code = bpf_anc_helper(inst);
if (ctx->target == NULL)
ctx->offsets[i] = ctx->idx * 4;
switch (code) {
case BPF_LD | BPF_IMM:
/* A <- k ==> li r_A, k */
ctx->flags |= SEEN_A;
emit_load_imm(r_A, k, ctx);
break;
case BPF_LD | BPF_W | BPF_LEN:
BUILD_BUG_ON(sizeof_field(struct sk_buff, len) != 4);
/* A <- len ==> lw r_A, offset(skb) */
ctx->flags |= SEEN_SKB | SEEN_A;
off = offsetof(struct sk_buff, len);
emit_load(r_A, r_skb, off, ctx);
break;
case BPF_LD | BPF_MEM:
/* A <- M[k] ==> lw r_A, offset(M) */
ctx->flags |= SEEN_MEM | SEEN_A;
emit_load(r_A, r_M, SCRATCH_OFF(k), ctx);
break;
case BPF_LD | BPF_W | BPF_ABS:
/* A <- P[k:4] */
sk_load_func = CHOOSE_LOAD_FUNC(k, sk_load_word);
goto load;
case BPF_LD | BPF_H | BPF_ABS:
/* A <- P[k:2] */
sk_load_func = CHOOSE_LOAD_FUNC(k, sk_load_half);
goto load;
case BPF_LD | BPF_B | BPF_ABS:
/* A <- P[k:1] */
sk_load_func = CHOOSE_LOAD_FUNC(k, sk_load_byte);
load:
emit_load_imm(r_off, k, ctx);
load_common:
ctx->flags |= SEEN_CALL | SEEN_OFF |
SEEN_SKB | SEEN_A | SEEN_SKB_DATA;
emit_load_func(r_s0, (ptr)sk_load_func, ctx);
emit_reg_move(MIPS_R_A0, r_skb, ctx);
emit_jalr(MIPS_R_RA, r_s0, ctx);
/* Load second argument to delay slot */
emit_reg_move(MIPS_R_A1, r_off, ctx);
/* Check the error value */
emit_bcond(MIPS_COND_EQ, r_ret, 0, b_imm(i + 1, ctx),
ctx);
/* Load return register on DS for failures */
emit_reg_move(r_ret, r_zero, ctx);
/* Return with error */
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_b(b_off, ctx);
emit_nop(ctx);
break;
case BPF_LD | BPF_W | BPF_IND:
/* A <- P[X + k:4] */
sk_load_func = sk_load_word;
goto load_ind;
case BPF_LD | BPF_H | BPF_IND:
/* A <- P[X + k:2] */
sk_load_func = sk_load_half;
goto load_ind;
case BPF_LD | BPF_B | BPF_IND:
/* A <- P[X + k:1] */
sk_load_func = sk_load_byte;
load_ind:
ctx->flags |= SEEN_OFF | SEEN_X;
emit_addiu(r_off, r_X, k, ctx);
goto load_common;
case BPF_LDX | BPF_IMM:
/* X <- k */
ctx->flags |= SEEN_X;
emit_load_imm(r_X, k, ctx);
break;
case BPF_LDX | BPF_MEM:
/* X <- M[k] */
ctx->flags |= SEEN_X | SEEN_MEM;
emit_load(r_X, r_M, SCRATCH_OFF(k), ctx);
break;
case BPF_LDX | BPF_W | BPF_LEN:
/* X <- len */
ctx->flags |= SEEN_X | SEEN_SKB;
off = offsetof(struct sk_buff, len);
emit_load(r_X, r_skb, off, ctx);
break;
case BPF_LDX | BPF_B | BPF_MSH:
/* X <- 4 * (P[k:1] & 0xf) */
ctx->flags |= SEEN_X | SEEN_CALL | SEEN_SKB;
/* Load offset to a1 */
emit_load_func(r_s0, (ptr)sk_load_byte, ctx);
/*
* This may emit two instructions so it may not fit
* in the delay slot. So use a0 in the delay slot.
*/
emit_load_imm(MIPS_R_A1, k, ctx);
emit_jalr(MIPS_R_RA, r_s0, ctx);
emit_reg_move(MIPS_R_A0, r_skb, ctx); /* delay slot */
/* Check the error value */
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_bcond(MIPS_COND_NE, r_ret, 0, b_off, ctx);
emit_reg_move(r_ret, r_zero, ctx);
/* We are good */
/* X <- P[1:K] & 0xf */
emit_andi(r_X, r_A, 0xf, ctx);
/* X << 2 */
emit_b(b_imm(i + 1, ctx), ctx);
emit_sll(r_X, r_X, 2, ctx); /* delay slot */
break;
case BPF_ST:
/* M[k] <- A */
ctx->flags |= SEEN_MEM | SEEN_A;
emit_store(r_A, r_M, SCRATCH_OFF(k), ctx);
break;
case BPF_STX:
/* M[k] <- X */
ctx->flags |= SEEN_MEM | SEEN_X;
emit_store(r_X, r_M, SCRATCH_OFF(k), ctx);
break;
case BPF_ALU | BPF_ADD | BPF_K:
/* A += K */
ctx->flags |= SEEN_A;
emit_addiu(r_A, r_A, k, ctx);
break;
case BPF_ALU | BPF_ADD | BPF_X:
/* A += X */
ctx->flags |= SEEN_A | SEEN_X;
emit_addu(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_SUB | BPF_K:
/* A -= K */
ctx->flags |= SEEN_A;
emit_addiu(r_A, r_A, -k, ctx);
break;
case BPF_ALU | BPF_SUB | BPF_X:
/* A -= X */
ctx->flags |= SEEN_A | SEEN_X;
emit_subu(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_MUL | BPF_K:
/* A *= K */
/* Load K to scratch register before MUL */
ctx->flags |= SEEN_A;
emit_load_imm(r_s0, k, ctx);
emit_mul(r_A, r_A, r_s0, ctx);
break;
case BPF_ALU | BPF_MUL | BPF_X:
/* A *= X */
ctx->flags |= SEEN_A | SEEN_X;
emit_mul(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_DIV | BPF_K:
/* A /= k */
if (k == 1)
break;
if (optimize_div(&k)) {
ctx->flags |= SEEN_A;
emit_srl(r_A, r_A, k, ctx);
break;
}
ctx->flags |= SEEN_A;
emit_load_imm(r_s0, k, ctx);
emit_div(r_A, r_s0, ctx);
break;
case BPF_ALU | BPF_MOD | BPF_K:
/* A %= k */
if (k == 1) {
ctx->flags |= SEEN_A;
emit_jit_reg_move(r_A, r_zero, ctx);
} else {
ctx->flags |= SEEN_A;
emit_load_imm(r_s0, k, ctx);
emit_mod(r_A, r_s0, ctx);
}
break;
case BPF_ALU | BPF_DIV | BPF_X:
/* A /= X */
ctx->flags |= SEEN_X | SEEN_A;
/* Check if r_X is zero */
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_bcond(MIPS_COND_EQ, r_X, r_zero, b_off, ctx);
emit_load_imm(r_ret, 0, ctx); /* delay slot */
emit_div(r_A, r_X, ctx);
break;
case BPF_ALU | BPF_MOD | BPF_X:
/* A %= X */
ctx->flags |= SEEN_X | SEEN_A;
/* Check if r_X is zero */
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_bcond(MIPS_COND_EQ, r_X, r_zero, b_off, ctx);
emit_load_imm(r_ret, 0, ctx); /* delay slot */
emit_mod(r_A, r_X, ctx);
break;
case BPF_ALU | BPF_OR | BPF_K:
/* A |= K */
ctx->flags |= SEEN_A;
emit_ori(r_A, r_A, k, ctx);
break;
case BPF_ALU | BPF_OR | BPF_X:
/* A |= X */
ctx->flags |= SEEN_A;
emit_ori(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_XOR | BPF_K:
/* A ^= k */
ctx->flags |= SEEN_A;
emit_xori(r_A, r_A, k, ctx);
break;
case BPF_ANC | SKF_AD_ALU_XOR_X:
case BPF_ALU | BPF_XOR | BPF_X:
/* A ^= X */
ctx->flags |= SEEN_A;
emit_xor(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_AND | BPF_K:
/* A &= K */
ctx->flags |= SEEN_A;
emit_andi(r_A, r_A, k, ctx);
break;
case BPF_ALU | BPF_AND | BPF_X:
/* A &= X */
ctx->flags |= SEEN_A | SEEN_X;
emit_and(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_LSH | BPF_K:
/* A <<= K */
ctx->flags |= SEEN_A;
emit_sll(r_A, r_A, k, ctx);
break;
case BPF_ALU | BPF_LSH | BPF_X:
/* A <<= X */
ctx->flags |= SEEN_A | SEEN_X;
emit_sllv(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_RSH | BPF_K:
/* A >>= K */
ctx->flags |= SEEN_A;
emit_srl(r_A, r_A, k, ctx);
break;
case BPF_ALU | BPF_RSH | BPF_X:
ctx->flags |= SEEN_A | SEEN_X;
emit_srlv(r_A, r_A, r_X, ctx);
break;
case BPF_ALU | BPF_NEG:
/* A = -A */
ctx->flags |= SEEN_A;
emit_neg(r_A, ctx);
break;
case BPF_JMP | BPF_JA:
/* pc += K */
b_off = b_imm(i + k + 1, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_b(b_off, ctx);
emit_nop(ctx);
break;
case BPF_JMP | BPF_JEQ | BPF_K:
/* pc += ( A == K ) ? pc->jt : pc->jf */
condt = MIPS_COND_EQ | MIPS_COND_K;
goto jmp_cmp;
case BPF_JMP | BPF_JEQ | BPF_X:
ctx->flags |= SEEN_X;
/* pc += ( A == X ) ? pc->jt : pc->jf */
condt = MIPS_COND_EQ | MIPS_COND_X;
goto jmp_cmp;
case BPF_JMP | BPF_JGE | BPF_K:
/* pc += ( A >= K ) ? pc->jt : pc->jf */
condt = MIPS_COND_GE | MIPS_COND_K;
goto jmp_cmp;
case BPF_JMP | BPF_JGE | BPF_X:
ctx->flags |= SEEN_X;
/* pc += ( A >= X ) ? pc->jt : pc->jf */
condt = MIPS_COND_GE | MIPS_COND_X;
goto jmp_cmp;
case BPF_JMP | BPF_JGT | BPF_K:
/* pc += ( A > K ) ? pc->jt : pc->jf */
condt = MIPS_COND_GT | MIPS_COND_K;
goto jmp_cmp;
case BPF_JMP | BPF_JGT | BPF_X:
ctx->flags |= SEEN_X;
/* pc += ( A > X ) ? pc->jt : pc->jf */
condt = MIPS_COND_GT | MIPS_COND_X;
jmp_cmp:
/* Greater or Equal */
if ((condt & MIPS_COND_GE) ||
(condt & MIPS_COND_GT)) {
if (condt & MIPS_COND_K) { /* K */
ctx->flags |= SEEN_A;
emit_sltiu(r_s0, r_A, k, ctx);
} else { /* X */
ctx->flags |= SEEN_A |
SEEN_X;
emit_sltu(r_s0, r_A, r_X, ctx);
}
/* A < (K|X) ? r_scrach = 1 */
b_off = b_imm(i + inst->jf + 1, ctx);
emit_bcond(MIPS_COND_NE, r_s0, r_zero, b_off,
ctx);
emit_nop(ctx);
/* A > (K|X) ? scratch = 0 */
if (condt & MIPS_COND_GT) {
/* Checking for equality */
ctx->flags |= SEEN_A | SEEN_X;
if (condt & MIPS_COND_K)
emit_load_imm(r_s0, k, ctx);
else
emit_jit_reg_move(r_s0, r_X,
ctx);
b_off = b_imm(i + inst->jf + 1, ctx);
emit_bcond(MIPS_COND_EQ, r_A, r_s0,
b_off, ctx);
emit_nop(ctx);
/* Finally, A > K|X */
b_off = b_imm(i + inst->jt + 1, ctx);
emit_b(b_off, ctx);
emit_nop(ctx);
} else {
/* A >= (K|X) so jump */
b_off = b_imm(i + inst->jt + 1, ctx);
emit_b(b_off, ctx);
emit_nop(ctx);
}
} else {
/* A == K|X */
if (condt & MIPS_COND_K) { /* K */
ctx->flags |= SEEN_A;
emit_load_imm(r_s0, k, ctx);
/* jump true */
b_off = b_imm(i + inst->jt + 1, ctx);
emit_bcond(MIPS_COND_EQ, r_A, r_s0,
b_off, ctx);
emit_nop(ctx);
/* jump false */
b_off = b_imm(i + inst->jf + 1,
ctx);
emit_bcond(MIPS_COND_NE, r_A, r_s0,
b_off, ctx);
emit_nop(ctx);
} else { /* X */
/* jump true */
ctx->flags |= SEEN_A | SEEN_X;
b_off = b_imm(i + inst->jt + 1,
ctx);
emit_bcond(MIPS_COND_EQ, r_A, r_X,
b_off, ctx);
emit_nop(ctx);
/* jump false */
b_off = b_imm(i + inst->jf + 1, ctx);
emit_bcond(MIPS_COND_NE, r_A, r_X,
b_off, ctx);
emit_nop(ctx);
}
}
break;
case BPF_JMP | BPF_JSET | BPF_K:
ctx->flags |= SEEN_A;
/* pc += (A & K) ? pc -> jt : pc -> jf */
emit_load_imm(r_s1, k, ctx);
emit_and(r_s0, r_A, r_s1, ctx);
/* jump true */
b_off = b_imm(i + inst->jt + 1, ctx);
emit_bcond(MIPS_COND_NE, r_s0, r_zero, b_off, ctx);
emit_nop(ctx);
/* jump false */
b_off = b_imm(i + inst->jf + 1, ctx);
emit_b(b_off, ctx);
emit_nop(ctx);
break;
case BPF_JMP | BPF_JSET | BPF_X:
ctx->flags |= SEEN_X | SEEN_A;
/* pc += (A & X) ? pc -> jt : pc -> jf */
emit_and(r_s0, r_A, r_X, ctx);
/* jump true */
b_off = b_imm(i + inst->jt + 1, ctx);
emit_bcond(MIPS_COND_NE, r_s0, r_zero, b_off, ctx);
emit_nop(ctx);
/* jump false */
b_off = b_imm(i + inst->jf + 1, ctx);
emit_b(b_off, ctx);
emit_nop(ctx);
break;
case BPF_RET | BPF_A:
ctx->flags |= SEEN_A;
if (i != prog->len - 1) {
/*
* If this is not the last instruction
* then jump to the epilogue
*/
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_b(b_off, ctx);
}
emit_reg_move(r_ret, r_A, ctx); /* delay slot */
break;
case BPF_RET | BPF_K:
/*
* It can emit two instructions so it does not fit on
* the delay slot.
*/
emit_load_imm(r_ret, k, ctx);
if (i != prog->len - 1) {
/*
* If this is not the last instruction
* then jump to the epilogue
*/
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_b(b_off, ctx);
emit_nop(ctx);
}
break;
case BPF_MISC | BPF_TAX:
/* X = A */
ctx->flags |= SEEN_X | SEEN_A;
emit_jit_reg_move(r_X, r_A, ctx);
break;
case BPF_MISC | BPF_TXA:
/* A = X */
ctx->flags |= SEEN_A | SEEN_X;
emit_jit_reg_move(r_A, r_X, ctx);
break;
/* AUX */
case BPF_ANC | SKF_AD_PROTOCOL:
/* A = ntohs(skb->protocol */
ctx->flags |= SEEN_SKB | SEEN_OFF | SEEN_A;
BUILD_BUG_ON(sizeof_field(struct sk_buff,
protocol) != 2);
off = offsetof(struct sk_buff, protocol);
emit_half_load(r_A, r_skb, off, ctx);
#ifdef CONFIG_CPU_LITTLE_ENDIAN
/* This needs little endian fixup */
if (cpu_has_wsbh) {
/* R2 and later have the wsbh instruction */
emit_wsbh(r_A, r_A, ctx);
} else {
/* Get first byte */
emit_andi(r_tmp_imm, r_A, 0xff, ctx);
/* Shift it */
emit_sll(r_tmp, r_tmp_imm, 8, ctx);
/* Get second byte */
emit_srl(r_tmp_imm, r_A, 8, ctx);
emit_andi(r_tmp_imm, r_tmp_imm, 0xff, ctx);
/* Put everyting together in r_A */
emit_or(r_A, r_tmp, r_tmp_imm, ctx);
}
#endif
break;
case BPF_ANC | SKF_AD_CPU:
ctx->flags |= SEEN_A | SEEN_OFF;
/* A = current_thread_info()->cpu */
BUILD_BUG_ON(sizeof_field(struct thread_info,
cpu) != 4);
off = offsetof(struct thread_info, cpu);
/* $28/gp points to the thread_info struct */
emit_load(r_A, 28, off, ctx);
break;
case BPF_ANC | SKF_AD_IFINDEX:
/* A = skb->dev->ifindex */
case BPF_ANC | SKF_AD_HATYPE:
/* A = skb->dev->type */
ctx->flags |= SEEN_SKB | SEEN_A;
off = offsetof(struct sk_buff, dev);
/* Load *dev pointer */
emit_load_ptr(r_s0, r_skb, off, ctx);
/* error (0) in the delay slot */
b_off = b_imm(prog->len, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_bcond(MIPS_COND_EQ, r_s0, r_zero, b_off, ctx);
emit_reg_move(r_ret, r_zero, ctx);
if (code == (BPF_ANC | SKF_AD_IFINDEX)) {
BUILD_BUG_ON(sizeof_field(struct net_device, ifindex) != 4);
off = offsetof(struct net_device, ifindex);
emit_load(r_A, r_s0, off, ctx);
} else { /* (code == (BPF_ANC | SKF_AD_HATYPE) */
BUILD_BUG_ON(sizeof_field(struct net_device, type) != 2);
off = offsetof(struct net_device, type);
emit_half_load_unsigned(r_A, r_s0, off, ctx);
}
break;
case BPF_ANC | SKF_AD_MARK:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(sizeof_field(struct sk_buff, mark) != 4);
off = offsetof(struct sk_buff, mark);
emit_load(r_A, r_skb, off, ctx);
break;
case BPF_ANC | SKF_AD_RXHASH:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(sizeof_field(struct sk_buff, hash) != 4);
off = offsetof(struct sk_buff, hash);
emit_load(r_A, r_skb, off, ctx);
break;
case BPF_ANC | SKF_AD_VLAN_TAG:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(sizeof_field(struct sk_buff,
vlan_tci) != 2);
off = offsetof(struct sk_buff, vlan_tci);
emit_half_load_unsigned(r_A, r_skb, off, ctx);
break;
case BPF_ANC | SKF_AD_VLAN_TAG_PRESENT:
ctx->flags |= SEEN_SKB | SEEN_A;
emit_load_byte(r_A, r_skb, PKT_VLAN_PRESENT_OFFSET(), ctx);
if (PKT_VLAN_PRESENT_BIT)
emit_srl(r_A, r_A, PKT_VLAN_PRESENT_BIT, ctx);
if (PKT_VLAN_PRESENT_BIT < 7)
emit_andi(r_A, r_A, 1, ctx);
break;
case BPF_ANC | SKF_AD_PKTTYPE:
ctx->flags |= SEEN_SKB;
emit_load_byte(r_tmp, r_skb, PKT_TYPE_OFFSET(), ctx);
/* Keep only the last 3 bits */
emit_andi(r_A, r_tmp, PKT_TYPE_MAX, ctx);
#ifdef __BIG_ENDIAN_BITFIELD
/* Get the actual packet type to the lower 3 bits */
emit_srl(r_A, r_A, 5, ctx);
#endif
break;
case BPF_ANC | SKF_AD_QUEUE:
ctx->flags |= SEEN_SKB | SEEN_A;
BUILD_BUG_ON(sizeof_field(struct sk_buff,
queue_mapping) != 2);
BUILD_BUG_ON(offsetof(struct sk_buff,
queue_mapping) > 0xff);
off = offsetof(struct sk_buff, queue_mapping);
emit_half_load_unsigned(r_A, r_skb, off, ctx);
break;
default:
pr_debug("%s: Unhandled opcode: 0x%02x\n", __FILE__,
inst->code);
return -1;
}
}
/* compute offsets only during the first pass */
if (ctx->target == NULL)
ctx->offsets[i] = ctx->idx * 4;
return 0;
}
void bpf_jit_compile(struct bpf_prog *fp)
{
struct jit_ctx ctx;
unsigned int alloc_size, tmp_idx;
if (!bpf_jit_enable)
return;
memset(&ctx, 0, sizeof(ctx));
ctx.offsets = kcalloc(fp->len + 1, sizeof(*ctx.offsets), GFP_KERNEL);
if (ctx.offsets == NULL)
return;
ctx.skf = fp;
if (build_body(&ctx))
goto out;
tmp_idx = ctx.idx;
build_prologue(&ctx);
ctx.prologue_bytes = (ctx.idx - tmp_idx) * 4;
/* just to complete the ctx.idx count */
build_epilogue(&ctx);
alloc_size = 4 * ctx.idx;
ctx.target = module_alloc(alloc_size);
if (ctx.target == NULL)
goto out;
/* Clean it */
memset(ctx.target, 0, alloc_size);
ctx.idx = 0;
/* Generate the actual JIT code */
build_prologue(&ctx);
if (build_body(&ctx)) {
module_memfree(ctx.target);
goto out;
}
build_epilogue(&ctx);
/* Update the icache */
flush_icache_range((ptr)ctx.target, (ptr)(ctx.target + ctx.idx));
if (bpf_jit_enable > 1)
/* Dump JIT code */
bpf_jit_dump(fp->len, alloc_size, 2, ctx.target);
fp->bpf_func = (void *)ctx.target;
fp->jited = 1;
out:
kfree(ctx.offsets);
}
void bpf_jit_free(struct bpf_prog *fp)
{
if (fp->jited)
module_memfree(fp->bpf_func);
bpf_prog_unlock_free(fp);
}
arch/mips/net/bpf_jit.h deleted 100644 → 0
View file @ 9d057872
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Just-In-Time compiler for BPF filters on MIPS
*
* Copyright (c) 2014 Imagination Technologies Ltd.
* Author: Markos Chandras <markos.chandras@imgtec.com>
*/
#ifndef BPF_JIT_MIPS_OP_H
#define BPF_JIT_MIPS_OP_H
/* Registers used by JIT */
#define MIPS_R_ZERO 0
#define MIPS_R_V0 2
#define MIPS_R_A0 4
#define MIPS_R_A1 5
#define MIPS_R_T4 12
#define MIPS_R_T5 13
#define MIPS_R_T6 14
#define MIPS_R_T7 15
#define MIPS_R_S0 16
#define MIPS_R_S1 17
#define MIPS_R_S2 18
#define MIPS_R_S3 19
#define MIPS_R_S4 20
#define MIPS_R_S5 21
#define MIPS_R_S6 22
#define MIPS_R_S7 23
#define MIPS_R_SP 29
#define MIPS_R_RA 31
/* Conditional codes */
#define MIPS_COND_EQ 0x1
#define MIPS_COND_GE (0x1 << 1)
#define MIPS_COND_GT (0x1 << 2)
#define MIPS_COND_NE (0x1 << 3)
#define MIPS_COND_ALL (0x1 << 4)
/* Conditionals on X register or K immediate */
#define MIPS_COND_X (0x1 << 5)
#define MIPS_COND_K (0x1 << 6)
#define r_ret MIPS_R_V0
/*
* Use 2 scratch registers to avoid pipeline interlocks.
* There is no overhead during epilogue and prologue since
* any of the $s0-$s6 registers will only be preserved if
* they are going to actually be used.
*/
#define r_skb_hl MIPS_R_S0 /* skb header length */
#define r_skb_data MIPS_R_S1 /* skb actual data */
#define r_off MIPS_R_S2
#define r_A MIPS_R_S3
#define r_X MIPS_R_S4
#define r_skb MIPS_R_S5
#define r_M MIPS_R_S6
#define r_skb_len MIPS_R_S7
#define r_s0 MIPS_R_T4 /* scratch reg 1 */
#define r_s1 MIPS_R_T5 /* scratch reg 2 */
#define r_tmp_imm MIPS_R_T6 /* No need to preserve this */
#define r_tmp MIPS_R_T7 /* No need to preserve this */
#define r_zero MIPS_R_ZERO
#define r_sp MIPS_R_SP
#define r_ra MIPS_R_RA
#ifndef __ASSEMBLY__
/* Declare ASM helpers */
#define DECLARE_LOAD_FUNC(func) \
extern u8 func(unsigned long *skb, int offset); \
extern u8 func##_negative(unsigned long *skb, int offset); \
extern u8 func##_positive(unsigned long *skb, int offset)
DECLARE_LOAD_FUNC(sk_load_word);
DECLARE_LOAD_FUNC(sk_load_half);
DECLARE_LOAD_FUNC(sk_load_byte);
#endif
#endif /* BPF_JIT_MIPS_OP_H */
arch/mips/net/bpf_jit_asm.S deleted 100644 → 0
View file @ 9d057872
/*
* bpf_jib_asm.S: Packet/header access helper functions for MIPS/MIPS64 BPF
* compiler.
*
* Copyright (C) 2015 Imagination Technologies Ltd.
* Author: Markos Chandras <markos.chandras@imgtec.com>
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; version 2 of the License.
*/
#include <asm/asm.h>
#include <asm/isa-rev.h>
#include <asm/regdef.h>
#include "bpf_jit.h"
/* ABI
*
* r_skb_hl skb header length
* r_skb_data skb data
* r_off(a1) offset register
* r_A BPF register A
* r_X PF register X
* r_skb(a0) *skb
* r_M *scratch memory
* r_skb_le skb length
* r_s0 Scratch register 0
* r_s1 Scratch register 1
*
* On entry:
* a0: *skb
* a1: offset (imm or imm + X)
*
* All non-BPF-ABI registers are free for use. On return, we only
* care about r_ret. The BPF-ABI registers are assumed to remain
* unmodified during the entire filter operation.
*/
#define skb a0
#define offset a1
#define SKF_LL_OFF (-0x200000) /* Can't include linux/filter.h in assembly */
/* We know better :) so prevent assembler reordering etc */
.set noreorder
#define is_offset_negative(TYPE) \
/* If offset is negative we have more work to do */ \
slti t0, offset, 0; \
bgtz t0, bpf_slow_path_##TYPE##_neg; \
/* Be careful what follows in DS. */
#define is_offset_in_header(SIZE, TYPE) \
/* Reading from header? */ \
addiu $r_s0, $r_skb_hl, -SIZE; \
slt t0, $r_s0, offset; \
bgtz t0, bpf_slow_path_##TYPE; \
LEAF(sk_load_word)
is_offset_negative(word)
FEXPORT(sk_load_word_positive)
is_offset_in_header(4, word)
/* Offset within header boundaries */
PTR_ADDU t1, $r_skb_data, offset
.set reorder
lw $r_A, 0(t1)
.set noreorder
#ifdef CONFIG_CPU_LITTLE_ENDIAN
# if MIPS_ISA_REV >= 2
wsbh t0, $r_A
rotr $r_A, t0, 16
# else
sll t0, $r_A, 24
srl t1, $r_A, 24
srl t2, $r_A, 8
or t0, t0, t1
andi t2, t2, 0xff00
andi t1, $r_A, 0xff00
or t0, t0, t2
sll t1, t1, 8
or $r_A, t0, t1
# endif
#endif
jr $r_ra
move $r_ret, zero
END(sk_load_word)
LEAF(sk_load_half)
is_offset_negative(half)
FEXPORT(sk_load_half_positive)
is_offset_in_header(2, half)
/* Offset within header boundaries */
PTR_ADDU t1, $r_skb_data, offset
lhu $r_A, 0(t1)
#ifdef CONFIG_CPU_LITTLE_ENDIAN
# if MIPS_ISA_REV >= 2
wsbh $r_A, $r_A
# else
sll t0, $r_A, 8
srl t1, $r_A, 8
andi t0, t0, 0xff00
or $r_A, t0, t1
# endif
#endif
jr $r_ra
move $r_ret, zero
END(sk_load_half)
LEAF(sk_load_byte)
is_offset_negative(byte)
FEXPORT(sk_load_byte_positive)
is_offset_in_header(1, byte)
/* Offset within header boundaries */
PTR_ADDU t1, $r_skb_data, offset
lbu $r_A, 0(t1)
jr $r_ra
move $r_ret, zero
END(sk_load_byte)
/*
* call skb_copy_bits:
* (prototype in linux/skbuff.h)
*
* int skb_copy_bits(sk_buff *skb, int offset, void *to, int len)
*
* o32 mandates we leave 4 spaces for argument registers in case
* the callee needs to use them. Even though we don't care about
* the argument registers ourselves, we need to allocate that space
* to remain ABI compliant since the callee may want to use that space.
* We also allocate 2 more spaces for $r_ra and our return register (*to).
*
* n64 is a bit different. The *caller* will allocate the space to preserve
* the arguments. So in 64-bit kernels, we allocate the 4-arg space for no
* good reason but it does not matter that much really.
*
* (void *to) is returned in r_s0
*
*/
#ifdef CONFIG_CPU_LITTLE_ENDIAN
#define DS_OFFSET(SIZE) (4 * SZREG)
#else
#define DS_OFFSET(SIZE) ((4 * SZREG) + (4 - SIZE))
#endif
#define bpf_slow_path_common(SIZE) \
/* Quick check. Are we within reasonable boundaries? */ \
LONG_ADDIU $r_s1, $r_skb_len, -SIZE; \
sltu $r_s0, offset, $r_s1; \
beqz $r_s0, fault; \
/* Load 4th argument in DS */ \
LONG_ADDIU a3, zero, SIZE; \
PTR_ADDIU $r_sp, $r_sp, -(6 * SZREG); \
PTR_LA t0, skb_copy_bits; \
PTR_S $r_ra, (5 * SZREG)($r_sp); \
/* Assign low slot to a2 */ \
PTR_ADDIU a2, $r_sp, DS_OFFSET(SIZE); \
jalr t0; \
/* Reset our destination slot (DS but it's ok) */ \
INT_S zero, (4 * SZREG)($r_sp); \
/* \
* skb_copy_bits returns 0 on success and -EFAULT \
* on error. Our data live in a2. Do not bother with \
* our data if an error has been returned. \
*/ \
/* Restore our frame */ \
PTR_L $r_ra, (5 * SZREG)($r_sp); \
INT_L $r_s0, (4 * SZREG)($r_sp); \
bltz v0, fault; \
PTR_ADDIU $r_sp, $r_sp, 6 * SZREG; \
move $r_ret, zero; \
NESTED(bpf_slow_path_word, (6 * SZREG), $r_sp)
bpf_slow_path_common(4)
#ifdef CONFIG_CPU_LITTLE_ENDIAN
# if MIPS_ISA_REV >= 2
wsbh t0, $r_s0
jr $r_ra
rotr $r_A, t0, 16
# else
sll t0, $r_s0, 24
srl t1, $r_s0, 24
srl t2, $r_s0, 8
or t0, t0, t1
andi t2, t2, 0xff00
andi t1, $r_s0, 0xff00
or t0, t0, t2
sll t1, t1, 8
jr $r_ra
or $r_A, t0, t1
# endif
#else
jr $r_ra
move $r_A, $r_s0
#endif
END(bpf_slow_path_word)
NESTED(bpf_slow_path_half, (6 * SZREG), $r_sp)
bpf_slow_path_common(2)
#ifdef CONFIG_CPU_LITTLE_ENDIAN
# if MIPS_ISA_REV >= 2
jr $r_ra
wsbh $r_A, $r_s0
# else
sll t0, $r_s0, 8
andi t1, $r_s0, 0xff00
andi t0, t0, 0xff00
srl t1, t1, 8
jr $r_ra
or $r_A, t0, t1
# endif
#else
jr $r_ra
move $r_A, $r_s0
#endif
END(bpf_slow_path_half)
NESTED(bpf_slow_path_byte, (6 * SZREG), $r_sp)
bpf_slow_path_common(1)
jr $r_ra
move $r_A, $r_s0
END(bpf_slow_path_byte)
/*
* Negative entry points
*/
.macro bpf_is_end_of_data
li t0, SKF_LL_OFF
/* Reading link layer data? */
slt t1, offset, t0
bgtz t1, fault
/* Be careful what follows in DS. */
.endm
/*
* call skb_copy_bits:
* (prototype in linux/filter.h)
*
* void *bpf_internal_load_pointer_neg_helper(const struct sk_buff *skb,
* int k, unsigned int size)
*
* see above (bpf_slow_path_common) for ABI restrictions
*/
#define bpf_negative_common(SIZE) \
PTR_ADDIU $r_sp, $r_sp, -(6 * SZREG); \
PTR_LA t0, bpf_internal_load_pointer_neg_helper; \
PTR_S $r_ra, (5 * SZREG)($r_sp); \
jalr t0; \
li a2, SIZE; \
PTR_L $r_ra, (5 * SZREG)($r_sp); \
/* Check return pointer */ \
beqz v0, fault; \
PTR_ADDIU $r_sp, $r_sp, 6 * SZREG; \
/* Preserve our pointer */ \
move $r_s0, v0; \
/* Set return value */ \
move $r_ret, zero; \
bpf_slow_path_word_neg:
bpf_is_end_of_data
NESTED(sk_load_word_negative, (6 * SZREG), $r_sp)
bpf_negative_common(4)
jr $r_ra
lw $r_A, 0($r_s0)
END(sk_load_word_negative)
bpf_slow_path_half_neg:
bpf_is_end_of_data
NESTED(sk_load_half_negative, (6 * SZREG), $r_sp)
bpf_negative_common(2)
jr $r_ra
lhu $r_A, 0($r_s0)
END(sk_load_half_negative)
bpf_slow_path_byte_neg:
bpf_is_end_of_data
NESTED(sk_load_byte_negative, (6 * SZREG), $r_sp)
bpf_negative_common(1)
jr $r_ra
lbu $r_A, 0($r_s0)
END(sk_load_byte_negative)
fault:
jr $r_ra
addiu $r_ret, zero, 1
arch/mips/net/bpf_jit_comp.c 0 → 100644
View file @ f438ee21
// SPDX-License-Identifier: GPL-2.0-only
/*
* Just-In-Time compiler for eBPF bytecode on MIPS.
* Implementation of JIT functions common to 32-bit and 64-bit CPUs.
*
* Copyright (c) 2021 Anyfi Networks AB.
* Author: Johan Almbladh <johan.almbladh@gmail.com>
*
* Based on code and ideas from
* Copyright (c) 2017 Cavium, Inc.
* Copyright (c) 2017 Shubham Bansal <illusionist.neo@gmail.com>
* Copyright (c) 2011 Mircea Gherzan <mgherzan@gmail.com>
*/
/*
* Code overview
* =============
*
* - bpf_jit_comp.h
* Common definitions and utilities.
*
* - bpf_jit_comp.c
* Implementation of JIT top-level logic and exported JIT API functions.
* Implementation of internal operations shared by 32-bit and 64-bit code.
* JMP and ALU JIT control code, register control code, shared ALU and
* JMP/JMP32 JIT operations.
*
* - bpf_jit_comp32.c
* Implementation of functions to JIT prologue, epilogue and a single eBPF
* instruction for 32-bit MIPS CPUs. The functions use shared operations
* where possible, and implement the rest for 32-bit MIPS such as ALU64
* operations.
*
* - bpf_jit_comp64.c
* Ditto, for 64-bit MIPS CPUs.
*
* Zero and sign extension
* ========================
* 32-bit MIPS instructions on 64-bit MIPS registers use sign extension,
* but the eBPF instruction set mandates zero extension. We let the verifier
* insert explicit zero-extensions after 32-bit ALU operations, both for
* 32-bit and 64-bit MIPS JITs. Conditional JMP32 operations on 64-bit MIPs
* are JITed with sign extensions inserted when so expected.
*
* ALU operations
* ==============
* ALU operations on 32/64-bit MIPS and ALU64 operations on 64-bit MIPS are
* JITed in the following steps. ALU64 operations on 32-bit MIPS are more
* complicated and therefore only processed by special implementations in
* step (3).
*
* 1) valid_alu_i:
* Determine if an immediate operation can be emitted as such, or if
* we must fall back to the register version.
*
* 2) rewrite_alu_i:
* Convert BPF operation and immediate value to a canonical form for
* JITing. In some degenerate cases this form may be a no-op.
*
* 3) emit_alu_{i,i64,r,64}:
* Emit instructions for an ALU or ALU64 immediate or register operation.
*
* JMP operations
* ==============
* JMP and JMP32 operations require an JIT instruction offset table for
* translating the jump offset. This table is computed by dry-running the
* JIT without actually emitting anything. However, the computed PC-relative
* offset may overflow the 18-bit offset field width of the native MIPS
* branch instruction. In such cases, the long jump is converted into the
* following sequence.
*
* <branch> !<cond> +2 Inverted PC-relative branch
* nop Delay slot
* j <offset> Unconditional absolute long jump
* nop Delay slot
*
* Since this converted sequence alters the offset table, all offsets must
* be re-calculated. This may in turn trigger new branch conversions, so
* the process is repeated until no further changes are made. Normally it
* completes in 1-2 iterations. If JIT_MAX_ITERATIONS should reached, we
* fall back to converting every remaining jump operation. The branch
* conversion is independent of how the JMP or JMP32 condition is JITed.
*
* JMP32 and JMP operations are JITed as follows.
*
* 1) setup_jmp_{i,r}:
* Convert jump conditional and offset into a form that can be JITed.
* This form may be a no-op, a canonical form, or an inverted PC-relative
* jump if branch conversion is necessary.
*
* 2) valid_jmp_i:
* Determine if an immediate operations can be emitted as such, or if
* we must fall back to the register version. Applies to JMP32 for 32-bit
* MIPS, and both JMP and JMP32 for 64-bit MIPS.
*
* 3) emit_jmp_{i,i64,r,r64}:
* Emit instructions for an JMP or JMP32 immediate or register operation.
*
* 4) finish_jmp_{i,r}:
* Emit any instructions needed to finish the jump. This includes a nop
* for the delay slot if a branch was emitted, and a long absolute jump
* if the branch was converted.
*/
#include <linux/limits.h>
#include <linux/bitops.h>
#include <linux/errno.h>
#include <linux/filter.h>
#include <linux/bpf.h>
#include <linux/slab.h>
#include <asm/bitops.h>
#include <asm/cacheflush.h>
#include <asm/cpu-features.h>
#include <asm/isa-rev.h>
#include <asm/uasm.h>
#include "bpf_jit_comp.h"
/* Convenience macros for descriptor access */
#define CONVERTED(desc) ((desc) & JIT_DESC_CONVERT)
#define INDEX(desc) ((desc) & ~JIT_DESC_CONVERT)
/*
* Push registers on the stack, starting at a given depth from the stack
* pointer and increasing. The next depth to be written is returned.
*/
int push_regs(struct jit_context *ctx, u32 mask, u32 excl, int depth)
{
int reg;
for (reg = 0; reg < BITS_PER_BYTE * sizeof(mask); reg++)
if (mask & BIT(reg)) {
if ((excl & BIT(reg)) == 0) {
if (sizeof(long) == 4)
emit(ctx, sw, reg, depth, MIPS_R_SP);
else /* sizeof(long) == 8 */
emit(ctx, sd, reg, depth, MIPS_R_SP);
}
depth += sizeof(long);
}
ctx->stack_used = max((int)ctx->stack_used, depth);
return depth;
}
/*
* Pop registers from the stack, starting at a given depth from the stack
* pointer and increasing. The next depth to be read is returned.
*/
int pop_regs(struct jit_context *ctx, u32 mask, u32 excl, int depth)
{
int reg;
for (reg = 0; reg < BITS_PER_BYTE * sizeof(mask); reg++)
if (mask & BIT(reg)) {
if ((excl & BIT(reg)) == 0) {
if (sizeof(long) == 4)
emit(ctx, lw, reg, depth, MIPS_R_SP);
else /* sizeof(long) == 8 */
emit(ctx, ld, reg, depth, MIPS_R_SP);
}
depth += sizeof(long);
}
return depth;
}
/* Compute the 28-bit jump target address from a BPF program location */
int get_target(struct jit_context *ctx, u32 loc)
{
u32 index = INDEX(ctx->descriptors[loc]);
unsigned long pc = (unsigned long)&ctx->target[ctx->jit_index];
unsigned long addr = (unsigned long)&ctx->target[index];
if (!ctx->target)
return 0;
if ((addr ^ pc) & ~MIPS_JMP_MASK)
return -1;
return addr & MIPS_JMP_MASK;
}
/* Compute the PC-relative offset to relative BPF program offset */
int get_offset(const struct jit_context *ctx, int off)
{
return (INDEX(ctx->descriptors[ctx->bpf_index + off]) -
ctx->jit_index - 1) * sizeof(u32);
}
/* dst = imm (register width) */
void emit_mov_i(struct jit_context *ctx, u8 dst, s32 imm)
{
if (imm >= -0x8000 && imm <= 0x7fff) {
emit(ctx, addiu, dst, MIPS_R_ZERO, imm);
} else {
emit(ctx, lui, dst, (s16)((u32)imm >> 16));
emit(ctx, ori, dst, dst, (u16)(imm & 0xffff));
}
clobber_reg(ctx, dst);
}
/* dst = src (register width) */
void emit_mov_r(struct jit_context *ctx, u8 dst, u8 src)
{
emit(ctx, ori, dst, src, 0);
clobber_reg(ctx, dst);
}
/* Validate ALU immediate range */
bool valid_alu_i(u8 op, s32 imm)
{
switch (BPF_OP(op)) {
case BPF_NEG:
case BPF_LSH:
case BPF_RSH:
case BPF_ARSH:
/* All legal eBPF values are valid */
return true;
case BPF_ADD:
/* imm must be 16 bits */
return imm >= -0x8000 && imm <= 0x7fff;
case BPF_SUB:
/* -imm must be 16 bits */
return imm >= -0x7fff && imm <= 0x8000;
case BPF_AND:
case BPF_OR:
case BPF_XOR:
/* imm must be 16 bits unsigned */
return imm >= 0 && imm <= 0xffff;
case BPF_MUL:
/* imm must be zero or a positive power of two */
return imm == 0 || (imm > 0 && is_power_of_2(imm));
case BPF_DIV:
case BPF_MOD:
/* imm must be an 17-bit power of two */
return (u32)imm <= 0x10000 && is_power_of_2((u32)imm);
}
return false;
}
/* Rewrite ALU immediate operation */
bool rewrite_alu_i(u8 op, s32 imm, u8 *alu, s32 *val)
{
bool act = true;
switch (BPF_OP(op)) {
case BPF_LSH:
case BPF_RSH:
case BPF_ARSH:
case BPF_ADD:
case BPF_SUB:
case BPF_OR:
case BPF_XOR:
/* imm == 0 is a no-op */
act = imm != 0;
break;
case BPF_MUL:
if (imm == 1) {
/* dst * 1 is a no-op */
act = false;
} else if (imm == 0) {
/* dst * 0 is dst & 0 */
op = BPF_AND;
} else {
/* dst * (1 << n) is dst << n */
op = BPF_LSH;
imm = ilog2(abs(imm));
}
break;
case BPF_DIV:
if (imm == 1) {
/* dst / 1 is a no-op */
act = false;
} else {
/* dst / (1 << n) is dst >> n */
op = BPF_RSH;
imm = ilog2(imm);
}
break;
case BPF_MOD:
/* dst % (1 << n) is dst & ((1 << n) - 1) */
op = BPF_AND;
imm--;
break;
}
*alu = op;
*val = imm;
return act;
}
/* ALU immediate operation (32-bit) */
void emit_alu_i(struct jit_context *ctx, u8 dst, s32 imm, u8 op)
{
switch (BPF_OP(op)) {
/* dst = -dst */
case BPF_NEG:
emit(ctx, subu, dst, MIPS_R_ZERO, dst);
break;
/* dst = dst & imm */
case BPF_AND:
emit(ctx, andi, dst, dst, (u16)imm);
break;
/* dst = dst | imm */
case BPF_OR:
emit(ctx, ori, dst, dst, (u16)imm);
break;
/* dst = dst ^ imm */
case BPF_XOR:
emit(ctx, xori, dst, dst, (u16)imm);
break;
/* dst = dst << imm */
case BPF_LSH:
emit(ctx, sll, dst, dst, imm);
break;
/* dst = dst >> imm */
case BPF_RSH:
emit(ctx, srl, dst, dst, imm);
break;
/* dst = dst >> imm (arithmetic) */
case BPF_ARSH:
emit(ctx, sra, dst, dst, imm);
break;
/* dst = dst + imm */
case BPF_ADD:
emit(ctx, addiu, dst, dst, imm);
break;
/* dst = dst - imm */
case BPF_SUB:
emit(ctx, addiu, dst, dst, -imm);
break;
}
clobber_reg(ctx, dst);
}
/* ALU register operation (32-bit) */
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op)
{
switch (BPF_OP(op)) {
/* dst = dst & src */
case BPF_AND:
emit(ctx, and, dst, dst, src);
break;
/* dst = dst | src */
case BPF_OR:
emit(ctx, or, dst, dst, src);
break;
/* dst = dst ^ src */
case BPF_XOR:
emit(ctx, xor, dst, dst, src);
break;
/* dst = dst << src */
case BPF_LSH:
emit(ctx, sllv, dst, dst, src);
break;
/* dst = dst >> src */
case BPF_RSH:
emit(ctx, srlv, dst, dst, src);
break;
/* dst = dst >> src (arithmetic) */
case BPF_ARSH:
emit(ctx, srav, dst, dst, src);
break;
/* dst = dst + src */
case BPF_ADD:
emit(ctx, addu, dst, dst, src);
break;
/* dst = dst - src */
case BPF_SUB:
emit(ctx, subu, dst, dst, src);
break;
/* dst = dst * src */
case BPF_MUL:
if (cpu_has_mips32r1 || cpu_has_mips32r6) {
emit(ctx, mul, dst, dst, src);
} else {
emit(ctx, multu, dst, src);
emit(ctx, mflo, dst);
}
break;
/* dst = dst / src */
case BPF_DIV:
if (cpu_has_mips32r6) {
emit(ctx, divu_r6, dst, dst, src);
} else {
emit(ctx, divu, dst, src);
emit(ctx, mflo, dst);
}
break;
/* dst = dst % src */
case BPF_MOD:
if (cpu_has_mips32r6) {
emit(ctx, modu, dst, dst, src);
} else {
emit(ctx, divu, dst, src);
emit(ctx, mfhi, dst);
}
break;
}
clobber_reg(ctx, dst);
}
/* Atomic read-modify-write (32-bit) */
void emit_atomic_r(struct jit_context *ctx, u8 dst, u8 src, s16 off, u8 code)
{
LLSC_sync(ctx);
emit(ctx, ll, MIPS_R_T9, off, dst);
switch (code) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
emit(ctx, addu, MIPS_R_T8, MIPS_R_T9, src);
break;
case BPF_AND:
case BPF_AND | BPF_FETCH:
emit(ctx, and, MIPS_R_T8, MIPS_R_T9, src);
break;
case BPF_OR:
case BPF_OR | BPF_FETCH:
emit(ctx, or, MIPS_R_T8, MIPS_R_T9, src);
break;
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
emit(ctx, xor, MIPS_R_T8, MIPS_R_T9, src);
break;
case BPF_XCHG:
emit(ctx, move, MIPS_R_T8, src);
break;
}
emit(ctx, sc, MIPS_R_T8, off, dst);
emit(ctx, LLSC_beqz, MIPS_R_T8, -16 - LLSC_offset);
emit(ctx, nop); /* Delay slot */
if (code & BPF_FETCH) {
emit(ctx, move, src, MIPS_R_T9);
clobber_reg(ctx, src);
}
}
/* Atomic compare-and-exchange (32-bit) */
void emit_cmpxchg_r(struct jit_context *ctx, u8 dst, u8 src, u8 res, s16 off)
{
LLSC_sync(ctx);
emit(ctx, ll, MIPS_R_T9, off, dst);
emit(ctx, bne, MIPS_R_T9, res, 12);
emit(ctx, move, MIPS_R_T8, src); /* Delay slot */
emit(ctx, sc, MIPS_R_T8, off, dst);
emit(ctx, LLSC_beqz, MIPS_R_T8, -20 - LLSC_offset);
emit(ctx, move, res, MIPS_R_T9); /* Delay slot */
clobber_reg(ctx, res);
}
/* Swap bytes and truncate a register word or half word */
void emit_bswap_r(struct jit_context *ctx, u8 dst, u32 width)
{
u8 tmp = MIPS_R_T8;
u8 msk = MIPS_R_T9;
switch (width) {
/* Swap bytes in a word */
case 32:
if (cpu_has_mips32r2 || cpu_has_mips32r6) {
emit(ctx, wsbh, dst, dst);
emit(ctx, rotr, dst, dst, 16);
} else {
emit(ctx, sll, tmp, dst, 16); /* tmp = dst << 16 */
emit(ctx, srl, dst, dst, 16); /* dst = dst >> 16 */
emit(ctx, or, dst, dst, tmp); /* dst = dst | tmp */
emit(ctx, lui, msk, 0xff); /* msk = 0x00ff0000 */
emit(ctx, ori, msk, msk, 0xff); /* msk = msk | 0xff */
emit(ctx, and, tmp, dst, msk); /* tmp = dst & msk */
emit(ctx, sll, tmp, tmp, 8); /* tmp = tmp << 8 */
emit(ctx, srl, dst, dst, 8); /* dst = dst >> 8 */
emit(ctx, and, dst, dst, msk); /* dst = dst & msk */
emit(ctx, or, dst, dst, tmp); /* reg = dst | tmp */
}
break;
/* Swap bytes in a half word */
case 16:
if (cpu_has_mips32r2 || cpu_has_mips32r6) {
emit(ctx, wsbh, dst, dst);
emit(ctx, andi, dst, dst, 0xffff);
} else {
emit(ctx, andi, tmp, dst, 0xff00); /* t = d & 0xff00 */
emit(ctx, srl, tmp, tmp, 8); /* t = t >> 8 */
emit(ctx, andi, dst, dst, 0x00ff); /* d = d & 0x00ff */
emit(ctx, sll, dst, dst, 8); /* d = d << 8 */
emit(ctx, or, dst, dst, tmp); /* d = d | t */
}
break;
}
clobber_reg(ctx, dst);
}
/* Validate jump immediate range */
bool valid_jmp_i(u8 op, s32 imm)
{
switch (op) {
case JIT_JNOP:
/* Immediate value not used */
return true;
case BPF_JEQ:
case BPF_JNE:
/* No immediate operation */
return false;
case BPF_JSET:
case JIT_JNSET:
/* imm must be 16 bits unsigned */
return imm >= 0 && imm <= 0xffff;
case BPF_JGE:
case BPF_JLT:
case BPF_JSGE:
case BPF_JSLT:
/* imm must be 16 bits */
return imm >= -0x8000 && imm <= 0x7fff;
case BPF_JGT:
case BPF_JLE:
case BPF_JSGT:
case BPF_JSLE:
/* imm + 1 must be 16 bits */
return imm >= -0x8001 && imm <= 0x7ffe;
}
return false;
}
/* Invert a conditional jump operation */
static u8 invert_jmp(u8 op)
{
switch (op) {
case BPF_JA: return JIT_JNOP;
case BPF_JEQ: return BPF_JNE;
case BPF_JNE: return BPF_JEQ;
case BPF_JSET: return JIT_JNSET;
case BPF_JGT: return BPF_JLE;
case BPF_JGE: return BPF_JLT;
case BPF_JLT: return BPF_JGE;
case BPF_JLE: return BPF_JGT;
case BPF_JSGT: return BPF_JSLE;
case BPF_JSGE: return BPF_JSLT;
case BPF_JSLT: return BPF_JSGE;
case BPF_JSLE: return BPF_JSGT;
}
return 0;
}
/* Prepare a PC-relative jump operation */
static void setup_jmp(struct jit_context *ctx, u8 bpf_op,
s16 bpf_off, u8 *jit_op, s32 *jit_off)
{
u32 *descp = &ctx->descriptors[ctx->bpf_index];
int op = bpf_op;
int offset = 0;
/* Do not compute offsets on the first pass */
if (INDEX(*descp) == 0)
goto done;
/* Skip jumps never taken */
if (bpf_op == JIT_JNOP)
goto done;
/* Convert jumps always taken */
if (bpf_op == BPF_JA)
*descp |= JIT_DESC_CONVERT;
/*
* Current ctx->jit_index points to the start of the branch preamble.
* Since the preamble differs among different branch conditionals,
* the current index cannot be used to compute the branch offset.
* Instead, we use the offset table value for the next instruction,
* which gives the index immediately after the branch delay slot.
*/
if (!CONVERTED(*descp)) {
int target = ctx->bpf_index + bpf_off + 1;
int origin = ctx->bpf_index + 1;
offset = (INDEX(ctx->descriptors[target]) -
INDEX(ctx->descriptors[origin]) + 1) * sizeof(u32);
}
/*
* The PC-relative branch offset field on MIPS is 18 bits signed,
* so if the computed offset is larger than this we generate a an
* absolute jump that we skip with an inverted conditional branch.
*/
if (CONVERTED(*descp) || offset < -0x20000 || offset > 0x1ffff) {
offset = 3 * sizeof(u32);
op = invert_jmp(bpf_op);
ctx->changes += !CONVERTED(*descp);
*descp |= JIT_DESC_CONVERT;
}
done:
*jit_off = offset;
*jit_op = op;
}
/* Prepare a PC-relative jump operation with immediate conditional */
void setup_jmp_i(struct jit_context *ctx, s32 imm, u8 width,
u8 bpf_op, s16 bpf_off, u8 *jit_op, s32 *jit_off)
{
bool always = false;
bool never = false;
switch (bpf_op) {
case BPF_JEQ:
case BPF_JNE:
break;
case BPF_JSET:
case BPF_JLT:
never = imm == 0;
break;
case BPF_JGE:
always = imm == 0;
break;
case BPF_JGT:
never = (u32)imm == U32_MAX;
break;
case BPF_JLE:
always = (u32)imm == U32_MAX;
break;
case BPF_JSGT:
never = imm == S32_MAX && width == 32;
break;
case BPF_JSGE:
always = imm == S32_MIN && width == 32;
break;
case BPF_JSLT:
never = imm == S32_MIN && width == 32;
break;
case BPF_JSLE:
always = imm == S32_MAX && width == 32;
break;
}
if (never)
bpf_op = JIT_JNOP;
if (always)
bpf_op = BPF_JA;
setup_jmp(ctx, bpf_op, bpf_off, jit_op, jit_off);
}
/* Prepare a PC-relative jump operation with register conditional */
void setup_jmp_r(struct jit_context *ctx, bool same_reg,
u8 bpf_op, s16 bpf_off, u8 *jit_op, s32 *jit_off)
{
switch (bpf_op) {
case BPF_JSET:
break;
case BPF_JEQ:
case BPF_JGE:
case BPF_JLE:
case BPF_JSGE:
case BPF_JSLE:
if (same_reg)
bpf_op = BPF_JA;
break;
case BPF_JNE:
case BPF_JLT:
case BPF_JGT:
case BPF_JSGT:
case BPF_JSLT:
if (same_reg)
bpf_op = JIT_JNOP;
break;
}
setup_jmp(ctx, bpf_op, bpf_off, jit_op, jit_off);
}
/* Finish a PC-relative jump operation */
int finish_jmp(struct jit_context *ctx, u8 jit_op, s16 bpf_off)
{
/* Emit conditional branch delay slot */
if (jit_op != JIT_JNOP)
emit(ctx, nop);
/*
* Emit an absolute long jump with delay slot,
* if the PC-relative branch was converted.
*/
if (CONVERTED(ctx->descriptors[ctx->bpf_index])) {
int target = get_target(ctx, ctx->bpf_index + bpf_off + 1);
if (target < 0)
return -1;
emit(ctx, j, target);
emit(ctx, nop);
}
return 0;
}
/* Jump immediate (32-bit) */
void emit_jmp_i(struct jit_context *ctx, u8 dst, s32 imm, s32 off, u8 op)
{
switch (op) {
/* No-op, used internally for branch optimization */
case JIT_JNOP:
break;
/* PC += off if dst & imm */
case BPF_JSET:
emit(ctx, andi, MIPS_R_T9, dst, (u16)imm);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if (dst & imm) == 0 (not in BPF, used for long jumps) */
case JIT_JNSET:
emit(ctx, andi, MIPS_R_T9, dst, (u16)imm);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst > imm */
case BPF_JGT:
emit(ctx, sltiu, MIPS_R_T9, dst, imm + 1);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst >= imm */
case BPF_JGE:
emit(ctx, sltiu, MIPS_R_T9, dst, imm);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst < imm */
case BPF_JLT:
emit(ctx, sltiu, MIPS_R_T9, dst, imm);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst <= imm */
case BPF_JLE:
emit(ctx, sltiu, MIPS_R_T9, dst, imm + 1);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst > imm (signed) */
case BPF_JSGT:
emit(ctx, slti, MIPS_R_T9, dst, imm + 1);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst >= imm (signed) */
case BPF_JSGE:
emit(ctx, slti, MIPS_R_T9, dst, imm);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst < imm (signed) */
case BPF_JSLT:
emit(ctx, slti, MIPS_R_T9, dst, imm);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst <= imm (signed) */
case BPF_JSLE:
emit(ctx, slti, MIPS_R_T9, dst, imm + 1);
emit(ctx, bnez, MIPS_R_T9, off);
break;
}
}
/* Jump register (32-bit) */
void emit_jmp_r(struct jit_context *ctx, u8 dst, u8 src, s32 off, u8 op)
{
switch (op) {
/* No-op, used internally for branch optimization */
case JIT_JNOP:
break;
/* PC += off if dst == src */
case BPF_JEQ:
emit(ctx, beq, dst, src, off);
break;
/* PC += off if dst != src */
case BPF_JNE:
emit(ctx, bne, dst, src, off);
break;
/* PC += off if dst & src */
case BPF_JSET:
emit(ctx, and, MIPS_R_T9, dst, src);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if (dst & imm) == 0 (not in BPF, used for long jumps) */
case JIT_JNSET:
emit(ctx, and, MIPS_R_T9, dst, src);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst > src */
case BPF_JGT:
emit(ctx, sltu, MIPS_R_T9, src, dst);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst >= src */
case BPF_JGE:
emit(ctx, sltu, MIPS_R_T9, dst, src);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst < src */
case BPF_JLT:
emit(ctx, sltu, MIPS_R_T9, dst, src);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst <= src */
case BPF_JLE:
emit(ctx, sltu, MIPS_R_T9, src, dst);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst > src (signed) */
case BPF_JSGT:
emit(ctx, slt, MIPS_R_T9, src, dst);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst >= src (signed) */
case BPF_JSGE:
emit(ctx, slt, MIPS_R_T9, dst, src);
emit(ctx, beqz, MIPS_R_T9, off);
break;
/* PC += off if dst < src (signed) */
case BPF_JSLT:
emit(ctx, slt, MIPS_R_T9, dst, src);
emit(ctx, bnez, MIPS_R_T9, off);
break;
/* PC += off if dst <= src (signed) */
case BPF_JSLE:
emit(ctx, slt, MIPS_R_T9, src, dst);
emit(ctx, beqz, MIPS_R_T9, off);
break;
}
}
/* Jump always */
int emit_ja(struct jit_context *ctx, s16 off)
{
int target = get_target(ctx, ctx->bpf_index + off + 1);
if (target < 0)
return -1;
emit(ctx, j, target);
emit(ctx, nop);
return 0;
}
/* Jump to epilogue */
int emit_exit(struct jit_context *ctx)
{
int target = get_target(ctx, ctx->program->len);
if (target < 0)
return -1;
emit(ctx, j, target);
emit(ctx, nop);
return 0;
}
/* Build the program body from eBPF bytecode */
static int build_body(struct jit_context *ctx)
{
const struct bpf_prog *prog = ctx->program;
unsigned int i;
ctx->stack_used = 0;
for (i = 0; i < prog->len; i++) {
const struct bpf_insn *insn = &prog->insnsi[i];
u32 *descp = &ctx->descriptors[i];
int ret;
access_reg(ctx, insn->src_reg);
access_reg(ctx, insn->dst_reg);
ctx->bpf_index = i;
if (ctx->target == NULL) {
ctx->changes += INDEX(*descp) != ctx->jit_index;
*descp &= JIT_DESC_CONVERT;
*descp |= ctx->jit_index;
}
ret = build_insn(insn, ctx);
if (ret < 0)
return ret;
if (ret > 0) {
i++;
if (ctx->target == NULL)
descp[1] = ctx->jit_index;
}
}
/* Store the end offset, where the epilogue begins */
ctx->descriptors[prog->len] = ctx->jit_index;
return 0;
}
/* Set the branch conversion flag on all instructions */
static void set_convert_flag(struct jit_context *ctx, bool enable)
{
const struct bpf_prog *prog = ctx->program;
u32 flag = enable ? JIT_DESC_CONVERT : 0;
unsigned int i;
for (i = 0; i <= prog->len; i++)
ctx->descriptors[i] = INDEX(ctx->descriptors[i]) | flag;
}
static void jit_fill_hole(void *area, unsigned int size)
{
u32 *p;
/* We are guaranteed to have aligned memory. */
for (p = area; size >= sizeof(u32); size -= sizeof(u32))
uasm_i_break(&p, BRK_BUG); /* Increments p */
}
bool bpf_jit_needs_zext(void)
{
return true;
}
struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
{
struct bpf_prog *tmp, *orig_prog = prog;
struct bpf_binary_header *header = NULL;
struct jit_context ctx;
bool tmp_blinded = false;
unsigned int tmp_idx;
unsigned int image_size;
u8 *image_ptr;
int tries;
/*
* If BPF JIT was not enabled then we must fall back to
* the interpreter.
*/
if (!prog->jit_requested)
return orig_prog;
/*
* If constant blinding was enabled and we failed during blinding
* then we must fall back to the interpreter. Otherwise, we save
* the new JITed code.
*/
tmp = bpf_jit_blind_constants(prog);
if (IS_ERR(tmp))
return orig_prog;
if (tmp != prog) {
tmp_blinded = true;
prog = tmp;
}
memset(&ctx, 0, sizeof(ctx));
ctx.program = prog;
/*
* Not able to allocate memory for descriptors[], then
* we must fall back to the interpreter
*/
ctx.descriptors = kcalloc(prog->len + 1, sizeof(*ctx.descriptors),
GFP_KERNEL);
if (ctx.descriptors == NULL)
goto out_err;
/* First pass discovers used resources */
if (build_body(&ctx) < 0)
goto out_err;
/*
* Second pass computes instruction offsets.
* If any PC-relative branches are out of range, a sequence of
* a PC-relative branch + a jump is generated, and we have to
* try again from the beginning to generate the new offsets.
* This is done until no additional conversions are necessary.
* The last two iterations are done with all branches being
* converted, to guarantee offset table convergence within a
* fixed number of iterations.
*/
ctx.jit_index = 0;
build_prologue(&ctx);
tmp_idx = ctx.jit_index;
tries = JIT_MAX_ITERATIONS;
do {
ctx.jit_index = tmp_idx;
ctx.changes = 0;
if (tries == 2)
set_convert_flag(&ctx, true);
if (build_body(&ctx) < 0)
goto out_err;
} while (ctx.changes > 0 && --tries > 0);
if (WARN_ONCE(ctx.changes > 0, "JIT offsets failed to converge"))
goto out_err;
build_epilogue(&ctx, MIPS_R_RA);
/* Now we know the size of the structure to make */
image_size = sizeof(u32) * ctx.jit_index;
header = bpf_jit_binary_alloc(image_size, &image_ptr,
sizeof(u32), jit_fill_hole);
/*
* Not able to allocate memory for the structure then
* we must fall back to the interpretation
*/
if (header == NULL)
goto out_err;
/* Actual pass to generate final JIT code */
ctx.target = (u32 *)image_ptr;
ctx.jit_index = 0;
/*
* If building the JITed code fails somehow,
* we fall back to the interpretation.
*/
build_prologue(&ctx);
if (build_body(&ctx) < 0)
goto out_err;
build_epilogue(&ctx, MIPS_R_RA);
/* Populate line info meta data */
set_convert_flag(&ctx, false);
bpf_prog_fill_jited_linfo(prog, &ctx.descriptors[1]);
/* Set as read-only exec and flush instruction cache */
bpf_jit_binary_lock_ro(header);
flush_icache_range((unsigned long)header,
(unsigned long)&ctx.target[ctx.jit_index]);
if (bpf_jit_enable > 1)
bpf_jit_dump(prog->len, image_size, 2, ctx.target);
prog->bpf_func = (void *)ctx.target;
prog->jited = 1;
prog->jited_len = image_size;
out:
if (tmp_blinded)
bpf_jit_prog_release_other(prog, prog == orig_prog ?
tmp : orig_prog);
kfree(ctx.descriptors);
return prog;
out_err:
prog = orig_prog;
if (header)
bpf_jit_binary_free(header);
goto out;
}
arch/mips/net/bpf_jit_comp.h 0 → 100644
View file @ f438ee21
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Just-In-Time compiler for eBPF bytecode on 32-bit and 64-bit MIPS.
*
* Copyright (c) 2021 Anyfi Networks AB.
* Author: Johan Almbladh <johan.almbladh@gmail.com>
*
* Based on code and ideas from
* Copyright (c) 2017 Cavium, Inc.
* Copyright (c) 2017 Shubham Bansal <illusionist.neo@gmail.com>
* Copyright (c) 2011 Mircea Gherzan <mgherzan@gmail.com>
*/
#ifndef _BPF_JIT_COMP_H
#define _BPF_JIT_COMP_H
/* MIPS registers */
#define MIPS_R_ZERO 0 /* Const zero */
#define MIPS_R_AT 1 /* Asm temp */
#define MIPS_R_V0 2 /* Result */
#define MIPS_R_V1 3 /* Result */
#define MIPS_R_A0 4 /* Argument */
#define MIPS_R_A1 5 /* Argument */
#define MIPS_R_A2 6 /* Argument */
#define MIPS_R_A3 7 /* Argument */
#define MIPS_R_A4 8 /* Arg (n64) */
#define MIPS_R_A5 9 /* Arg (n64) */
#define MIPS_R_A6 10 /* Arg (n64) */
#define MIPS_R_A7 11 /* Arg (n64) */
#define MIPS_R_T0 8 /* Temp (o32) */
#define MIPS_R_T1 9 /* Temp (o32) */
#define MIPS_R_T2 10 /* Temp (o32) */
#define MIPS_R_T3 11 /* Temp (o32) */
#define MIPS_R_T4 12 /* Temporary */
#define MIPS_R_T5 13 /* Temporary */
#define MIPS_R_T6 14 /* Temporary */
#define MIPS_R_T7 15 /* Temporary */
#define MIPS_R_S0 16 /* Saved */
#define MIPS_R_S1 17 /* Saved */
#define MIPS_R_S2 18 /* Saved */
#define MIPS_R_S3 19 /* Saved */
#define MIPS_R_S4 20 /* Saved */
#define MIPS_R_S5 21 /* Saved */
#define MIPS_R_S6 22 /* Saved */
#define MIPS_R_S7 23 /* Saved */
#define MIPS_R_T8 24 /* Temporary */
#define MIPS_R_T9 25 /* Temporary */
/* MIPS_R_K0 26 Reserved */
/* MIPS_R_K1 27 Reserved */
#define MIPS_R_GP 28 /* Global ptr */
#define MIPS_R_SP 29 /* Stack ptr */
#define MIPS_R_FP 30 /* Frame ptr */
#define MIPS_R_RA 31 /* Return */
/*
* Jump address mask for immediate jumps. The four most significant bits
* must be equal to PC.
*/
#define MIPS_JMP_MASK 0x0fffffffUL
/* Maximum number of iterations in offset table computation */
#define JIT_MAX_ITERATIONS 8
/*
* Jump pseudo-instructions used internally
* for branch conversion and branch optimization.
*/
#define JIT_JNSET 0xe0
#define JIT_JNOP 0xf0
/* Descriptor flag for PC-relative branch conversion */
#define JIT_DESC_CONVERT BIT(31)
/* JIT context for an eBPF program */
struct jit_context {
struct bpf_prog *program; /* The eBPF program being JITed */
u32 *descriptors; /* eBPF to JITed CPU insn descriptors */
u32 *target; /* JITed code buffer */
u32 bpf_index; /* Index of current BPF program insn */
u32 jit_index; /* Index of current JIT target insn */
u32 changes; /* Number of PC-relative branch conv */
u32 accessed; /* Bit mask of read eBPF registers */
u32 clobbered; /* Bit mask of modified CPU registers */
u32 stack_size; /* Total allocated stack size in bytes */
u32 saved_size; /* Size of callee-saved registers */
u32 stack_used; /* Stack size used for function calls */
};
/* Emit the instruction if the JIT memory space has been allocated */
#define __emit(ctx, func, ...) \
do { \
if ((ctx)->target != NULL) { \
u32 *p = &(ctx)->target[ctx->jit_index]; \
uasm_i_##func(&p, ##__VA_ARGS__); \
} \
(ctx)->jit_index++; \
} while (0)
#define emit(...) __emit(__VA_ARGS__)
/* Workaround for R10000 ll/sc errata */
#ifdef CONFIG_WAR_R10000
#define LLSC_beqz beqzl
#else
#define LLSC_beqz beqz
#endif
/* Workaround for Loongson-3 ll/sc errata */
#ifdef CONFIG_CPU_LOONGSON3_WORKAROUNDS
#define LLSC_sync(ctx) emit(ctx, sync, 0)
#define LLSC_offset 4
#else
#define LLSC_sync(ctx)
#define LLSC_offset 0
#endif
/* Workaround for Loongson-2F jump errata */
#ifdef CONFIG_CPU_JUMP_WORKAROUNDS
#define JALR_MASK 0xffffffffcfffffffULL
#else
#define JALR_MASK (~0ULL)
#endif
/*
* Mark a BPF register as accessed, it needs to be
* initialized by the program if expected, e.g. FP.
*/
static inline void access_reg(struct jit_context *ctx, u8 reg)
{
ctx->accessed |= BIT(reg);
}
/*
* Mark a CPU register as clobbered, it needs to be
* saved/restored by the program if callee-saved.
*/
static inline void clobber_reg(struct jit_context *ctx, u8 reg)
{
ctx->clobbered |= BIT(reg);
}
/*
* Push registers on the stack, starting at a given depth from the stack
* pointer and increasing. The next depth to be written is returned.
*/
int push_regs(struct jit_context *ctx, u32 mask, u32 excl, int depth);
/*
* Pop registers from the stack, starting at a given depth from the stack
* pointer and increasing. The next depth to be read is returned.
*/
int pop_regs(struct jit_context *ctx, u32 mask, u32 excl, int depth);
/* Compute the 28-bit jump target address from a BPF program location */
int get_target(struct jit_context *ctx, u32 loc);
/* Compute the PC-relative offset to relative BPF program offset */
int get_offset(const struct jit_context *ctx, int off);
/* dst = imm (32-bit) */
void emit_mov_i(struct jit_context *ctx, u8 dst, s32 imm);
/* dst = src (32-bit) */
void emit_mov_r(struct jit_context *ctx, u8 dst, u8 src);
/* Validate ALU/ALU64 immediate range */
bool valid_alu_i(u8 op, s32 imm);
/* Rewrite ALU/ALU64 immediate operation */
bool rewrite_alu_i(u8 op, s32 imm, u8 *alu, s32 *val);
/* ALU immediate operation (32-bit) */
void emit_alu_i(struct jit_context *ctx, u8 dst, s32 imm, u8 op);
/* ALU register operation (32-bit) */
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op);
/* Atomic read-modify-write (32-bit) */
void emit_atomic_r(struct jit_context *ctx, u8 dst, u8 src, s16 off, u8 code);
/* Atomic compare-and-exchange (32-bit) */
void emit_cmpxchg_r(struct jit_context *ctx, u8 dst, u8 src, u8 res, s16 off);
/* Swap bytes and truncate a register word or half word */
void emit_bswap_r(struct jit_context *ctx, u8 dst, u32 width);
/* Validate JMP/JMP32 immediate range */
bool valid_jmp_i(u8 op, s32 imm);
/* Prepare a PC-relative jump operation with immediate conditional */
void setup_jmp_i(struct jit_context *ctx, s32 imm, u8 width,
u8 bpf_op, s16 bpf_off, u8 *jit_op, s32 *jit_off);
/* Prepare a PC-relative jump operation with register conditional */
void setup_jmp_r(struct jit_context *ctx, bool same_reg,
u8 bpf_op, s16 bpf_off, u8 *jit_op, s32 *jit_off);
/* Finish a PC-relative jump operation */
int finish_jmp(struct jit_context *ctx, u8 jit_op, s16 bpf_off);
/* Conditional JMP/JMP32 immediate */
void emit_jmp_i(struct jit_context *ctx, u8 dst, s32 imm, s32 off, u8 op);
/* Conditional JMP/JMP32 register */
void emit_jmp_r(struct jit_context *ctx, u8 dst, u8 src, s32 off, u8 op);
/* Jump always */
int emit_ja(struct jit_context *ctx, s16 off);
/* Jump to epilogue */
int emit_exit(struct jit_context *ctx);
/*
* Build program prologue to set up the stack and registers.
* This function is implemented separately for 32-bit and 64-bit JITs.
*/
void build_prologue(struct jit_context *ctx);
/*
* Build the program epilogue to restore the stack and registers.
* This function is implemented separately for 32-bit and 64-bit JITs.
*/
void build_epilogue(struct jit_context *ctx, int dest_reg);
/*
* Convert an eBPF instruction to native instruction, i.e
* JITs an eBPF instruction.
* Returns :
* 0 - Successfully JITed an 8-byte eBPF instruction
* >0 - Successfully JITed a 16-byte eBPF instruction
* <0 - Failed to JIT.
* This function is implemented separately for 32-bit and 64-bit JITs.
*/
int build_insn(const struct bpf_insn *insn, struct jit_context *ctx);
#endif /* _BPF_JIT_COMP_H */
arch/mips/net/bpf_jit_comp32.c 0 → 100644
View file @ f438ee21
// SPDX-License-Identifier: GPL-2.0-only
/*
* Just-In-Time compiler for eBPF bytecode on MIPS.
* Implementation of JIT functions for 32-bit CPUs.
*
* Copyright (c) 2021 Anyfi Networks AB.
* Author: Johan Almbladh <johan.almbladh@gmail.com>
*
* Based on code and ideas from
* Copyright (c) 2017 Cavium, Inc.
* Copyright (c) 2017 Shubham Bansal <illusionist.neo@gmail.com>
* Copyright (c) 2011 Mircea Gherzan <mgherzan@gmail.com>
*/
#include <linux/math64.h>
#include <linux/errno.h>
#include <linux/filter.h>
#include <linux/bpf.h>
#include <asm/cpu-features.h>
#include <asm/isa-rev.h>
#include <asm/uasm.h>
#include "bpf_jit_comp.h"
/* MIPS a4-a7 are not available in the o32 ABI */
#undef MIPS_R_A4
#undef MIPS_R_A5
#undef MIPS_R_A6
#undef MIPS_R_A7
/* Stack is 8-byte aligned in o32 ABI */
#define MIPS_STACK_ALIGNMENT 8
/*
* The top 16 bytes of a stack frame is reserved for the callee in O32 ABI.
* This corresponds to stack space for register arguments a0-a3.
*/
#define JIT_RESERVED_STACK 16
/* Temporary 64-bit register used by JIT */
#define JIT_REG_TMP MAX_BPF_JIT_REG
/*
* Number of prologue bytes to skip when doing a tail call.
* Tail call count (TCC) initialization (8 bytes) always, plus
* R0-to-v0 assignment (4 bytes) if big endian.
*/
#ifdef __BIG_ENDIAN
#define JIT_TCALL_SKIP 12
#else
#define JIT_TCALL_SKIP 8
#endif
/* CPU registers holding the callee return value */
#define JIT_RETURN_REGS \
(BIT(MIPS_R_V0) | \
BIT(MIPS_R_V1))
/* CPU registers arguments passed to callee directly */
#define JIT_ARG_REGS \
(BIT(MIPS_R_A0) | \
BIT(MIPS_R_A1) | \
BIT(MIPS_R_A2) | \
BIT(MIPS_R_A3))
/* CPU register arguments passed to callee on stack */
#define JIT_STACK_REGS \
(BIT(MIPS_R_T0) | \
BIT(MIPS_R_T1) | \
BIT(MIPS_R_T2) | \
BIT(MIPS_R_T3) | \
BIT(MIPS_R_T4) | \
BIT(MIPS_R_T5))
/* Caller-saved CPU registers */
#define JIT_CALLER_REGS \
(JIT_RETURN_REGS | \
JIT_ARG_REGS | \
JIT_STACK_REGS)
/* Callee-saved CPU registers */
#define JIT_CALLEE_REGS \
(BIT(MIPS_R_S0) | \
BIT(MIPS_R_S1) | \
BIT(MIPS_R_S2) | \
BIT(MIPS_R_S3) | \
BIT(MIPS_R_S4) | \
BIT(MIPS_R_S5) | \
BIT(MIPS_R_S6) | \
BIT(MIPS_R_S7) | \
BIT(MIPS_R_GP) | \
BIT(MIPS_R_FP) | \
BIT(MIPS_R_RA))
/*
* Mapping of 64-bit eBPF registers to 32-bit native MIPS registers.
*
* 1) Native register pairs are ordered according to CPU endiannes, following
* the MIPS convention for passing 64-bit arguments and return values.
* 2) The eBPF return value, arguments and callee-saved registers are mapped
* to their native MIPS equivalents.
* 3) Since the 32 highest bits in the eBPF FP register are always zero,
* only one general-purpose register is actually needed for the mapping.
* We use the fp register for this purpose, and map the highest bits to
* the MIPS register r0 (zero).
* 4) We use the MIPS gp and at registers as internal temporary registers
* for constant blinding. The gp register is callee-saved.
* 5) One 64-bit temporary register is mapped for use when sign-extending
* immediate operands. MIPS registers t6-t9 are available to the JIT
* for as temporaries when implementing complex 64-bit operations.
*
* With this scheme all eBPF registers are being mapped to native MIPS
* registers without having to use any stack scratch space. The direct
* register mapping (2) simplifies the handling of function calls.
*/
static const u8 bpf2mips32[][2] = {
/* Return value from in-kernel function, and exit value from eBPF */
[BPF_REG_0] = {MIPS_R_V1, MIPS_R_V0},
/* Arguments from eBPF program to in-kernel function */
[BPF_REG_1] = {MIPS_R_A1, MIPS_R_A0},
[BPF_REG_2] = {MIPS_R_A3, MIPS_R_A2},
/* Remaining arguments, to be passed on the stack per O32 ABI */
[BPF_REG_3] = {MIPS_R_T1, MIPS_R_T0},
[BPF_REG_4] = {MIPS_R_T3, MIPS_R_T2},
[BPF_REG_5] = {MIPS_R_T5, MIPS_R_T4},
/* Callee-saved registers that in-kernel function will preserve */
[BPF_REG_6] = {MIPS_R_S1, MIPS_R_S0},
[BPF_REG_7] = {MIPS_R_S3, MIPS_R_S2},
[BPF_REG_8] = {MIPS_R_S5, MIPS_R_S4},
[BPF_REG_9] = {MIPS_R_S7, MIPS_R_S6},
/* Read-only frame pointer to access the eBPF stack */
#ifdef __BIG_ENDIAN
[BPF_REG_FP] = {MIPS_R_FP, MIPS_R_ZERO},
#else
[BPF_REG_FP] = {MIPS_R_ZERO, MIPS_R_FP},
#endif
/* Temporary register for blinding constants */
[BPF_REG_AX] = {MIPS_R_GP, MIPS_R_AT},
/* Temporary register for internal JIT use */
[JIT_REG_TMP] = {MIPS_R_T7, MIPS_R_T6},
};
/* Get low CPU register for a 64-bit eBPF register mapping */
static inline u8 lo(const u8 reg[])
{
#ifdef __BIG_ENDIAN
return reg[0];
#else
return reg[1];
#endif
}
/* Get high CPU register for a 64-bit eBPF register mapping */
static inline u8 hi(const u8 reg[])
{
#ifdef __BIG_ENDIAN
return reg[1];
#else
return reg[0];
#endif
}
/*
* Mark a 64-bit CPU register pair as clobbered, it needs to be
* saved/restored by the program if callee-saved.
*/
static void clobber_reg64(struct jit_context *ctx, const u8 reg[])
{
clobber_reg(ctx, reg[0]);
clobber_reg(ctx, reg[1]);
}
/* dst = imm (sign-extended) */
static void emit_mov_se_i64(struct jit_context *ctx, const u8 dst[], s32 imm)
{
emit_mov_i(ctx, lo(dst), imm);
if (imm < 0)
emit(ctx, addiu, hi(dst), MIPS_R_ZERO, -1);
else
emit(ctx, move, hi(dst), MIPS_R_ZERO);
clobber_reg64(ctx, dst);
}
/* Zero extension, if verifier does not do it for us */
static void emit_zext_ver(struct jit_context *ctx, const u8 dst[])
{
if (!ctx->program->aux->verifier_zext) {
emit(ctx, move, hi(dst), MIPS_R_ZERO);
clobber_reg(ctx, hi(dst));
}
}
/* Load delay slot, if ISA mandates it */
static void emit_load_delay(struct jit_context *ctx)
{
if (!cpu_has_mips_2_3_4_5_r)
emit(ctx, nop);
}
/* ALU immediate operation (64-bit) */
static void emit_alu_i64(struct jit_context *ctx,
const u8 dst[], s32 imm, u8 op)
{
u8 src = MIPS_R_T6;
/*
* ADD/SUB with all but the max negative imm can be handled by
* inverting the operation and the imm value, saving one insn.
*/
if (imm > S32_MIN && imm < 0)
switch (op) {
case BPF_ADD:
op = BPF_SUB;
imm = -imm;
break;
case BPF_SUB:
op = BPF_ADD;
imm = -imm;
break;
}
/* Move immediate to temporary register */
emit_mov_i(ctx, src, imm);
switch (op) {
/* dst = dst + imm */
case BPF_ADD:
emit(ctx, addu, lo(dst), lo(dst), src);
emit(ctx, sltu, MIPS_R_T9, lo(dst), src);
emit(ctx, addu, hi(dst), hi(dst), MIPS_R_T9);
if (imm < 0)
emit(ctx, addiu, hi(dst), hi(dst), -1);
break;
/* dst = dst - imm */
case BPF_SUB:
emit(ctx, sltu, MIPS_R_T9, lo(dst), src);
emit(ctx, subu, lo(dst), lo(dst), src);
emit(ctx, subu, hi(dst), hi(dst), MIPS_R_T9);
if (imm < 0)
emit(ctx, addiu, hi(dst), hi(dst), 1);
break;
/* dst = dst | imm */
case BPF_OR:
emit(ctx, or, lo(dst), lo(dst), src);
if (imm < 0)
emit(ctx, addiu, hi(dst), MIPS_R_ZERO, -1);
break;
/* dst = dst & imm */
case BPF_AND:
emit(ctx, and, lo(dst), lo(dst), src);
if (imm >= 0)
emit(ctx, move, hi(dst), MIPS_R_ZERO);
break;
/* dst = dst ^ imm */
case BPF_XOR:
emit(ctx, xor, lo(dst), lo(dst), src);
if (imm < 0) {
emit(ctx, subu, hi(dst), MIPS_R_ZERO, hi(dst));
emit(ctx, addiu, hi(dst), hi(dst), -1);
}
break;
}
clobber_reg64(ctx, dst);
}
/* ALU register operation (64-bit) */
static void emit_alu_r64(struct jit_context *ctx,
const u8 dst[], const u8 src[], u8 op)
{
switch (BPF_OP(op)) {
/* dst = dst + src */
case BPF_ADD:
if (src == dst) {
emit(ctx, srl, MIPS_R_T9, lo(dst), 31);
emit(ctx, addu, lo(dst), lo(dst), lo(dst));
} else {
emit(ctx, addu, lo(dst), lo(dst), lo(src));
emit(ctx, sltu, MIPS_R_T9, lo(dst), lo(src));
}
emit(ctx, addu, hi(dst), hi(dst), hi(src));
emit(ctx, addu, hi(dst), hi(dst), MIPS_R_T9);
break;
/* dst = dst - src */
case BPF_SUB:
emit(ctx, sltu, MIPS_R_T9, lo(dst), lo(src));
emit(ctx, subu, lo(dst), lo(dst), lo(src));
emit(ctx, subu, hi(dst), hi(dst), hi(src));
emit(ctx, subu, hi(dst), hi(dst), MIPS_R_T9);
break;
/* dst = dst | src */
case BPF_OR:
emit(ctx, or, lo(dst), lo(dst), lo(src));
emit(ctx, or, hi(dst), hi(dst), hi(src));
break;
/* dst = dst & src */
case BPF_AND:
emit(ctx, and, lo(dst), lo(dst), lo(src));
emit(ctx, and, hi(dst), hi(dst), hi(src));
break;
/* dst = dst ^ src */
case BPF_XOR:
emit(ctx, xor, lo(dst), lo(dst), lo(src));
emit(ctx, xor, hi(dst), hi(dst), hi(src));
break;
}
clobber_reg64(ctx, dst);
}
/* ALU invert (64-bit) */
static void emit_neg_i64(struct jit_context *ctx, const u8 dst[])
{
emit(ctx, sltu, MIPS_R_T9, MIPS_R_ZERO, lo(dst));
emit(ctx, subu, lo(dst), MIPS_R_ZERO, lo(dst));
emit(ctx, subu, hi(dst), MIPS_R_ZERO, hi(dst));
emit(ctx, subu, hi(dst), hi(dst), MIPS_R_T9);
clobber_reg64(ctx, dst);
}
/* ALU shift immediate (64-bit) */
static void emit_shift_i64(struct jit_context *ctx,
const u8 dst[], u32 imm, u8 op)
{
switch (BPF_OP(op)) {
/* dst = dst << imm */
case BPF_LSH:
if (imm < 32) {
emit(ctx, srl, MIPS_R_T9, lo(dst), 32 - imm);
emit(ctx, sll, lo(dst), lo(dst), imm);
emit(ctx, sll, hi(dst), hi(dst), imm);
emit(ctx, or, hi(dst), hi(dst), MIPS_R_T9);
} else {
emit(ctx, sll, hi(dst), lo(dst), imm - 32);
emit(ctx, move, lo(dst), MIPS_R_ZERO);
}
break;
/* dst = dst >> imm */
case BPF_RSH:
if (imm < 32) {
emit(ctx, sll, MIPS_R_T9, hi(dst), 32 - imm);
emit(ctx, srl, lo(dst), lo(dst), imm);
emit(ctx, srl, hi(dst), hi(dst), imm);
emit(ctx, or, lo(dst), lo(dst), MIPS_R_T9);
} else {
emit(ctx, srl, lo(dst), hi(dst), imm - 32);
emit(ctx, move, hi(dst), MIPS_R_ZERO);
}
break;
/* dst = dst >> imm (arithmetic) */
case BPF_ARSH:
if (imm < 32) {
emit(ctx, sll, MIPS_R_T9, hi(dst), 32 - imm);
emit(ctx, srl, lo(dst), lo(dst), imm);
emit(ctx, sra, hi(dst), hi(dst), imm);
emit(ctx, or, lo(dst), lo(dst), MIPS_R_T9);
} else {
emit(ctx, sra, lo(dst), hi(dst), imm - 32);
emit(ctx, sra, hi(dst), hi(dst), 31);
}
break;
}
clobber_reg64(ctx, dst);
}
/* ALU shift register (64-bit) */
static void emit_shift_r64(struct jit_context *ctx,
const u8 dst[], u8 src, u8 op)
{
u8 t1 = MIPS_R_T8;
u8 t2 = MIPS_R_T9;
emit(ctx, andi, t1, src, 32); /* t1 = src & 32 */
emit(ctx, beqz, t1, 16); /* PC += 16 if t1 == 0 */
emit(ctx, nor, t2, src, MIPS_R_ZERO); /* t2 = ~src (delay slot) */
switch (BPF_OP(op)) {
/* dst = dst << src */
case BPF_LSH:
/* Next: shift >= 32 */
emit(ctx, sllv, hi(dst), lo(dst), src); /* dh = dl << src */
emit(ctx, move, lo(dst), MIPS_R_ZERO); /* dl = 0 */
emit(ctx, b, 20); /* PC += 20 */
/* +16: shift < 32 */
emit(ctx, srl, t1, lo(dst), 1); /* t1 = dl >> 1 */
emit(ctx, srlv, t1, t1, t2); /* t1 = t1 >> t2 */
emit(ctx, sllv, lo(dst), lo(dst), src); /* dl = dl << src */
emit(ctx, sllv, hi(dst), hi(dst), src); /* dh = dh << src */
emit(ctx, or, hi(dst), hi(dst), t1); /* dh = dh | t1 */
break;
/* dst = dst >> src */
case BPF_RSH:
/* Next: shift >= 32 */
emit(ctx, srlv, lo(dst), hi(dst), src); /* dl = dh >> src */
emit(ctx, move, hi(dst), MIPS_R_ZERO); /* dh = 0 */
emit(ctx, b, 20); /* PC += 20 */
/* +16: shift < 32 */
emit(ctx, sll, t1, hi(dst), 1); /* t1 = dl << 1 */
emit(ctx, sllv, t1, t1, t2); /* t1 = t1 << t2 */
emit(ctx, srlv, lo(dst), lo(dst), src); /* dl = dl >> src */
emit(ctx, srlv, hi(dst), hi(dst), src); /* dh = dh >> src */
emit(ctx, or, lo(dst), lo(dst), t1); /* dl = dl | t1 */
break;
/* dst = dst >> src (arithmetic) */
case BPF_ARSH:
/* Next: shift >= 32 */
emit(ctx, srav, lo(dst), hi(dst), src); /* dl = dh >>a src */
emit(ctx, sra, hi(dst), hi(dst), 31); /* dh = dh >>a 31 */
emit(ctx, b, 20); /* PC += 20 */
/* +16: shift < 32 */
emit(ctx, sll, t1, hi(dst), 1); /* t1 = dl << 1 */
emit(ctx, sllv, t1, t1, t2); /* t1 = t1 << t2 */
emit(ctx, srlv, lo(dst), lo(dst), src); /* dl = dl >>a src */
emit(ctx, srav, hi(dst), hi(dst), src); /* dh = dh >> src */
emit(ctx, or, lo(dst), lo(dst), t1); /* dl = dl | t1 */
break;
}
/* +20: Done */
clobber_reg64(ctx, dst);
}
/* ALU mul immediate (64x32-bit) */
static void emit_mul_i64(struct jit_context *ctx, const u8 dst[], s32 imm)
{
u8 src = MIPS_R_T6;
u8 tmp = MIPS_R_T9;
switch (imm) {
/* dst = dst * 1 is a no-op */
case 1:
break;
/* dst = dst * -1 */
case -1:
emit_neg_i64(ctx, dst);
break;
case 0:
emit_mov_r(ctx, lo(dst), MIPS_R_ZERO);
emit_mov_r(ctx, hi(dst), MIPS_R_ZERO);
break;
/* Full 64x32 multiply */
default:
/* hi(dst) = hi(dst) * src(imm) */
emit_mov_i(ctx, src, imm);
if (cpu_has_mips32r1 || cpu_has_mips32r6) {
emit(ctx, mul, hi(dst), hi(dst), src);
} else {
emit(ctx, multu, hi(dst), src);
emit(ctx, mflo, hi(dst));
}
/* hi(dst) = hi(dst) - lo(dst) */
if (imm < 0)
emit(ctx, subu, hi(dst), hi(dst), lo(dst));
/* tmp = lo(dst) * src(imm) >> 32 */
/* lo(dst) = lo(dst) * src(imm) */
if (cpu_has_mips32r6) {
emit(ctx, muhu, tmp, lo(dst), src);
emit(ctx, mulu, lo(dst), lo(dst), src);
} else {
emit(ctx, multu, lo(dst), src);
emit(ctx, mflo, lo(dst));
emit(ctx, mfhi, tmp);
}
/* hi(dst) += tmp */
emit(ctx, addu, hi(dst), hi(dst), tmp);
clobber_reg64(ctx, dst);
break;
}
}
/* ALU mul register (64x64-bit) */
static void emit_mul_r64(struct jit_context *ctx,
const u8 dst[], const u8 src[])
{
u8 acc = MIPS_R_T8;
u8 tmp = MIPS_R_T9;
/* acc = hi(dst) * lo(src) */
if (cpu_has_mips32r1 || cpu_has_mips32r6) {
emit(ctx, mul, acc, hi(dst), lo(src));
} else {
emit(ctx, multu, hi(dst), lo(src));
emit(ctx, mflo, acc);
}
/* tmp = lo(dst) * hi(src) */
if (cpu_has_mips32r1 || cpu_has_mips32r6) {
emit(ctx, mul, tmp, lo(dst), hi(src));
} else {
emit(ctx, multu, lo(dst), hi(src));
emit(ctx, mflo, tmp);
}
/* acc += tmp */
emit(ctx, addu, acc, acc, tmp);
/* tmp = lo(dst) * lo(src) >> 32 */
/* lo(dst) = lo(dst) * lo(src) */
if (cpu_has_mips32r6) {
emit(ctx, muhu, tmp, lo(dst), lo(src));
emit(ctx, mulu, lo(dst), lo(dst), lo(src));
} else {
emit(ctx, multu, lo(dst), lo(src));
emit(ctx, mflo, lo(dst));
emit(ctx, mfhi, tmp);
}
/* hi(dst) = acc + tmp */
emit(ctx, addu, hi(dst), acc, tmp);
clobber_reg64(ctx, dst);
}
/* Helper function for 64-bit modulo */
static u64 jit_mod64(u64 a, u64 b)
{
u64 rem;
div64_u64_rem(a, b, &rem);
return rem;
}
/* ALU div/mod register (64-bit) */
static void emit_divmod_r64(struct jit_context *ctx,
const u8 dst[], const u8 src[], u8 op)
{
const u8 *r0 = bpf2mips32[BPF_REG_0]; /* Mapped to v0-v1 */
const u8 *r1 = bpf2mips32[BPF_REG_1]; /* Mapped to a0-a1 */
const u8 *r2 = bpf2mips32[BPF_REG_2]; /* Mapped to a2-a3 */
int exclude, k;
u32 addr = 0;
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
0, JIT_RESERVED_STACK);
/* Put 64-bit arguments 1 and 2 in registers a0-a3 */
for (k = 0; k < 2; k++) {
emit(ctx, move, MIPS_R_T9, src[k]);
emit(ctx, move, r1[k], dst[k]);
emit(ctx, move, r2[k], MIPS_R_T9);
}
/* Emit function call */
switch (BPF_OP(op)) {
/* dst = dst / src */
case BPF_DIV:
addr = (u32)&div64_u64;
break;
/* dst = dst % src */
case BPF_MOD:
addr = (u32)&jit_mod64;
break;
}
emit_mov_i(ctx, MIPS_R_T9, addr);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
/* Store the 64-bit result in dst */
emit(ctx, move, dst[0], r0[0]);
emit(ctx, move, dst[1], r0[1]);
/* Restore caller-saved registers, excluding the computed result */
exclude = BIT(lo(dst)) | BIT(hi(dst));
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
exclude, JIT_RESERVED_STACK);
emit_load_delay(ctx);
clobber_reg64(ctx, dst);
clobber_reg(ctx, MIPS_R_V0);
clobber_reg(ctx, MIPS_R_V1);
clobber_reg(ctx, MIPS_R_RA);
}
/* Swap bytes in a register word */
static void emit_swap8_r(struct jit_context *ctx, u8 dst, u8 src, u8 mask)
{
u8 tmp = MIPS_R_T9;
emit(ctx, and, tmp, src, mask); /* tmp = src & 0x00ff00ff */
emit(ctx, sll, tmp, tmp, 8); /* tmp = tmp << 8 */
emit(ctx, srl, dst, src, 8); /* dst = src >> 8 */
emit(ctx, and, dst, dst, mask); /* dst = dst & 0x00ff00ff */
emit(ctx, or, dst, dst, tmp); /* dst = dst | tmp */
}
/* Swap half words in a register word */
static void emit_swap16_r(struct jit_context *ctx, u8 dst, u8 src)
{
u8 tmp = MIPS_R_T9;
emit(ctx, sll, tmp, src, 16); /* tmp = src << 16 */
emit(ctx, srl, dst, src, 16); /* dst = src >> 16 */
emit(ctx, or, dst, dst, tmp); /* dst = dst | tmp */
}
/* Swap bytes and truncate a register double word, word or half word */
static void emit_bswap_r64(struct jit_context *ctx, const u8 dst[], u32 width)
{
u8 tmp = MIPS_R_T8;
switch (width) {
/* Swap bytes in a double word */
case 64:
if (cpu_has_mips32r2 || cpu_has_mips32r6) {
emit(ctx, rotr, tmp, hi(dst), 16);
emit(ctx, rotr, hi(dst), lo(dst), 16);
emit(ctx, wsbh, lo(dst), tmp);
emit(ctx, wsbh, hi(dst), hi(dst));
} else {
emit_swap16_r(ctx, tmp, lo(dst));
emit_swap16_r(ctx, lo(dst), hi(dst));
emit(ctx, move, hi(dst), tmp);
emit(ctx, lui, tmp, 0xff); /* tmp = 0x00ff0000 */
emit(ctx, ori, tmp, tmp, 0xff); /* tmp = 0x00ff00ff */
emit_swap8_r(ctx, lo(dst), lo(dst), tmp);
emit_swap8_r(ctx, hi(dst), hi(dst), tmp);
}
break;
/* Swap bytes in a word */
/* Swap bytes in a half word */
case 32:
case 16:
emit_bswap_r(ctx, lo(dst), width);
emit(ctx, move, hi(dst), MIPS_R_ZERO);
break;
}
clobber_reg64(ctx, dst);
}
/* Truncate a register double word, word or half word */
static void emit_trunc_r64(struct jit_context *ctx, const u8 dst[], u32 width)
{
switch (width) {
case 64:
break;
/* Zero-extend a word */
case 32:
emit(ctx, move, hi(dst), MIPS_R_ZERO);
clobber_reg(ctx, hi(dst));
break;
/* Zero-extend a half word */
case 16:
emit(ctx, move, hi(dst), MIPS_R_ZERO);
emit(ctx, andi, lo(dst), lo(dst), 0xffff);
clobber_reg64(ctx, dst);
break;
}
}
/* Load operation: dst = *(size*)(src + off) */
static void emit_ldx(struct jit_context *ctx,
const u8 dst[], u8 src, s16 off, u8 size)
{
switch (size) {
/* Load a byte */
case BPF_B:
emit(ctx, lbu, lo(dst), off, src);
emit(ctx, move, hi(dst), MIPS_R_ZERO);
break;
/* Load a half word */
case BPF_H:
emit(ctx, lhu, lo(dst), off, src);
emit(ctx, move, hi(dst), MIPS_R_ZERO);
break;
/* Load a word */
case BPF_W:
emit(ctx, lw, lo(dst), off, src);
emit(ctx, move, hi(dst), MIPS_R_ZERO);
break;
/* Load a double word */
case BPF_DW:
if (dst[1] == src) {
emit(ctx, lw, dst[0], off + 4, src);
emit(ctx, lw, dst[1], off, src);
} else {
emit(ctx, lw, dst[1], off, src);
emit(ctx, lw, dst[0], off + 4, src);
}
emit_load_delay(ctx);
break;
}
clobber_reg64(ctx, dst);
}
/* Store operation: *(size *)(dst + off) = src */
static void emit_stx(struct jit_context *ctx,
const u8 dst, const u8 src[], s16 off, u8 size)
{
switch (size) {
/* Store a byte */
case BPF_B:
emit(ctx, sb, lo(src), off, dst);
break;
/* Store a half word */
case BPF_H:
emit(ctx, sh, lo(src), off, dst);
break;
/* Store a word */
case BPF_W:
emit(ctx, sw, lo(src), off, dst);
break;
/* Store a double word */
case BPF_DW:
emit(ctx, sw, src[1], off, dst);
emit(ctx, sw, src[0], off + 4, dst);
break;
}
}
/* Atomic read-modify-write (32-bit, non-ll/sc fallback) */
static void emit_atomic_r32(struct jit_context *ctx,
u8 dst, u8 src, s16 off, u8 code)
{
u32 exclude = 0;
u32 addr = 0;
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
0, JIT_RESERVED_STACK);
/*
* Argument 1: dst+off if xchg, otherwise src, passed in register a0
* Argument 2: src if xchg, othersize dst+off, passed in register a1
*/
emit(ctx, move, MIPS_R_T9, dst);
if (code == BPF_XCHG) {
emit(ctx, move, MIPS_R_A1, src);
emit(ctx, addiu, MIPS_R_A0, MIPS_R_T9, off);
} else {
emit(ctx, move, MIPS_R_A0, src);
emit(ctx, addiu, MIPS_R_A1, MIPS_R_T9, off);
}
/* Emit function call */
switch (code) {
case BPF_ADD:
addr = (u32)&atomic_add;
break;
case BPF_ADD | BPF_FETCH:
addr = (u32)&atomic_fetch_add;
break;
case BPF_SUB:
addr = (u32)&atomic_sub;
break;
case BPF_SUB | BPF_FETCH:
addr = (u32)&atomic_fetch_sub;
break;
case BPF_OR:
addr = (u32)&atomic_or;
break;
case BPF_OR | BPF_FETCH:
addr = (u32)&atomic_fetch_or;
break;
case BPF_AND:
addr = (u32)&atomic_and;
break;
case BPF_AND | BPF_FETCH:
addr = (u32)&atomic_fetch_and;
break;
case BPF_XOR:
addr = (u32)&atomic_xor;
break;
case BPF_XOR | BPF_FETCH:
addr = (u32)&atomic_fetch_xor;
break;
case BPF_XCHG:
addr = (u32)&atomic_xchg;
break;
}
emit_mov_i(ctx, MIPS_R_T9, addr);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
/* Update src register with old value, if specified */
if (code & BPF_FETCH) {
emit(ctx, move, src, MIPS_R_V0);
exclude = BIT(src);
clobber_reg(ctx, src);
}
/* Restore caller-saved registers, except any fetched value */
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
exclude, JIT_RESERVED_STACK);
emit_load_delay(ctx);
clobber_reg(ctx, MIPS_R_RA);
}
/* Helper function for 64-bit atomic exchange */
static s64 jit_xchg64(s64 a, atomic64_t *v)
{
return atomic64_xchg(v, a);
}
/* Atomic read-modify-write (64-bit) */
static void emit_atomic_r64(struct jit_context *ctx,
u8 dst, const u8 src[], s16 off, u8 code)
{
const u8 *r0 = bpf2mips32[BPF_REG_0]; /* Mapped to v0-v1 */
const u8 *r1 = bpf2mips32[BPF_REG_1]; /* Mapped to a0-a1 */
u32 exclude = 0;
u32 addr = 0;
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
0, JIT_RESERVED_STACK);
/*
* Argument 1: 64-bit src, passed in registers a0-a1
* Argument 2: 32-bit dst+off, passed in register a2
*/
emit(ctx, move, MIPS_R_T9, dst);
emit(ctx, move, r1[0], src[0]);
emit(ctx, move, r1[1], src[1]);
emit(ctx, addiu, MIPS_R_A2, MIPS_R_T9, off);
/* Emit function call */
switch (code) {
case BPF_ADD:
addr = (u32)&atomic64_add;
break;
case BPF_ADD | BPF_FETCH:
addr = (u32)&atomic64_fetch_add;
break;
case BPF_SUB:
addr = (u32)&atomic64_sub;
break;
case BPF_SUB | BPF_FETCH:
addr = (u32)&atomic64_fetch_sub;
break;
case BPF_OR:
addr = (u32)&atomic64_or;
break;
case BPF_OR | BPF_FETCH:
addr = (u32)&atomic64_fetch_or;
break;
case BPF_AND:
addr = (u32)&atomic64_and;
break;
case BPF_AND | BPF_FETCH:
addr = (u32)&atomic64_fetch_and;
break;
case BPF_XOR:
addr = (u32)&atomic64_xor;
break;
case BPF_XOR | BPF_FETCH:
addr = (u32)&atomic64_fetch_xor;
break;
case BPF_XCHG:
addr = (u32)&jit_xchg64;
break;
}
emit_mov_i(ctx, MIPS_R_T9, addr);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
/* Update src register with old value, if specified */
if (code & BPF_FETCH) {
emit(ctx, move, lo(src), lo(r0));
emit(ctx, move, hi(src), hi(r0));
exclude = BIT(src[0]) | BIT(src[1]);
clobber_reg64(ctx, src);
}
/* Restore caller-saved registers, except any fetched value */
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
exclude, JIT_RESERVED_STACK);
emit_load_delay(ctx);
clobber_reg(ctx, MIPS_R_RA);
}
/* Atomic compare-and-exchange (32-bit, non-ll/sc fallback) */
static void emit_cmpxchg_r32(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
const u8 *r0 = bpf2mips32[BPF_REG_0];
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
JIT_RETURN_REGS, JIT_RESERVED_STACK + 2 * sizeof(u32));
/*
* Argument 1: 32-bit dst+off, passed in register a0
* Argument 2: 32-bit r0, passed in register a1
* Argument 3: 32-bit src, passed in register a2
*/
emit(ctx, addiu, MIPS_R_T9, dst, off);
emit(ctx, move, MIPS_R_T8, src);
emit(ctx, move, MIPS_R_A1, lo(r0));
emit(ctx, move, MIPS_R_A0, MIPS_R_T9);
emit(ctx, move, MIPS_R_A2, MIPS_R_T8);
/* Emit function call */
emit_mov_i(ctx, MIPS_R_T9, (u32)&atomic_cmpxchg);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
#ifdef __BIG_ENDIAN
emit(ctx, move, lo(r0), MIPS_R_V0);
#endif
/* Restore caller-saved registers, except the return value */
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
JIT_RETURN_REGS, JIT_RESERVED_STACK + 2 * sizeof(u32));
emit_load_delay(ctx);
clobber_reg(ctx, MIPS_R_V0);
clobber_reg(ctx, MIPS_R_V1);
clobber_reg(ctx, MIPS_R_RA);
}
/* Atomic compare-and-exchange (64-bit) */
static void emit_cmpxchg_r64(struct jit_context *ctx,
u8 dst, const u8 src[], s16 off)
{
const u8 *r0 = bpf2mips32[BPF_REG_0];
const u8 *r2 = bpf2mips32[BPF_REG_2];
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
JIT_RETURN_REGS, JIT_RESERVED_STACK + 2 * sizeof(u32));
/*
* Argument 1: 32-bit dst+off, passed in register a0 (a1 unused)
* Argument 2: 64-bit r0, passed in registers a2-a3
* Argument 3: 64-bit src, passed on stack
*/
push_regs(ctx, BIT(src[0]) | BIT(src[1]), 0, JIT_RESERVED_STACK);
emit(ctx, addiu, MIPS_R_T9, dst, off);
emit(ctx, move, r2[0], r0[0]);
emit(ctx, move, r2[1], r0[1]);
emit(ctx, move, MIPS_R_A0, MIPS_R_T9);
/* Emit function call */
emit_mov_i(ctx, MIPS_R_T9, (u32)&atomic64_cmpxchg);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
/* Restore caller-saved registers, except the return value */
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS,
JIT_RETURN_REGS, JIT_RESERVED_STACK + 2 * sizeof(u32));
emit_load_delay(ctx);
clobber_reg(ctx, MIPS_R_V0);
clobber_reg(ctx, MIPS_R_V1);
clobber_reg(ctx, MIPS_R_RA);
}
/*
* Conditional movz or an emulated equivalent.
* Note that the rs register may be modified.
*/
static void emit_movz_r(struct jit_context *ctx, u8 rd, u8 rs, u8 rt)
{
if (cpu_has_mips_2) {
emit(ctx, movz, rd, rs, rt); /* rd = rt ? rd : rs */
} else if (cpu_has_mips32r6) {
if (rs != MIPS_R_ZERO)
emit(ctx, seleqz, rs, rs, rt); /* rs = 0 if rt == 0 */
emit(ctx, selnez, rd, rd, rt); /* rd = 0 if rt != 0 */
if (rs != MIPS_R_ZERO)
emit(ctx, or, rd, rd, rs); /* rd = rd | rs */
} else {
emit(ctx, bnez, rt, 8); /* PC += 8 if rd != 0 */
emit(ctx, nop); /* +0: delay slot */
emit(ctx, or, rd, rs, MIPS_R_ZERO); /* +4: rd = rs */
}
clobber_reg(ctx, rd);
clobber_reg(ctx, rs);
}
/*
* Conditional movn or an emulated equivalent.
* Note that the rs register may be modified.
*/
static void emit_movn_r(struct jit_context *ctx, u8 rd, u8 rs, u8 rt)
{
if (cpu_has_mips_2) {
emit(ctx, movn, rd, rs, rt); /* rd = rt ? rs : rd */
} else if (cpu_has_mips32r6) {
if (rs != MIPS_R_ZERO)
emit(ctx, selnez, rs, rs, rt); /* rs = 0 if rt == 0 */
emit(ctx, seleqz, rd, rd, rt); /* rd = 0 if rt != 0 */
if (rs != MIPS_R_ZERO)
emit(ctx, or, rd, rd, rs); /* rd = rd | rs */
} else {
emit(ctx, beqz, rt, 8); /* PC += 8 if rd == 0 */
emit(ctx, nop); /* +0: delay slot */
emit(ctx, or, rd, rs, MIPS_R_ZERO); /* +4: rd = rs */
}
clobber_reg(ctx, rd);
clobber_reg(ctx, rs);
}
/* Emulation of 64-bit sltiu rd, rs, imm, where imm may be S32_MAX + 1 */
static void emit_sltiu_r64(struct jit_context *ctx, u8 rd,
const u8 rs[], s64 imm)
{
u8 tmp = MIPS_R_T9;
if (imm < 0) {
emit_mov_i(ctx, rd, imm); /* rd = imm */
emit(ctx, sltu, rd, lo(rs), rd); /* rd = rsl < rd */
emit(ctx, sltiu, tmp, hi(rs), -1); /* tmp = rsh < ~0U */
emit(ctx, or, rd, rd, tmp); /* rd = rd | tmp */
} else { /* imm >= 0 */
if (imm > 0x7fff) {
emit_mov_i(ctx, rd, (s32)imm); /* rd = imm */
emit(ctx, sltu, rd, lo(rs), rd); /* rd = rsl < rd */
} else {
emit(ctx, sltiu, rd, lo(rs), imm); /* rd = rsl < imm */
}
emit_movn_r(ctx, rd, MIPS_R_ZERO, hi(rs)); /* rd = 0 if rsh */
}
}
/* Emulation of 64-bit sltu rd, rs, rt */
static void emit_sltu_r64(struct jit_context *ctx, u8 rd,
const u8 rs[], const u8 rt[])
{
u8 tmp = MIPS_R_T9;
emit(ctx, sltu, rd, lo(rs), lo(rt)); /* rd = rsl < rtl */
emit(ctx, subu, tmp, hi(rs), hi(rt)); /* tmp = rsh - rth */
emit_movn_r(ctx, rd, MIPS_R_ZERO, tmp); /* rd = 0 if tmp != 0 */
emit(ctx, sltu, tmp, hi(rs), hi(rt)); /* tmp = rsh < rth */
emit(ctx, or, rd, rd, tmp); /* rd = rd | tmp */
}
/* Emulation of 64-bit slti rd, rs, imm, where imm may be S32_MAX + 1 */
static void emit_slti_r64(struct jit_context *ctx, u8 rd,
const u8 rs[], s64 imm)
{
u8 t1 = MIPS_R_T8;
u8 t2 = MIPS_R_T9;
u8 cmp;
/*
* if ((rs < 0) ^ (imm < 0)) t1 = imm >u rsl
* else t1 = rsl <u imm
*/
emit_mov_i(ctx, rd, (s32)imm);
emit(ctx, sltu, t1, lo(rs), rd); /* t1 = rsl <u imm */
emit(ctx, sltu, t2, rd, lo(rs)); /* t2 = imm <u rsl */
emit(ctx, srl, rd, hi(rs), 31); /* rd = rsh >> 31 */
if (imm < 0)
emit_movz_r(ctx, t1, t2, rd); /* t1 = rd ? t1 : t2 */
else
emit_movn_r(ctx, t1, t2, rd); /* t1 = rd ? t2 : t1 */
/*
* if ((imm < 0 && rsh != 0xffffffff) ||
* (imm >= 0 && rsh != 0))
* t1 = 0
*/
if (imm < 0) {
emit(ctx, addiu, rd, hi(rs), 1); /* rd = rsh + 1 */
cmp = rd;
} else { /* imm >= 0 */
cmp = hi(rs);
}
emit_movn_r(ctx, t1, MIPS_R_ZERO, cmp); /* t1 = 0 if cmp != 0 */
/*
* if (imm < 0) rd = rsh < -1
* else rd = rsh != 0
* rd = rd | t1
*/
emit(ctx, slti, rd, hi(rs), imm < 0 ? -1 : 0); /* rd = rsh < hi(imm) */
emit(ctx, or, rd, rd, t1); /* rd = rd | t1 */
}
/* Emulation of 64-bit(slt rd, rs, rt) */
static void emit_slt_r64(struct jit_context *ctx, u8 rd,
const u8 rs[], const u8 rt[])
{
u8 t1 = MIPS_R_T7;
u8 t2 = MIPS_R_T8;
u8 t3 = MIPS_R_T9;
/*
* if ((rs < 0) ^ (rt < 0)) t1 = rtl <u rsl
* else t1 = rsl <u rtl
* if (rsh == rth) t1 = 0
*/
emit(ctx, sltu, t1, lo(rs), lo(rt)); /* t1 = rsl <u rtl */
emit(ctx, sltu, t2, lo(rt), lo(rs)); /* t2 = rtl <u rsl */
emit(ctx, xor, t3, hi(rs), hi(rt)); /* t3 = rlh ^ rth */
emit(ctx, srl, rd, t3, 31); /* rd = t3 >> 31 */
emit_movn_r(ctx, t1, t2, rd); /* t1 = rd ? t2 : t1 */
emit_movn_r(ctx, t1, MIPS_R_ZERO, t3); /* t1 = 0 if t3 != 0 */
/* rd = (rsh < rth) | t1 */
emit(ctx, slt, rd, hi(rs), hi(rt)); /* rd = rsh <s rth */
emit(ctx, or, rd, rd, t1); /* rd = rd | t1 */
}
/* Jump immediate (64-bit) */
static void emit_jmp_i64(struct jit_context *ctx,
const u8 dst[], s32 imm, s32 off, u8 op)
{
u8 tmp = MIPS_R_T6;
switch (op) {
/* No-op, used internally for branch optimization */
case JIT_JNOP:
break;
/* PC += off if dst == imm */
/* PC += off if dst != imm */
case BPF_JEQ:
case BPF_JNE:
if (imm >= -0x7fff && imm <= 0x8000) {
emit(ctx, addiu, tmp, lo(dst), -imm);
} else if ((u32)imm <= 0xffff) {
emit(ctx, xori, tmp, lo(dst), imm);
} else { /* Register fallback */
emit_mov_i(ctx, tmp, imm);
emit(ctx, xor, tmp, lo(dst), tmp);
}
if (imm < 0) { /* Compare sign extension */
emit(ctx, addu, MIPS_R_T9, hi(dst), 1);
emit(ctx, or, tmp, tmp, MIPS_R_T9);
} else { /* Compare zero extension */
emit(ctx, or, tmp, tmp, hi(dst));
}
if (op == BPF_JEQ)
emit(ctx, beqz, tmp, off);
else /* BPF_JNE */
emit(ctx, bnez, tmp, off);
break;
/* PC += off if dst & imm */
/* PC += off if (dst & imm) == 0 (not in BPF, used for long jumps) */
case BPF_JSET:
case JIT_JNSET:
if ((u32)imm <= 0xffff) {
emit(ctx, andi, tmp, lo(dst), imm);
} else { /* Register fallback */
emit_mov_i(ctx, tmp, imm);
emit(ctx, and, tmp, lo(dst), tmp);
}
if (imm < 0) /* Sign-extension pulls in high word */
emit(ctx, or, tmp, tmp, hi(dst));
if (op == BPF_JSET)
emit(ctx, bnez, tmp, off);
else /* JIT_JNSET */
emit(ctx, beqz, tmp, off);
break;
/* PC += off if dst > imm */
case BPF_JGT:
emit_sltiu_r64(ctx, tmp, dst, (s64)imm + 1);
emit(ctx, beqz, tmp, off);
break;
/* PC += off if dst >= imm */
case BPF_JGE:
emit_sltiu_r64(ctx, tmp, dst, imm);
emit(ctx, beqz, tmp, off);
break;
/* PC += off if dst < imm */
case BPF_JLT:
emit_sltiu_r64(ctx, tmp, dst, imm);
emit(ctx, bnez, tmp, off);
break;
/* PC += off if dst <= imm */
case BPF_JLE:
emit_sltiu_r64(ctx, tmp, dst, (s64)imm + 1);
emit(ctx, bnez, tmp, off);
break;
/* PC += off if dst > imm (signed) */
case BPF_JSGT:
emit_slti_r64(ctx, tmp, dst, (s64)imm + 1);
emit(ctx, beqz, tmp, off);
break;
/* PC += off if dst >= imm (signed) */
case BPF_JSGE:
emit_slti_r64(ctx, tmp, dst, imm);
emit(ctx, beqz, tmp, off);
break;
/* PC += off if dst < imm (signed) */
case BPF_JSLT:
emit_slti_r64(ctx, tmp, dst, imm);
emit(ctx, bnez, tmp, off);
break;
/* PC += off if dst <= imm (signed) */
case BPF_JSLE:
emit_slti_r64(ctx, tmp, dst, (s64)imm + 1);
emit(ctx, bnez, tmp, off);
break;
}
}
/* Jump register (64-bit) */
static void emit_jmp_r64(struct jit_context *ctx,
const u8 dst[], const u8 src[], s32 off, u8 op)
{
u8 t1 = MIPS_R_T6;
u8 t2 = MIPS_R_T7;
switch (op) {
/* No-op, used internally for branch optimization */
case JIT_JNOP:
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
case BPF_JEQ:
case BPF_JNE:
emit(ctx, subu, t1, lo(dst), lo(src));
emit(ctx, subu, t2, hi(dst), hi(src));
emit(ctx, or, t1, t1, t2);
if (op == BPF_JEQ)
emit(ctx, beqz, t1, off);
else /* BPF_JNE */
emit(ctx, bnez, t1, off);
break;
/* PC += off if dst & src */
/* PC += off if (dst & imm) == 0 (not in BPF, used for long jumps) */
case BPF_JSET:
case JIT_JNSET:
emit(ctx, and, t1, lo(dst), lo(src));
emit(ctx, and, t2, hi(dst), hi(src));
emit(ctx, or, t1, t1, t2);
if (op == BPF_JSET)
emit(ctx, bnez, t1, off);
else /* JIT_JNSET */
emit(ctx, beqz, t1, off);
break;
/* PC += off if dst > src */
case BPF_JGT:
emit_sltu_r64(ctx, t1, src, dst);
emit(ctx, bnez, t1, off);
break;
/* PC += off if dst >= src */
case BPF_JGE:
emit_sltu_r64(ctx, t1, dst, src);
emit(ctx, beqz, t1, off);
break;
/* PC += off if dst < src */
case BPF_JLT:
emit_sltu_r64(ctx, t1, dst, src);
emit(ctx, bnez, t1, off);
break;
/* PC += off if dst <= src */
case BPF_JLE:
emit_sltu_r64(ctx, t1, src, dst);
emit(ctx, beqz, t1, off);
break;
/* PC += off if dst > src (signed) */
case BPF_JSGT:
emit_slt_r64(ctx, t1, src, dst);
emit(ctx, bnez, t1, off);
break;
/* PC += off if dst >= src (signed) */
case BPF_JSGE:
emit_slt_r64(ctx, t1, dst, src);
emit(ctx, beqz, t1, off);
break;
/* PC += off if dst < src (signed) */
case BPF_JSLT:
emit_slt_r64(ctx, t1, dst, src);
emit(ctx, bnez, t1, off);
break;
/* PC += off if dst <= src (signed) */
case BPF_JSLE:
emit_slt_r64(ctx, t1, src, dst);
emit(ctx, beqz, t1, off);
break;
}
}
/* Function call */
static int emit_call(struct jit_context *ctx, const struct bpf_insn *insn)
{
bool fixed;
u64 addr;
/* Decode the call address */
if (bpf_jit_get_func_addr(ctx->program, insn, false,
&addr, &fixed) < 0)
return -1;
if (!fixed)
return -1;
/* Push stack arguments */
push_regs(ctx, JIT_STACK_REGS, 0, JIT_RESERVED_STACK);
/* Emit function call */
emit_mov_i(ctx, MIPS_R_T9, addr);
emit(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
emit(ctx, nop); /* Delay slot */
clobber_reg(ctx, MIPS_R_RA);
clobber_reg(ctx, MIPS_R_V0);
clobber_reg(ctx, MIPS_R_V1);
return 0;
}
/* Function tail call */
static int emit_tail_call(struct jit_context *ctx)
{
u8 ary = lo(bpf2mips32[BPF_REG_2]);
u8 ind = lo(bpf2mips32[BPF_REG_3]);
u8 t1 = MIPS_R_T8;
u8 t2 = MIPS_R_T9;
int off;
/*
* Tail call:
* eBPF R1 - function argument (context ptr), passed in a0-a1
* eBPF R2 - ptr to object with array of function entry points
* eBPF R3 - array index of function to be called
* stack[sz] - remaining tail call count, initialized in prologue
*/
/* if (ind >= ary->map.max_entries) goto out */
off = offsetof(struct bpf_array, map.max_entries);
if (off > 0x7fff)
return -1;
emit(ctx, lw, t1, off, ary); /* t1 = ary->map.max_entries*/
emit_load_delay(ctx); /* Load delay slot */
emit(ctx, sltu, t1, ind, t1); /* t1 = ind < t1 */
emit(ctx, beqz, t1, get_offset(ctx, 1)); /* PC += off(1) if t1 == 0 */
/* (next insn delay slot) */
/* if (TCC-- <= 0) goto out */
emit(ctx, lw, t2, ctx->stack_size, MIPS_R_SP); /* t2 = *(SP + size) */
emit_load_delay(ctx); /* Load delay slot */
emit(ctx, blez, t2, get_offset(ctx, 1)); /* PC += off(1) if t2 < 0 */
emit(ctx, addiu, t2, t2, -1); /* t2-- (delay slot) */
emit(ctx, sw, t2, ctx->stack_size, MIPS_R_SP); /* *(SP + size) = t2 */
/* prog = ary->ptrs[ind] */
off = offsetof(struct bpf_array, ptrs);
if (off > 0x7fff)
return -1;
emit(ctx, sll, t1, ind, 2); /* t1 = ind << 2 */
emit(ctx, addu, t1, t1, ary); /* t1 += ary */
emit(ctx, lw, t2, off, t1); /* t2 = *(t1 + off) */
emit_load_delay(ctx); /* Load delay slot */
/* if (prog == 0) goto out */
emit(ctx, beqz, t2, get_offset(ctx, 1)); /* PC += off(1) if t2 == 0 */
emit(ctx, nop); /* Delay slot */
/* func = prog->bpf_func + 8 (prologue skip offset) */
off = offsetof(struct bpf_prog, bpf_func);
if (off > 0x7fff)
return -1;
emit(ctx, lw, t1, off, t2); /* t1 = *(t2 + off) */
emit_load_delay(ctx); /* Load delay slot */
emit(ctx, addiu, t1, t1, JIT_TCALL_SKIP); /* t1 += skip (8 or 12) */
/* goto func */
build_epilogue(ctx, t1);
return 0;
}
/*
* Stack frame layout for a JITed program (stack grows down).
*
* Higher address : Caller's stack frame :
* :----------------------------:
* : 64-bit eBPF args r3-r5 :
* :----------------------------:
* : Reserved / tail call count :
* +============================+ <--- MIPS sp before call
* | Callee-saved registers, |
* | including RA and FP |
* +----------------------------+ <--- eBPF FP (MIPS zero,fp)
* | Local eBPF variables |
* | allocated by program |
* +----------------------------+
* | Reserved for caller-saved |
* | registers |
* +----------------------------+
* | Reserved for 64-bit eBPF |
* | args r3-r5 & args passed |
* | on stack in kernel calls |
* Lower address +============================+ <--- MIPS sp
*/
/* Build program prologue to set up the stack and registers */
void build_prologue(struct jit_context *ctx)
{
const u8 *r1 = bpf2mips32[BPF_REG_1];
const u8 *fp = bpf2mips32[BPF_REG_FP];
int stack, saved, locals, reserved;
/*
* The first two instructions initialize TCC in the reserved (for us)
* 16-byte area in the parent's stack frame. On a tail call, the
* calling function jumps into the prologue after these instructions.
*/
emit(ctx, ori, MIPS_R_T9, MIPS_R_ZERO,
min(MAX_TAIL_CALL_CNT + 1, 0xffff));
emit(ctx, sw, MIPS_R_T9, 0, MIPS_R_SP);
/*
* Register eBPF R1 contains the 32-bit context pointer argument.
* A 32-bit argument is always passed in MIPS register a0, regardless
* of CPU endianness. Initialize R1 accordingly and zero-extend.
*/
#ifdef __BIG_ENDIAN
emit(ctx, move, lo(r1), MIPS_R_A0);
#endif
/* === Entry-point for tail calls === */
/* Zero-extend the 32-bit argument */
emit(ctx, move, hi(r1), MIPS_R_ZERO);
/* If the eBPF frame pointer was accessed it must be saved */
if (ctx->accessed & BIT(BPF_REG_FP))
clobber_reg64(ctx, fp);
/* Compute the stack space needed for callee-saved registers */
saved = hweight32(ctx->clobbered & JIT_CALLEE_REGS) * sizeof(u32);
saved = ALIGN(saved, MIPS_STACK_ALIGNMENT);
/* Stack space used by eBPF program local data */
locals = ALIGN(ctx->program->aux->stack_depth, MIPS_STACK_ALIGNMENT);
/*
* If we are emitting function calls, reserve extra stack space for
* caller-saved registers and function arguments passed on the stack.
* The required space is computed automatically during resource
* usage discovery (pass 1).
*/
reserved = ctx->stack_used;
/* Allocate the stack frame */
stack = ALIGN(saved + locals + reserved, MIPS_STACK_ALIGNMENT);
emit(ctx, addiu, MIPS_R_SP, MIPS_R_SP, -stack);
/* Store callee-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLEE_REGS, 0, stack - saved);
/* Initialize the eBPF frame pointer if accessed */
if (ctx->accessed & BIT(BPF_REG_FP))
emit(ctx, addiu, lo(fp), MIPS_R_SP, stack - saved);
ctx->saved_size = saved;
ctx->stack_size = stack;
}
/* Build the program epilogue to restore the stack and registers */
void build_epilogue(struct jit_context *ctx, int dest_reg)
{
/* Restore callee-saved registers from stack */
pop_regs(ctx, ctx->clobbered & JIT_CALLEE_REGS, 0,
ctx->stack_size - ctx->saved_size);
/*
* A 32-bit return value is always passed in MIPS register v0,
* but on big-endian targets the low part of R0 is mapped to v1.
*/
#ifdef __BIG_ENDIAN
emit(ctx, move, MIPS_R_V0, MIPS_R_V1);
#endif
/* Jump to the return address and adjust the stack pointer */
emit(ctx, jr, dest_reg);
emit(ctx, addiu, MIPS_R_SP, MIPS_R_SP, ctx->stack_size);
}
/* Build one eBPF instruction */
int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
{
const u8 *dst = bpf2mips32[insn->dst_reg];
const u8 *src = bpf2mips32[insn->src_reg];
const u8 *res = bpf2mips32[BPF_REG_0];
const u8 *tmp = bpf2mips32[JIT_REG_TMP];
u8 code = insn->code;
s16 off = insn->off;
s32 imm = insn->imm;
s32 val, rel;
u8 alu, jmp;
switch (code) {
/* ALU operations */
/* dst = imm */
case BPF_ALU | BPF_MOV | BPF_K:
emit_mov_i(ctx, lo(dst), imm);
emit_zext_ver(ctx, dst);
break;
/* dst = src */
case BPF_ALU | BPF_MOV | BPF_X:
if (imm == 1) {
/* Special mov32 for zext */
emit_mov_i(ctx, hi(dst), 0);
} else {
emit_mov_r(ctx, lo(dst), lo(src));
emit_zext_ver(ctx, dst);
}
break;
/* dst = -dst */
case BPF_ALU | BPF_NEG:
emit_alu_i(ctx, lo(dst), 0, BPF_NEG);
emit_zext_ver(ctx, dst);
break;
/* dst = dst & imm */
/* dst = dst | imm */
/* dst = dst ^ imm */
/* dst = dst << imm */
/* dst = dst >> imm */
/* dst = dst >> imm (arithmetic) */
/* dst = dst + imm */
/* dst = dst - imm */
/* dst = dst * imm */
/* dst = dst / imm */
/* dst = dst % imm */
case BPF_ALU | BPF_OR | BPF_K:
case BPF_ALU | BPF_AND | BPF_K:
case BPF_ALU | BPF_XOR | BPF_K:
case BPF_ALU | BPF_LSH | BPF_K:
case BPF_ALU | BPF_RSH | BPF_K:
case BPF_ALU | BPF_ARSH | BPF_K:
case BPF_ALU | BPF_ADD | BPF_K:
case BPF_ALU | BPF_SUB | BPF_K:
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
emit_mov_i(ctx, MIPS_R_T6, imm);
emit_alu_r(ctx, lo(dst), MIPS_R_T6, BPF_OP(code));
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i(ctx, lo(dst), val, alu);
}
emit_zext_ver(ctx, dst);
break;
/* dst = dst & src */
/* dst = dst | src */
/* dst = dst ^ src */
/* dst = dst << src */
/* dst = dst >> src */
/* dst = dst >> src (arithmetic) */
/* dst = dst + src */
/* dst = dst - src */
/* dst = dst * src */
/* dst = dst / src */
/* dst = dst % src */
case BPF_ALU | BPF_AND | BPF_X:
case BPF_ALU | BPF_OR | BPF_X:
case BPF_ALU | BPF_XOR | BPF_X:
case BPF_ALU | BPF_LSH | BPF_X:
case BPF_ALU | BPF_RSH | BPF_X:
case BPF_ALU | BPF_ARSH | BPF_X:
case BPF_ALU | BPF_ADD | BPF_X:
case BPF_ALU | BPF_SUB | BPF_X:
case BPF_ALU | BPF_MUL | BPF_X:
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU | BPF_MOD | BPF_X:
emit_alu_r(ctx, lo(dst), lo(src), BPF_OP(code));
emit_zext_ver(ctx, dst);
break;
/* dst = imm (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_K:
emit_mov_se_i64(ctx, dst, imm);
break;
/* dst = src (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_X:
emit_mov_r(ctx, lo(dst), lo(src));
emit_mov_r(ctx, hi(dst), hi(src));
break;
/* dst = -dst (64-bit) */
case BPF_ALU64 | BPF_NEG:
emit_neg_i64(ctx, dst);
break;
/* dst = dst & imm (64-bit) */
case BPF_ALU64 | BPF_AND | BPF_K:
emit_alu_i64(ctx, dst, imm, BPF_OP(code));
break;
/* dst = dst | imm (64-bit) */
/* dst = dst ^ imm (64-bit) */
/* dst = dst + imm (64-bit) */
/* dst = dst - imm (64-bit) */
case BPF_ALU64 | BPF_OR | BPF_K:
case BPF_ALU64 | BPF_XOR | BPF_K:
case BPF_ALU64 | BPF_ADD | BPF_K:
case BPF_ALU64 | BPF_SUB | BPF_K:
if (imm)
emit_alu_i64(ctx, dst, imm, BPF_OP(code));
break;
/* dst = dst << imm (64-bit) */
/* dst = dst >> imm (64-bit) */
/* dst = dst >> imm (64-bit, arithmetic) */
case BPF_ALU64 | BPF_LSH | BPF_K:
case BPF_ALU64 | BPF_RSH | BPF_K:
case BPF_ALU64 | BPF_ARSH | BPF_K:
if (imm)
emit_shift_i64(ctx, dst, imm, BPF_OP(code));
break;
/* dst = dst * imm (64-bit) */
case BPF_ALU64 | BPF_MUL | BPF_K:
emit_mul_i64(ctx, dst, imm);
break;
/* dst = dst / imm (64-bit) */
/* dst = dst % imm (64-bit) */
case BPF_ALU64 | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
/*
* Sign-extend the immediate value into a temporary register,
* and then do the operation on this register.
*/
emit_mov_se_i64(ctx, tmp, imm);
emit_divmod_r64(ctx, dst, tmp, BPF_OP(code));
break;
/* dst = dst & src (64-bit) */
/* dst = dst | src (64-bit) */
/* dst = dst ^ src (64-bit) */
/* dst = dst + src (64-bit) */
/* dst = dst - src (64-bit) */
case BPF_ALU64 | BPF_AND | BPF_X:
case BPF_ALU64 | BPF_OR | BPF_X:
case BPF_ALU64 | BPF_XOR | BPF_X:
case BPF_ALU64 | BPF_ADD | BPF_X:
case BPF_ALU64 | BPF_SUB | BPF_X:
emit_alu_r64(ctx, dst, src, BPF_OP(code));
break;
/* dst = dst << src (64-bit) */
/* dst = dst >> src (64-bit) */
/* dst = dst >> src (64-bit, arithmetic) */
case BPF_ALU64 | BPF_LSH | BPF_X:
case BPF_ALU64 | BPF_RSH | BPF_X:
case BPF_ALU64 | BPF_ARSH | BPF_X:
emit_shift_r64(ctx, dst, lo(src), BPF_OP(code));
break;
/* dst = dst * src (64-bit) */
case BPF_ALU64 | BPF_MUL | BPF_X:
emit_mul_r64(ctx, dst, src);
break;
/* dst = dst / src (64-bit) */
/* dst = dst % src (64-bit) */
case BPF_ALU64 | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
emit_divmod_r64(ctx, dst, src, BPF_OP(code));
break;
/* dst = htole(dst) */
/* dst = htobe(dst) */
case BPF_ALU | BPF_END | BPF_FROM_LE:
case BPF_ALU | BPF_END | BPF_FROM_BE:
if (BPF_SRC(code) ==
#ifdef __BIG_ENDIAN
BPF_FROM_LE
#else
BPF_FROM_BE
#endif
)
emit_bswap_r64(ctx, dst, imm);
else
emit_trunc_r64(ctx, dst, imm);
break;
/* dst = imm64 */
case BPF_LD | BPF_IMM | BPF_DW:
emit_mov_i(ctx, lo(dst), imm);
emit_mov_i(ctx, hi(dst), insn[1].imm);
return 1;
/* LDX: dst = *(size *)(src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
case BPF_LDX | BPF_MEM | BPF_H:
case BPF_LDX | BPF_MEM | BPF_B:
case BPF_LDX | BPF_MEM | BPF_DW:
emit_ldx(ctx, dst, lo(src), off, BPF_SIZE(code));
break;
/* ST: *(size *)(dst + off) = imm */
case BPF_ST | BPF_MEM | BPF_W:
case BPF_ST | BPF_MEM | BPF_H:
case BPF_ST | BPF_MEM | BPF_B:
case BPF_ST | BPF_MEM | BPF_DW:
switch (BPF_SIZE(code)) {
case BPF_DW:
/* Sign-extend immediate value into temporary reg */
emit_mov_se_i64(ctx, tmp, imm);
break;
case BPF_W:
case BPF_H:
case BPF_B:
emit_mov_i(ctx, lo(tmp), imm);
break;
}
emit_stx(ctx, lo(dst), tmp, off, BPF_SIZE(code));
break;
/* STX: *(size *)(dst + off) = src */
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_B:
case BPF_STX | BPF_MEM | BPF_DW:
emit_stx(ctx, lo(dst), src, off, BPF_SIZE(code));
break;
/* Speculation barrier */
case BPF_ST | BPF_NOSPEC:
break;
/* Atomics */
case BPF_STX | BPF_ATOMIC | BPF_W:
switch (imm) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
case BPF_AND:
case BPF_AND | BPF_FETCH:
case BPF_OR:
case BPF_OR | BPF_FETCH:
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
case BPF_XCHG:
if (cpu_has_llsc)
emit_atomic_r(ctx, lo(dst), lo(src), off, imm);
else /* Non-ll/sc fallback */
emit_atomic_r32(ctx, lo(dst), lo(src),
off, imm);
if (imm & BPF_FETCH)
emit_zext_ver(ctx, src);
break;
case BPF_CMPXCHG:
if (cpu_has_llsc)
emit_cmpxchg_r(ctx, lo(dst), lo(src),
lo(res), off);
else /* Non-ll/sc fallback */
emit_cmpxchg_r32(ctx, lo(dst), lo(src), off);
/* Result zero-extension inserted by verifier */
break;
default:
goto notyet;
}
break;
/* Atomics (64-bit) */
case BPF_STX | BPF_ATOMIC | BPF_DW:
switch (imm) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
case BPF_AND:
case BPF_AND | BPF_FETCH:
case BPF_OR:
case BPF_OR | BPF_FETCH:
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
case BPF_XCHG:
emit_atomic_r64(ctx, lo(dst), src, off, imm);
break;
case BPF_CMPXCHG:
emit_cmpxchg_r64(ctx, lo(dst), src, off);
break;
default:
goto notyet;
}
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
/* PC += off if dst > src */
/* PC += off if dst >= src */
/* PC += off if dst < src */
/* PC += off if dst <= src */
/* PC += off if dst > src (signed) */
/* PC += off if dst >= src (signed) */
/* PC += off if dst < src (signed) */
/* PC += off if dst <= src (signed) */
case BPF_JMP32 | BPF_JEQ | BPF_X:
case BPF_JMP32 | BPF_JNE | BPF_X:
case BPF_JMP32 | BPF_JSET | BPF_X:
case BPF_JMP32 | BPF_JGT | BPF_X:
case BPF_JMP32 | BPF_JGE | BPF_X:
case BPF_JMP32 | BPF_JLT | BPF_X:
case BPF_JMP32 | BPF_JLE | BPF_X:
case BPF_JMP32 | BPF_JSGT | BPF_X:
case BPF_JMP32 | BPF_JSGE | BPF_X:
case BPF_JMP32 | BPF_JSLT | BPF_X:
case BPF_JMP32 | BPF_JSLE | BPF_X:
if (off == 0)
break;
setup_jmp_r(ctx, dst == src, BPF_OP(code), off, &jmp, &rel);
emit_jmp_r(ctx, lo(dst), lo(src), rel, jmp);
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == imm */
/* PC += off if dst != imm */
/* PC += off if dst & imm */
/* PC += off if dst > imm */
/* PC += off if dst >= imm */
/* PC += off if dst < imm */
/* PC += off if dst <= imm */
/* PC += off if dst > imm (signed) */
/* PC += off if dst >= imm (signed) */
/* PC += off if dst < imm (signed) */
/* PC += off if dst <= imm (signed) */
case BPF_JMP32 | BPF_JEQ | BPF_K:
case BPF_JMP32 | BPF_JNE | BPF_K:
case BPF_JMP32 | BPF_JSET | BPF_K:
case BPF_JMP32 | BPF_JGT | BPF_K:
case BPF_JMP32 | BPF_JGE | BPF_K:
case BPF_JMP32 | BPF_JLT | BPF_K:
case BPF_JMP32 | BPF_JLE | BPF_K:
case BPF_JMP32 | BPF_JSGT | BPF_K:
case BPF_JMP32 | BPF_JSGE | BPF_K:
case BPF_JMP32 | BPF_JSLT | BPF_K:
case BPF_JMP32 | BPF_JSLE | BPF_K:
if (off == 0)
break;
setup_jmp_i(ctx, imm, 32, BPF_OP(code), off, &jmp, &rel);
if (valid_jmp_i(jmp, imm)) {
emit_jmp_i(ctx, lo(dst), imm, rel, jmp);
} else {
/* Move large immediate to register */
emit_mov_i(ctx, MIPS_R_T6, imm);
emit_jmp_r(ctx, lo(dst), MIPS_R_T6, rel, jmp);
}
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
/* PC += off if dst > src */
/* PC += off if dst >= src */
/* PC += off if dst < src */
/* PC += off if dst <= src */
/* PC += off if dst > src (signed) */
/* PC += off if dst >= src (signed) */
/* PC += off if dst < src (signed) */
/* PC += off if dst <= src (signed) */
case BPF_JMP | BPF_JEQ | BPF_X:
case BPF_JMP | BPF_JNE | BPF_X:
case BPF_JMP | BPF_JSET | BPF_X:
case BPF_JMP | BPF_JGT | BPF_X:
case BPF_JMP | BPF_JGE | BPF_X:
case BPF_JMP | BPF_JLT | BPF_X:
case BPF_JMP | BPF_JLE | BPF_X:
case BPF_JMP | BPF_JSGT | BPF_X:
case BPF_JMP | BPF_JSGE | BPF_X:
case BPF_JMP | BPF_JSLT | BPF_X:
case BPF_JMP | BPF_JSLE | BPF_X:
if (off == 0)
break;
setup_jmp_r(ctx, dst == src, BPF_OP(code), off, &jmp, &rel);
emit_jmp_r64(ctx, dst, src, rel, jmp);
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == imm */
/* PC += off if dst != imm */
/* PC += off if dst & imm */
/* PC += off if dst > imm */
/* PC += off if dst >= imm */
/* PC += off if dst < imm */
/* PC += off if dst <= imm */
/* PC += off if dst > imm (signed) */
/* PC += off if dst >= imm (signed) */
/* PC += off if dst < imm (signed) */
/* PC += off if dst <= imm (signed) */
case BPF_JMP | BPF_JEQ | BPF_K:
case BPF_JMP | BPF_JNE | BPF_K:
case BPF_JMP | BPF_JSET | BPF_K:
case BPF_JMP | BPF_JGT | BPF_K:
case BPF_JMP | BPF_JGE | BPF_K:
case BPF_JMP | BPF_JLT | BPF_K:
case BPF_JMP | BPF_JLE | BPF_K:
case BPF_JMP | BPF_JSGT | BPF_K:
case BPF_JMP | BPF_JSGE | BPF_K:
case BPF_JMP | BPF_JSLT | BPF_K:
case BPF_JMP | BPF_JSLE | BPF_K:
if (off == 0)
break;
setup_jmp_i(ctx, imm, 64, BPF_OP(code), off, &jmp, &rel);
emit_jmp_i64(ctx, dst, imm, rel, jmp);
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off */
case BPF_JMP | BPF_JA:
if (off == 0)
break;
if (emit_ja(ctx, off) < 0)
goto toofar;
break;
/* Tail call */
case BPF_JMP | BPF_TAIL_CALL:
if (emit_tail_call(ctx) < 0)
goto invalid;
break;
/* Function call */
case BPF_JMP | BPF_CALL:
if (emit_call(ctx, insn) < 0)
goto invalid;
break;
/* Function return */
case BPF_JMP | BPF_EXIT:
/*
* Optimization: when last instruction is EXIT
* simply continue to epilogue.
*/
if (ctx->bpf_index == ctx->program->len - 1)
break;
if (emit_exit(ctx) < 0)
goto toofar;
break;
default:
invalid:
pr_err_once("unknown opcode %02x\n", code);
return -EINVAL;
notyet:
pr_info_once("*** NOT YET: opcode %02x ***\n", code);
return -EFAULT;
toofar:
pr_info_once("*** TOO FAR: jump at %u opcode %02x ***\n",
ctx->bpf_index, code);
return -E2BIG;
}
return 0;
}
arch/mips/net/bpf_jit_comp64.c 0 → 100644
View file @ f438ee21
// SPDX-License-Identifier: GPL-2.0-only
/*
* Just-In-Time compiler for eBPF bytecode on MIPS.
* Implementation of JIT functions for 64-bit CPUs.
*
* Copyright (c) 2021 Anyfi Networks AB.
* Author: Johan Almbladh <johan.almbladh@gmail.com>
*
* Based on code and ideas from
* Copyright (c) 2017 Cavium, Inc.
* Copyright (c) 2017 Shubham Bansal <illusionist.neo@gmail.com>
* Copyright (c) 2011 Mircea Gherzan <mgherzan@gmail.com>
*/
#include <linux/errno.h>
#include <linux/filter.h>
#include <linux/bpf.h>
#include <asm/cpu-features.h>
#include <asm/isa-rev.h>
#include <asm/uasm.h>
#include "bpf_jit_comp.h"
/* MIPS t0-t3 are not available in the n64 ABI */
#undef MIPS_R_T0
#undef MIPS_R_T1
#undef MIPS_R_T2
#undef MIPS_R_T3
/* Stack is 16-byte aligned in n64 ABI */
#define MIPS_STACK_ALIGNMENT 16
/* Extra 64-bit eBPF registers used by JIT */
#define JIT_REG_TC (MAX_BPF_JIT_REG + 0)
#define JIT_REG_ZX (MAX_BPF_JIT_REG + 1)
/* Number of prologue bytes to skip when doing a tail call */
#define JIT_TCALL_SKIP 4
/* Callee-saved CPU registers that the JIT must preserve */
#define JIT_CALLEE_REGS \
(BIT(MIPS_R_S0) | \
BIT(MIPS_R_S1) | \
BIT(MIPS_R_S2) | \
BIT(MIPS_R_S3) | \
BIT(MIPS_R_S4) | \
BIT(MIPS_R_S5) | \
BIT(MIPS_R_S6) | \
BIT(MIPS_R_S7) | \
BIT(MIPS_R_GP) | \
BIT(MIPS_R_FP) | \
BIT(MIPS_R_RA))
/* Caller-saved CPU registers available for JIT use */
#define JIT_CALLER_REGS \
(BIT(MIPS_R_A5) | \
BIT(MIPS_R_A6) | \
BIT(MIPS_R_A7))
/*
* Mapping of 64-bit eBPF registers to 64-bit native MIPS registers.
* MIPS registers t4 - t7 may be used by the JIT as temporary registers.
* MIPS registers t8 - t9 are reserved for single-register common functions.
*/
static const u8 bpf2mips64[] = {
/* Return value from in-kernel function, and exit value from eBPF */
[BPF_REG_0] = MIPS_R_V0,
/* Arguments from eBPF program to in-kernel function */
[BPF_REG_1] = MIPS_R_A0,
[BPF_REG_2] = MIPS_R_A1,
[BPF_REG_3] = MIPS_R_A2,
[BPF_REG_4] = MIPS_R_A3,
[BPF_REG_5] = MIPS_R_A4,
/* Callee-saved registers that in-kernel function will preserve */
[BPF_REG_6] = MIPS_R_S0,
[BPF_REG_7] = MIPS_R_S1,
[BPF_REG_8] = MIPS_R_S2,
[BPF_REG_9] = MIPS_R_S3,
/* Read-only frame pointer to access the eBPF stack */
[BPF_REG_FP] = MIPS_R_FP,
/* Temporary register for blinding constants */
[BPF_REG_AX] = MIPS_R_AT,
/* Tail call count register, caller-saved */
[JIT_REG_TC] = MIPS_R_A5,
/* Constant for register zero-extension */
[JIT_REG_ZX] = MIPS_R_V1,
};
/*
* MIPS 32-bit operations on 64-bit registers generate a sign-extended
* result. However, the eBPF ISA mandates zero-extension, so we rely on the
* verifier to add that for us (emit_zext_ver). In addition, ALU arithmetic
* operations, right shift and byte swap require properly sign-extended
* operands or the result is unpredictable. We emit explicit sign-extensions
* in those cases.
*/
/* Sign extension */
static void emit_sext(struct jit_context *ctx, u8 dst, u8 src)
{
emit(ctx, sll, dst, src, 0);
clobber_reg(ctx, dst);
}
/* Zero extension */
static void emit_zext(struct jit_context *ctx, u8 dst)
{
if (cpu_has_mips64r2 || cpu_has_mips64r6) {
emit(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
} else {
emit(ctx, and, dst, dst, bpf2mips64[JIT_REG_ZX]);
access_reg(ctx, JIT_REG_ZX); /* We need the ZX register */
}
clobber_reg(ctx, dst);
}
/* Zero extension, if verifier does not do it for us */
static void emit_zext_ver(struct jit_context *ctx, u8 dst)
{
if (!ctx->program->aux->verifier_zext)
emit_zext(ctx, dst);
}
/* dst = imm (64-bit) */
static void emit_mov_i64(struct jit_context *ctx, u8 dst, u64 imm64)
{
if (imm64 >= 0xffffffffffff8000ULL || imm64 < 0x8000ULL) {
emit(ctx, daddiu, dst, MIPS_R_ZERO, (s16)imm64);
} else if (imm64 >= 0xffffffff80000000ULL ||
(imm64 < 0x80000000 && imm64 > 0xffff)) {
emit(ctx, lui, dst, (s16)(imm64 >> 16));
emit(ctx, ori, dst, dst, (u16)imm64 & 0xffff);
} else {
u8 acc = MIPS_R_ZERO;
int k;
for (k = 0; k < 4; k++) {
u16 half = imm64 >> (48 - 16 * k);
if (acc == dst)
emit(ctx, dsll, dst, dst, 16);
if (half) {
emit(ctx, ori, dst, acc, half);
acc = dst;
}
}
}
clobber_reg(ctx, dst);
}
/* ALU immediate operation (64-bit) */
static void emit_alu_i64(struct jit_context *ctx, u8 dst, s32 imm, u8 op)
{
switch (BPF_OP(op)) {
/* dst = dst | imm */
case BPF_OR:
emit(ctx, ori, dst, dst, (u16)imm);
break;
/* dst = dst ^ imm */
case BPF_XOR:
emit(ctx, xori, dst, dst, (u16)imm);
break;
/* dst = -dst */
case BPF_NEG:
emit(ctx, dsubu, dst, MIPS_R_ZERO, dst);
break;
/* dst = dst << imm */
case BPF_LSH:
emit(ctx, dsll_safe, dst, dst, imm);
break;
/* dst = dst >> imm */
case BPF_RSH:
emit(ctx, dsrl_safe, dst, dst, imm);
break;
/* dst = dst >> imm (arithmetic) */
case BPF_ARSH:
emit(ctx, dsra_safe, dst, dst, imm);
break;
/* dst = dst + imm */
case BPF_ADD:
emit(ctx, daddiu, dst, dst, imm);
break;
/* dst = dst - imm */
case BPF_SUB:
emit(ctx, daddiu, dst, dst, -imm);
break;
default:
/* Width-generic operations */
emit_alu_i(ctx, dst, imm, op);
}
clobber_reg(ctx, dst);
}
/* ALU register operation (64-bit) */
static void emit_alu_r64(struct jit_context *ctx, u8 dst, u8 src, u8 op)
{
switch (BPF_OP(op)) {
/* dst = dst << src */
case BPF_LSH:
emit(ctx, dsllv, dst, dst, src);
break;
/* dst = dst >> src */
case BPF_RSH:
emit(ctx, dsrlv, dst, dst, src);
break;
/* dst = dst >> src (arithmetic) */
case BPF_ARSH:
emit(ctx, dsrav, dst, dst, src);
break;
/* dst = dst + src */
case BPF_ADD:
emit(ctx, daddu, dst, dst, src);
break;
/* dst = dst - src */
case BPF_SUB:
emit(ctx, dsubu, dst, dst, src);
break;
/* dst = dst * src */
case BPF_MUL:
if (cpu_has_mips64r6) {
emit(ctx, dmulu, dst, dst, src);
} else {
emit(ctx, dmultu, dst, src);
emit(ctx, mflo, dst);
}
break;
/* dst = dst / src */
case BPF_DIV:
if (cpu_has_mips64r6) {
emit(ctx, ddivu_r6, dst, dst, src);
} else {
emit(ctx, ddivu, dst, src);
emit(ctx, mflo, dst);
}
break;
/* dst = dst % src */
case BPF_MOD:
if (cpu_has_mips64r6) {
emit(ctx, dmodu, dst, dst, src);
} else {
emit(ctx, ddivu, dst, src);
emit(ctx, mfhi, dst);
}
break;
default:
/* Width-generic operations */
emit_alu_r(ctx, dst, src, op);
}
clobber_reg(ctx, dst);
}
/* Swap sub words in a register double word */
static void emit_swap_r64(struct jit_context *ctx, u8 dst, u8 mask, u32 bits)
{
u8 tmp = MIPS_R_T9;
emit(ctx, and, tmp, dst, mask); /* tmp = dst & mask */
emit(ctx, dsll, tmp, tmp, bits); /* tmp = tmp << bits */
emit(ctx, dsrl, dst, dst, bits); /* dst = dst >> bits */
emit(ctx, and, dst, dst, mask); /* dst = dst & mask */
emit(ctx, or, dst, dst, tmp); /* dst = dst | tmp */
}
/* Swap bytes and truncate a register double word, word or half word */
static void emit_bswap_r64(struct jit_context *ctx, u8 dst, u32 width)
{
switch (width) {
/* Swap bytes in a double word */
case 64:
if (cpu_has_mips64r2 || cpu_has_mips64r6) {
emit(ctx, dsbh, dst, dst);
emit(ctx, dshd, dst, dst);
} else {
u8 t1 = MIPS_R_T6;
u8 t2 = MIPS_R_T7;
emit(ctx, dsll32, t2, dst, 0); /* t2 = dst << 32 */
emit(ctx, dsrl32, dst, dst, 0); /* dst = dst >> 32 */
emit(ctx, or, dst, dst, t2); /* dst = dst | t2 */
emit(ctx, ori, t2, MIPS_R_ZERO, 0xffff);
emit(ctx, dsll32, t1, t2, 0); /* t1 = t2 << 32 */
emit(ctx, or, t1, t1, t2); /* t1 = t1 | t2 */
emit_swap_r64(ctx, dst, t1, 16);/* dst = swap16(dst) */
emit(ctx, lui, t2, 0xff); /* t2 = 0x00ff0000 */
emit(ctx, ori, t2, t2, 0xff); /* t2 = t2 | 0x00ff */
emit(ctx, dsll32, t1, t2, 0); /* t1 = t2 << 32 */
emit(ctx, or, t1, t1, t2); /* t1 = t1 | t2 */
emit_swap_r64(ctx, dst, t1, 8); /* dst = swap8(dst) */
}
break;
/* Swap bytes in a half word */
/* Swap bytes in a word */
case 32:
case 16:
emit_sext(ctx, dst, dst);
emit_bswap_r(ctx, dst, width);
if (cpu_has_mips64r2 || cpu_has_mips64r6)
emit_zext(ctx, dst);
break;
}
clobber_reg(ctx, dst);
}
/* Truncate a register double word, word or half word */
static void emit_trunc_r64(struct jit_context *ctx, u8 dst, u32 width)
{
switch (width) {
case 64:
break;
/* Zero-extend a word */
case 32:
emit_zext(ctx, dst);
break;
/* Zero-extend a half word */
case 16:
emit(ctx, andi, dst, dst, 0xffff);
break;
}
clobber_reg(ctx, dst);
}
/* Load operation: dst = *(size*)(src + off) */
static void emit_ldx(struct jit_context *ctx, u8 dst, u8 src, s16 off, u8 size)
{
switch (size) {
/* Load a byte */
case BPF_B:
emit(ctx, lbu, dst, off, src);
break;
/* Load a half word */
case BPF_H:
emit(ctx, lhu, dst, off, src);
break;
/* Load a word */
case BPF_W:
emit(ctx, lwu, dst, off, src);
break;
/* Load a double word */
case BPF_DW:
emit(ctx, ld, dst, off, src);
break;
}
clobber_reg(ctx, dst);
}
/* Store operation: *(size *)(dst + off) = src */
static void emit_stx(struct jit_context *ctx, u8 dst, u8 src, s16 off, u8 size)
{
switch (size) {
/* Store a byte */
case BPF_B:
emit(ctx, sb, src, off, dst);
break;
/* Store a half word */
case BPF_H:
emit(ctx, sh, src, off, dst);
break;
/* Store a word */
case BPF_W:
emit(ctx, sw, src, off, dst);
break;
/* Store a double word */
case BPF_DW:
emit(ctx, sd, src, off, dst);
break;
}
}
/* Atomic read-modify-write */
static void emit_atomic_r64(struct jit_context *ctx,
u8 dst, u8 src, s16 off, u8 code)
{
u8 t1 = MIPS_R_T6;
u8 t2 = MIPS_R_T7;
LLSC_sync(ctx);
emit(ctx, lld, t1, off, dst);
switch (code) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
emit(ctx, daddu, t2, t1, src);
break;
case BPF_AND:
case BPF_AND | BPF_FETCH:
emit(ctx, and, t2, t1, src);
break;
case BPF_OR:
case BPF_OR | BPF_FETCH:
emit(ctx, or, t2, t1, src);
break;
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
emit(ctx, xor, t2, t1, src);
break;
case BPF_XCHG:
emit(ctx, move, t2, src);
break;
}
emit(ctx, scd, t2, off, dst);
emit(ctx, LLSC_beqz, t2, -16 - LLSC_offset);
emit(ctx, nop); /* Delay slot */
if (code & BPF_FETCH) {
emit(ctx, move, src, t1);
clobber_reg(ctx, src);
}
}
/* Atomic compare-and-exchange */
static void emit_cmpxchg_r64(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
u8 r0 = bpf2mips64[BPF_REG_0];
u8 t1 = MIPS_R_T6;
u8 t2 = MIPS_R_T7;
LLSC_sync(ctx);
emit(ctx, lld, t1, off, dst);
emit(ctx, bne, t1, r0, 12);
emit(ctx, move, t2, src); /* Delay slot */
emit(ctx, scd, t2, off, dst);
emit(ctx, LLSC_beqz, t2, -20 - LLSC_offset);
emit(ctx, move, r0, t1); /* Delay slot */
clobber_reg(ctx, r0);
}
/* Function call */
static int emit_call(struct jit_context *ctx, const struct bpf_insn *insn)
{
u8 zx = bpf2mips64[JIT_REG_ZX];
u8 tmp = MIPS_R_T6;
bool fixed;
u64 addr;
/* Decode the call address */
if (bpf_jit_get_func_addr(ctx->program, insn, false,
&addr, &fixed) < 0)
return -1;
if (!fixed)
return -1;
/* Push caller-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLER_REGS, 0, 0);
/* Emit function call */
emit_mov_i64(ctx, tmp, addr & JALR_MASK);
emit(ctx, jalr, MIPS_R_RA, tmp);
emit(ctx, nop); /* Delay slot */
/* Restore caller-saved registers */
pop_regs(ctx, ctx->clobbered & JIT_CALLER_REGS, 0, 0);
/* Re-initialize the JIT zero-extension register if accessed */
if (ctx->accessed & BIT(JIT_REG_ZX)) {
emit(ctx, daddiu, zx, MIPS_R_ZERO, -1);
emit(ctx, dsrl32, zx, zx, 0);
}
clobber_reg(ctx, MIPS_R_RA);
clobber_reg(ctx, MIPS_R_V0);
clobber_reg(ctx, MIPS_R_V1);
return 0;
}
/* Function tail call */
static int emit_tail_call(struct jit_context *ctx)
{
u8 ary = bpf2mips64[BPF_REG_2];
u8 ind = bpf2mips64[BPF_REG_3];
u8 tcc = bpf2mips64[JIT_REG_TC];
u8 tmp = MIPS_R_T6;
int off;
/*
* Tail call:
* eBPF R1 - function argument (context ptr), passed in a0-a1
* eBPF R2 - ptr to object with array of function entry points
* eBPF R3 - array index of function to be called
*/
/* if (ind >= ary->map.max_entries) goto out */
off = offsetof(struct bpf_array, map.max_entries);
if (off > 0x7fff)
return -1;
emit(ctx, lwu, tmp, off, ary); /* tmp = ary->map.max_entrs*/
emit(ctx, sltu, tmp, ind, tmp); /* tmp = ind < t1 */
emit(ctx, beqz, tmp, get_offset(ctx, 1)); /* PC += off(1) if tmp == 0*/
/* if (--TCC < 0) goto out */
emit(ctx, daddiu, tcc, tcc, -1); /* tcc-- (delay slot) */
emit(ctx, bltz, tcc, get_offset(ctx, 1)); /* PC += off(1) if tcc < 0 */
/* (next insn delay slot) */
/* prog = ary->ptrs[ind] */
off = offsetof(struct bpf_array, ptrs);
if (off > 0x7fff)
return -1;
emit(ctx, dsll, tmp, ind, 3); /* tmp = ind << 3 */
emit(ctx, daddu, tmp, tmp, ary); /* tmp += ary */
emit(ctx, ld, tmp, off, tmp); /* tmp = *(tmp + off) */
/* if (prog == 0) goto out */
emit(ctx, beqz, tmp, get_offset(ctx, 1)); /* PC += off(1) if tmp == 0*/
emit(ctx, nop); /* Delay slot */
/* func = prog->bpf_func + 8 (prologue skip offset) */
off = offsetof(struct bpf_prog, bpf_func);
if (off > 0x7fff)
return -1;
emit(ctx, ld, tmp, off, tmp); /* tmp = *(tmp + off) */
emit(ctx, daddiu, tmp, tmp, JIT_TCALL_SKIP); /* tmp += skip (4) */
/* goto func */
build_epilogue(ctx, tmp);
access_reg(ctx, JIT_REG_TC);
return 0;
}
/*
* Stack frame layout for a JITed program (stack grows down).
*
* Higher address : Previous stack frame :
* +===========================+ <--- MIPS sp before call
* | Callee-saved registers, |
* | including RA and FP |
* +---------------------------+ <--- eBPF FP (MIPS fp)
* | Local eBPF variables |
* | allocated by program |
* +---------------------------+
* | Reserved for caller-saved |
* | registers |
* Lower address +===========================+ <--- MIPS sp
*/
/* Build program prologue to set up the stack and registers */
void build_prologue(struct jit_context *ctx)
{
u8 fp = bpf2mips64[BPF_REG_FP];
u8 tc = bpf2mips64[JIT_REG_TC];
u8 zx = bpf2mips64[JIT_REG_ZX];
int stack, saved, locals, reserved;
/*
* The first instruction initializes the tail call count register.
* On a tail call, the calling function jumps into the prologue
* after this instruction.
*/
emit(ctx, addiu, tc, MIPS_R_ZERO, min(MAX_TAIL_CALL_CNT + 1, 0xffff));
/* === Entry-point for tail calls === */
/*
* If the eBPF frame pointer and tail call count registers were
* accessed they must be preserved. Mark them as clobbered here
* to save and restore them on the stack as needed.
*/
if (ctx->accessed & BIT(BPF_REG_FP))
clobber_reg(ctx, fp);
if (ctx->accessed & BIT(JIT_REG_TC))
clobber_reg(ctx, tc);
if (ctx->accessed & BIT(JIT_REG_ZX))
clobber_reg(ctx, zx);
/* Compute the stack space needed for callee-saved registers */
saved = hweight32(ctx->clobbered & JIT_CALLEE_REGS) * sizeof(u64);
saved = ALIGN(saved, MIPS_STACK_ALIGNMENT);
/* Stack space used by eBPF program local data */
locals = ALIGN(ctx->program->aux->stack_depth, MIPS_STACK_ALIGNMENT);
/*
* If we are emitting function calls, reserve extra stack space for
* caller-saved registers needed by the JIT. The required space is
* computed automatically during resource usage discovery (pass 1).
*/
reserved = ctx->stack_used;
/* Allocate the stack frame */
stack = ALIGN(saved + locals + reserved, MIPS_STACK_ALIGNMENT);
if (stack)
emit(ctx, daddiu, MIPS_R_SP, MIPS_R_SP, -stack);
/* Store callee-saved registers on stack */
push_regs(ctx, ctx->clobbered & JIT_CALLEE_REGS, 0, stack - saved);
/* Initialize the eBPF frame pointer if accessed */
if (ctx->accessed & BIT(BPF_REG_FP))
emit(ctx, daddiu, fp, MIPS_R_SP, stack - saved);
/* Initialize the ePF JIT zero-extension register if accessed */
if (ctx->accessed & BIT(JIT_REG_ZX)) {
emit(ctx, daddiu, zx, MIPS_R_ZERO, -1);
emit(ctx, dsrl32, zx, zx, 0);
}
ctx->saved_size = saved;
ctx->stack_size = stack;
}
/* Build the program epilogue to restore the stack and registers */
void build_epilogue(struct jit_context *ctx, int dest_reg)
{
/* Restore callee-saved registers from stack */
pop_regs(ctx, ctx->clobbered & JIT_CALLEE_REGS, 0,
ctx->stack_size - ctx->saved_size);
/* Release the stack frame */
if (ctx->stack_size)
emit(ctx, daddiu, MIPS_R_SP, MIPS_R_SP, ctx->stack_size);
/* Jump to return address and sign-extend the 32-bit return value */
emit(ctx, jr, dest_reg);
emit(ctx, sll, MIPS_R_V0, MIPS_R_V0, 0); /* Delay slot */
}
/* Build one eBPF instruction */
int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
{
u8 dst = bpf2mips64[insn->dst_reg];
u8 src = bpf2mips64[insn->src_reg];
u8 res = bpf2mips64[BPF_REG_0];
u8 code = insn->code;
s16 off = insn->off;
s32 imm = insn->imm;
s32 val, rel;
u8 alu, jmp;
switch (code) {
/* ALU operations */
/* dst = imm */
case BPF_ALU | BPF_MOV | BPF_K:
emit_mov_i(ctx, dst, imm);
emit_zext_ver(ctx, dst);
break;
/* dst = src */
case BPF_ALU | BPF_MOV | BPF_X:
if (imm == 1) {
/* Special mov32 for zext */
emit_zext(ctx, dst);
} else {
emit_mov_r(ctx, dst, src);
emit_zext_ver(ctx, dst);
}
break;
/* dst = -dst */
case BPF_ALU | BPF_NEG:
emit_sext(ctx, dst, dst);
emit_alu_i(ctx, dst, 0, BPF_NEG);
emit_zext_ver(ctx, dst);
break;
/* dst = dst & imm */
/* dst = dst | imm */
/* dst = dst ^ imm */
/* dst = dst << imm */
case BPF_ALU | BPF_OR | BPF_K:
case BPF_ALU | BPF_AND | BPF_K:
case BPF_ALU | BPF_XOR | BPF_K:
case BPF_ALU | BPF_LSH | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i(ctx, dst, val, alu);
}
emit_zext_ver(ctx, dst);
break;
/* dst = dst >> imm */
/* dst = dst >> imm (arithmetic) */
/* dst = dst + imm */
/* dst = dst - imm */
/* dst = dst * imm */
/* dst = dst / imm */
/* dst = dst % imm */
case BPF_ALU | BPF_RSH | BPF_K:
case BPF_ALU | BPF_ARSH | BPF_K:
case BPF_ALU | BPF_ADD | BPF_K:
case BPF_ALU | BPF_SUB | BPF_K:
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
emit_sext(ctx, dst, dst);
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_sext(ctx, dst, dst);
emit_alu_i(ctx, dst, val, alu);
}
emit_zext_ver(ctx, dst);
break;
/* dst = dst & src */
/* dst = dst | src */
/* dst = dst ^ src */
/* dst = dst << src */
case BPF_ALU | BPF_AND | BPF_X:
case BPF_ALU | BPF_OR | BPF_X:
case BPF_ALU | BPF_XOR | BPF_X:
case BPF_ALU | BPF_LSH | BPF_X:
emit_alu_r(ctx, dst, src, BPF_OP(code));
emit_zext_ver(ctx, dst);
break;
/* dst = dst >> src */
/* dst = dst >> src (arithmetic) */
/* dst = dst + src */
/* dst = dst - src */
/* dst = dst * src */
/* dst = dst / src */
/* dst = dst % src */
case BPF_ALU | BPF_RSH | BPF_X:
case BPF_ALU | BPF_ARSH | BPF_X:
case BPF_ALU | BPF_ADD | BPF_X:
case BPF_ALU | BPF_SUB | BPF_X:
case BPF_ALU | BPF_MUL | BPF_X:
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU | BPF_MOD | BPF_X:
emit_sext(ctx, dst, dst);
emit_sext(ctx, MIPS_R_T4, src);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
emit_zext_ver(ctx, dst);
break;
/* dst = imm (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_K:
emit_mov_i(ctx, dst, imm);
break;
/* dst = src (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_X:
emit_mov_r(ctx, dst, src);
break;
/* dst = -dst (64-bit) */
case BPF_ALU64 | BPF_NEG:
emit_alu_i64(ctx, dst, 0, BPF_NEG);
break;
/* dst = dst & imm (64-bit) */
/* dst = dst | imm (64-bit) */
/* dst = dst ^ imm (64-bit) */
/* dst = dst << imm (64-bit) */
/* dst = dst >> imm (64-bit) */
/* dst = dst >> imm ((64-bit, arithmetic) */
/* dst = dst + imm (64-bit) */
/* dst = dst - imm (64-bit) */
/* dst = dst * imm (64-bit) */
/* dst = dst / imm (64-bit) */
/* dst = dst % imm (64-bit) */
case BPF_ALU64 | BPF_AND | BPF_K:
case BPF_ALU64 | BPF_OR | BPF_K:
case BPF_ALU64 | BPF_XOR | BPF_K:
case BPF_ALU64 | BPF_LSH | BPF_K:
case BPF_ALU64 | BPF_RSH | BPF_K:
case BPF_ALU64 | BPF_ARSH | BPF_K:
case BPF_ALU64 | BPF_ADD | BPF_K:
case BPF_ALU64 | BPF_SUB | BPF_K:
case BPF_ALU64 | BPF_MUL | BPF_K:
case BPF_ALU64 | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r64(ctx, dst, MIPS_R_T4, BPF_OP(code));
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i64(ctx, dst, val, alu);
}
break;
/* dst = dst & src (64-bit) */
/* dst = dst | src (64-bit) */
/* dst = dst ^ src (64-bit) */
/* dst = dst << src (64-bit) */
/* dst = dst >> src (64-bit) */
/* dst = dst >> src (64-bit, arithmetic) */
/* dst = dst + src (64-bit) */
/* dst = dst - src (64-bit) */
/* dst = dst * src (64-bit) */
/* dst = dst / src (64-bit) */
/* dst = dst % src (64-bit) */
case BPF_ALU64 | BPF_AND | BPF_X:
case BPF_ALU64 | BPF_OR | BPF_X:
case BPF_ALU64 | BPF_XOR | BPF_X:
case BPF_ALU64 | BPF_LSH | BPF_X:
case BPF_ALU64 | BPF_RSH | BPF_X:
case BPF_ALU64 | BPF_ARSH | BPF_X:
case BPF_ALU64 | BPF_ADD | BPF_X:
case BPF_ALU64 | BPF_SUB | BPF_X:
case BPF_ALU64 | BPF_MUL | BPF_X:
case BPF_ALU64 | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
emit_alu_r64(ctx, dst, src, BPF_OP(code));
break;
/* dst = htole(dst) */
/* dst = htobe(dst) */
case BPF_ALU | BPF_END | BPF_FROM_LE:
case BPF_ALU | BPF_END | BPF_FROM_BE:
if (BPF_SRC(code) ==
#ifdef __BIG_ENDIAN
BPF_FROM_LE
#else
BPF_FROM_BE
#endif
)
emit_bswap_r64(ctx, dst, imm);
else
emit_trunc_r64(ctx, dst, imm);
break;
/* dst = imm64 */
case BPF_LD | BPF_IMM | BPF_DW:
emit_mov_i64(ctx, dst, (u32)imm | ((u64)insn[1].imm << 32));
return 1;
/* LDX: dst = *(size *)(src + off) */
case BPF_LDX | BPF_MEM | BPF_W:
case BPF_LDX | BPF_MEM | BPF_H:
case BPF_LDX | BPF_MEM | BPF_B:
case BPF_LDX | BPF_MEM | BPF_DW:
emit_ldx(ctx, dst, src, off, BPF_SIZE(code));
break;
/* ST: *(size *)(dst + off) = imm */
case BPF_ST | BPF_MEM | BPF_W:
case BPF_ST | BPF_MEM | BPF_H:
case BPF_ST | BPF_MEM | BPF_B:
case BPF_ST | BPF_MEM | BPF_DW:
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_stx(ctx, dst, MIPS_R_T4, off, BPF_SIZE(code));
break;
/* STX: *(size *)(dst + off) = src */
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_B:
case BPF_STX | BPF_MEM | BPF_DW:
emit_stx(ctx, dst, src, off, BPF_SIZE(code));
break;
/* Speculation barrier */
case BPF_ST | BPF_NOSPEC:
break;
/* Atomics */
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_ATOMIC | BPF_DW:
switch (imm) {
case BPF_ADD:
case BPF_ADD | BPF_FETCH:
case BPF_AND:
case BPF_AND | BPF_FETCH:
case BPF_OR:
case BPF_OR | BPF_FETCH:
case BPF_XOR:
case BPF_XOR | BPF_FETCH:
case BPF_XCHG:
if (BPF_SIZE(code) == BPF_DW) {
emit_atomic_r64(ctx, dst, src, off, imm);
} else if (imm & BPF_FETCH) {
u8 tmp = dst;
if (src == dst) { /* Don't overwrite dst */
emit_mov_r(ctx, MIPS_R_T4, dst);
tmp = MIPS_R_T4;
}
emit_sext(ctx, src, src);
emit_atomic_r(ctx, tmp, src, off, imm);
emit_zext_ver(ctx, src);
} else { /* 32-bit, no fetch */
emit_sext(ctx, MIPS_R_T4, src);
emit_atomic_r(ctx, dst, MIPS_R_T4, off, imm);
}
break;
case BPF_CMPXCHG:
if (BPF_SIZE(code) == BPF_DW) {
emit_cmpxchg_r64(ctx, dst, src, off);
} else {
u8 tmp = res;
if (res == dst) /* Don't overwrite dst */
tmp = MIPS_R_T4;
emit_sext(ctx, tmp, res);
emit_sext(ctx, MIPS_R_T5, src);
emit_cmpxchg_r(ctx, dst, MIPS_R_T5, tmp, off);
if (res == dst) /* Restore result */
emit_mov_r(ctx, res, MIPS_R_T4);
/* Result zext inserted by verifier */
}
break;
default:
goto notyet;
}
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
/* PC += off if dst > src */
/* PC += off if dst >= src */
/* PC += off if dst < src */
/* PC += off if dst <= src */
/* PC += off if dst > src (signed) */
/* PC += off if dst >= src (signed) */
/* PC += off if dst < src (signed) */
/* PC += off if dst <= src (signed) */
case BPF_JMP32 | BPF_JEQ | BPF_X:
case BPF_JMP32 | BPF_JNE | BPF_X:
case BPF_JMP32 | BPF_JSET | BPF_X:
case BPF_JMP32 | BPF_JGT | BPF_X:
case BPF_JMP32 | BPF_JGE | BPF_X:
case BPF_JMP32 | BPF_JLT | BPF_X:
case BPF_JMP32 | BPF_JLE | BPF_X:
case BPF_JMP32 | BPF_JSGT | BPF_X:
case BPF_JMP32 | BPF_JSGE | BPF_X:
case BPF_JMP32 | BPF_JSLT | BPF_X:
case BPF_JMP32 | BPF_JSLE | BPF_X:
if (off == 0)
break;
setup_jmp_r(ctx, dst == src, BPF_OP(code), off, &jmp, &rel);
emit_sext(ctx, MIPS_R_T4, dst); /* Sign-extended dst */
emit_sext(ctx, MIPS_R_T5, src); /* Sign-extended src */
emit_jmp_r(ctx, MIPS_R_T4, MIPS_R_T5, rel, jmp);
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == imm */
/* PC += off if dst != imm */
/* PC += off if dst & imm */
/* PC += off if dst > imm */
/* PC += off if dst >= imm */
/* PC += off if dst < imm */
/* PC += off if dst <= imm */
/* PC += off if dst > imm (signed) */
/* PC += off if dst >= imm (signed) */
/* PC += off if dst < imm (signed) */
/* PC += off if dst <= imm (signed) */
case BPF_JMP32 | BPF_JEQ | BPF_K:
case BPF_JMP32 | BPF_JNE | BPF_K:
case BPF_JMP32 | BPF_JSET | BPF_K:
case BPF_JMP32 | BPF_JGT | BPF_K:
case BPF_JMP32 | BPF_JGE | BPF_K:
case BPF_JMP32 | BPF_JLT | BPF_K:
case BPF_JMP32 | BPF_JLE | BPF_K:
case BPF_JMP32 | BPF_JSGT | BPF_K:
case BPF_JMP32 | BPF_JSGE | BPF_K:
case BPF_JMP32 | BPF_JSLT | BPF_K:
case BPF_JMP32 | BPF_JSLE | BPF_K:
if (off == 0)
break;
setup_jmp_i(ctx, imm, 32, BPF_OP(code), off, &jmp, &rel);
emit_sext(ctx, MIPS_R_T4, dst); /* Sign-extended dst */
if (valid_jmp_i(jmp, imm)) {
emit_jmp_i(ctx, MIPS_R_T4, imm, rel, jmp);
} else {
/* Move large immediate to register, sign-extended */
emit_mov_i(ctx, MIPS_R_T5, imm);
emit_jmp_r(ctx, MIPS_R_T4, MIPS_R_T5, rel, jmp);
}
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
/* PC += off if dst > src */
/* PC += off if dst >= src */
/* PC += off if dst < src */
/* PC += off if dst <= src */
/* PC += off if dst > src (signed) */
/* PC += off if dst >= src (signed) */
/* PC += off if dst < src (signed) */
/* PC += off if dst <= src (signed) */
case BPF_JMP | BPF_JEQ | BPF_X:
case BPF_JMP | BPF_JNE | BPF_X:
case BPF_JMP | BPF_JSET | BPF_X:
case BPF_JMP | BPF_JGT | BPF_X:
case BPF_JMP | BPF_JGE | BPF_X:
case BPF_JMP | BPF_JLT | BPF_X:
case BPF_JMP | BPF_JLE | BPF_X:
case BPF_JMP | BPF_JSGT | BPF_X:
case BPF_JMP | BPF_JSGE | BPF_X:
case BPF_JMP | BPF_JSLT | BPF_X:
case BPF_JMP | BPF_JSLE | BPF_X:
if (off == 0)
break;
setup_jmp_r(ctx, dst == src, BPF_OP(code), off, &jmp, &rel);
emit_jmp_r(ctx, dst, src, rel, jmp);
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off if dst == imm */
/* PC += off if dst != imm */
/* PC += off if dst & imm */
/* PC += off if dst > imm */
/* PC += off if dst >= imm */
/* PC += off if dst < imm */
/* PC += off if dst <= imm */
/* PC += off if dst > imm (signed) */
/* PC += off if dst >= imm (signed) */
/* PC += off if dst < imm (signed) */
/* PC += off if dst <= imm (signed) */
case BPF_JMP | BPF_JEQ | BPF_K:
case BPF_JMP | BPF_JNE | BPF_K:
case BPF_JMP | BPF_JSET | BPF_K:
case BPF_JMP | BPF_JGT | BPF_K:
case BPF_JMP | BPF_JGE | BPF_K:
case BPF_JMP | BPF_JLT | BPF_K:
case BPF_JMP | BPF_JLE | BPF_K:
case BPF_JMP | BPF_JSGT | BPF_K:
case BPF_JMP | BPF_JSGE | BPF_K:
case BPF_JMP | BPF_JSLT | BPF_K:
case BPF_JMP | BPF_JSLE | BPF_K:
if (off == 0)
break;
setup_jmp_i(ctx, imm, 64, BPF_OP(code), off, &jmp, &rel);
if (valid_jmp_i(jmp, imm)) {
emit_jmp_i(ctx, dst, imm, rel, jmp);
} else {
/* Move large immediate to register */
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_jmp_r(ctx, dst, MIPS_R_T4, rel, jmp);
}
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += off */
case BPF_JMP | BPF_JA:
if (off == 0)
break;
if (emit_ja(ctx, off) < 0)
goto toofar;
break;
/* Tail call */
case BPF_JMP | BPF_TAIL_CALL:
if (emit_tail_call(ctx) < 0)
goto invalid;
break;
/* Function call */
case BPF_JMP | BPF_CALL:
if (emit_call(ctx, insn) < 0)
goto invalid;
break;
/* Function return */
case BPF_JMP | BPF_EXIT:
/*
* Optimization: when last instruction is EXIT
* simply continue to epilogue.
*/
if (ctx->bpf_index == ctx->program->len - 1)
break;
if (emit_exit(ctx) < 0)
goto toofar;
break;
default:
invalid:
pr_err_once("unknown opcode %02x\n", code);
return -EINVAL;
notyet:
pr_info_once("*** NOT YET: opcode %02x ***\n", code);
return -EFAULT;
toofar:
pr_info_once("*** TOO FAR: jump at %u opcode %02x ***\n",
ctx->bpf_index, code);
return -E2BIG;
}
return 0;
}
arch/mips/net/ebpf_jit.c deleted 100644 → 0
View file @ 9d057872
// SPDX-License-Identifier: GPL-2.0-only
/*
* Just-In-Time compiler for eBPF filters on MIPS
*
* Copyright (c) 2017 Cavium, Inc.
*
* Based on code from:
*
* Copyright (c) 2014 Imagination Technologies Ltd.
* Author: Markos Chandras <markos.chandras@imgtec.com>
*/
#include <linux/bitops.h>
#include <linux/errno.h>
#include <linux/filter.h>
#include <linux/bpf.h>
#include <linux/slab.h>
#include <asm/bitops.h>
#include <asm/byteorder.h>
#include <asm/cacheflush.h>
#include <asm/cpu-features.h>
#include <asm/isa-rev.h>
#include <asm/uasm.h>
/* Registers used by JIT */
#define MIPS_R_ZERO 0
#define MIPS_R_AT 1
#define MIPS_R_V0 2 /* BPF_R0 */
#define MIPS_R_V1 3
#define MIPS_R_A0 4 /* BPF_R1 */
#define MIPS_R_A1 5 /* BPF_R2 */
#define MIPS_R_A2 6 /* BPF_R3 */
#define MIPS_R_A3 7 /* BPF_R4 */
#define MIPS_R_A4 8 /* BPF_R5 */
#define MIPS_R_T4 12 /* BPF_AX */
#define MIPS_R_T5 13
#define MIPS_R_T6 14
#define MIPS_R_T7 15
#define MIPS_R_S0 16 /* BPF_R6 */
#define MIPS_R_S1 17 /* BPF_R7 */
#define MIPS_R_S2 18 /* BPF_R8 */
#define MIPS_R_S3 19 /* BPF_R9 */
#define MIPS_R_S4 20 /* BPF_TCC */
#define MIPS_R_S5 21
#define MIPS_R_S6 22
#define MIPS_R_S7 23
#define MIPS_R_T8 24
#define MIPS_R_T9 25
#define MIPS_R_SP 29
#define MIPS_R_RA 31
/* eBPF flags */
#define EBPF_SAVE_S0 BIT(0)
#define EBPF_SAVE_S1 BIT(1)
#define EBPF_SAVE_S2 BIT(2)
#define EBPF_SAVE_S3 BIT(3)
#define EBPF_SAVE_S4 BIT(4)
#define EBPF_SAVE_RA BIT(5)
#define EBPF_SEEN_FP BIT(6)
#define EBPF_SEEN_TC BIT(7)
#define EBPF_TCC_IN_V1 BIT(8)
/*
* For the mips64 ISA, we need to track the value range or type for
* each JIT register. The BPF machine requires zero extended 32-bit
* values, but the mips64 ISA requires sign extended 32-bit values.
* At each point in the BPF program we track the state of every
* register so that we can zero extend or sign extend as the BPF
* semantics require.
*/
enum reg_val_type {
/* uninitialized */
REG_UNKNOWN,
/* not known to be 32-bit compatible. */
REG_64BIT,
/* 32-bit compatible, no truncation needed for 64-bit ops. */
REG_64BIT_32BIT,
/* 32-bit compatible, need truncation for 64-bit ops. */
REG_32BIT,
/* 32-bit no sign/zero extension needed. */
REG_32BIT_POS
};
/*
* high bit of offsets indicates if long branch conversion done at
* this insn.
*/
#define OFFSETS_B_CONV BIT(31)
/**
* struct jit_ctx - JIT context
* @skf: The sk_filter
* @stack_size: eBPF stack size
* @idx: Instruction index
* @flags: JIT flags
* @offsets: Instruction offsets
* @target: Memory location for the compiled filter
* @reg_val_types Packed enum reg_val_type for each register.
*/
struct jit_ctx {
const struct bpf_prog *skf;
int stack_size;
u32 idx;
u32 flags;
u32 *offsets;
u32 *target;
u64 *reg_val_types;
unsigned int long_b_conversion:1;
unsigned int gen_b_offsets:1;
unsigned int use_bbit_insns:1;
};
static void set_reg_val_type(u64 *rvt, int reg, enum reg_val_type type)
{
*rvt &= ~(7ull << (reg * 3));
*rvt |= ((u64)type << (reg * 3));
}
static enum reg_val_type get_reg_val_type(const struct jit_ctx *ctx,
int index, int reg)
{
return (ctx->reg_val_types[index] >> (reg * 3)) & 7;
}
/* Simply emit the instruction if the JIT memory space has been allocated */
#define emit_instr_long(ctx, func64, func32, ...) \
do { \
if ((ctx)->target != NULL) { \
u32 *p = &(ctx)->target[ctx->idx]; \
if (IS_ENABLED(CONFIG_64BIT)) \
uasm_i_##func64(&p, ##__VA_ARGS__); \
else \
uasm_i_##func32(&p, ##__VA_ARGS__); \
} \
(ctx)->idx++; \
} while (0)
#define emit_instr(ctx, func, ...) \
emit_instr_long(ctx, func, func, ##__VA_ARGS__)
static unsigned int j_target(struct jit_ctx *ctx, int target_idx)
{
unsigned long target_va, base_va;
unsigned int r;
if (!ctx->target)
return 0;
base_va = (unsigned long)ctx->target;
target_va = base_va + (ctx->offsets[target_idx] & ~OFFSETS_B_CONV);
if ((base_va & ~0x0ffffffful) != (target_va & ~0x0ffffffful))
return (unsigned int)-1;
r = target_va & 0x0ffffffful;
return r;
}
/* Compute the immediate value for PC-relative branches. */
static u32 b_imm(unsigned int tgt, struct jit_ctx *ctx)
{
if (!ctx->gen_b_offsets)
return 0;
/*
* We want a pc-relative branch. tgt is the instruction offset
* we want to jump to.
* Branch on MIPS:
* I: target_offset <- sign_extend(offset)
* I+1: PC += target_offset (delay slot)
*
* ctx->idx currently points to the branch instruction
* but the offset is added to the delay slot so we need
* to subtract 4.
*/
return (ctx->offsets[tgt] & ~OFFSETS_B_CONV) -
(ctx->idx * 4) - 4;
}
enum which_ebpf_reg {
src_reg,
src_reg_no_fp,
dst_reg,
dst_reg_fp_ok
};
/*
* For eBPF, the register mapping naturally falls out of the
* requirements of eBPF and the MIPS n64 ABI. We don't maintain a
* separate frame pointer, so BPF_REG_10 relative accesses are
* adjusted to be $sp relative.
*/
static int ebpf_to_mips_reg(struct jit_ctx *ctx,
const struct bpf_insn *insn,
enum which_ebpf_reg w)
{
int ebpf_reg = (w == src_reg || w == src_reg_no_fp) ?
insn->src_reg : insn->dst_reg;
switch (ebpf_reg) {
case BPF_REG_0:
return MIPS_R_V0;
case BPF_REG_1:
return MIPS_R_A0;
case BPF_REG_2:
return MIPS_R_A1;
case BPF_REG_3:
return MIPS_R_A2;
case BPF_REG_4:
return MIPS_R_A3;
case BPF_REG_5:
return MIPS_R_A4;
case BPF_REG_6:
ctx->flags |= EBPF_SAVE_S0;
return MIPS_R_S0;
case BPF_REG_7:
ctx->flags |= EBPF_SAVE_S1;
return MIPS_R_S1;
case BPF_REG_8:
ctx->flags |= EBPF_SAVE_S2;
return MIPS_R_S2;
case BPF_REG_9:
ctx->flags |= EBPF_SAVE_S3;
return MIPS_R_S3;
case BPF_REG_10:
if (w == dst_reg || w == src_reg_no_fp)
goto bad_reg;
ctx->flags |= EBPF_SEEN_FP;
/*
* Needs special handling, return something that
* cannot be clobbered just in case.
*/
return MIPS_R_ZERO;
case BPF_REG_AX:
return MIPS_R_T4;
default:
bad_reg:
WARN(1, "Illegal bpf reg: %d\n", ebpf_reg);
return -EINVAL;
}
}
/*
* eBPF stack frame will be something like:
*
* Entry $sp ------> +--------------------------------+
* | $ra (optional) |
* +--------------------------------+
* | $s0 (optional) |
* +--------------------------------+
* | $s1 (optional) |
* +--------------------------------+
* | $s2 (optional) |
* +--------------------------------+
* | $s3 (optional) |
* +--------------------------------+
* | $s4 (optional) |
* +--------------------------------+
* | tmp-storage (if $ra saved) |
* $sp + tmp_offset --> +--------------------------------+ <--BPF_REG_10
* | BPF_REG_10 relative storage |
* | MAX_BPF_STACK (optional) |
* | . |
* | . |
* | . |
* $sp --------> +--------------------------------+
*
* If BPF_REG_10 is never referenced, then the MAX_BPF_STACK sized
* area is not allocated.
*/
static int gen_int_prologue(struct jit_ctx *ctx)
{
int stack_adjust = 0;
int store_offset;
int locals_size;
if (ctx->flags & EBPF_SAVE_RA)
/*
* If RA we are doing a function call and may need
* extra 8-byte tmp area.
*/
stack_adjust += 2 * sizeof(long);
if (ctx->flags & EBPF_SAVE_S0)
stack_adjust += sizeof(long);
if (ctx->flags & EBPF_SAVE_S1)
stack_adjust += sizeof(long);
if (ctx->flags & EBPF_SAVE_S2)
stack_adjust += sizeof(long);
if (ctx->flags & EBPF_SAVE_S3)
stack_adjust += sizeof(long);
if (ctx->flags & EBPF_SAVE_S4)
stack_adjust += sizeof(long);
BUILD_BUG_ON(MAX_BPF_STACK & 7);
locals_size = (ctx->flags & EBPF_SEEN_FP) ? MAX_BPF_STACK : 0;
stack_adjust += locals_size;
ctx->stack_size = stack_adjust;
/*
* First instruction initializes the tail call count (TCC).
* On tail call we skip this instruction, and the TCC is
* passed in $v1 from the caller.
*/
emit_instr(ctx, addiu, MIPS_R_V1, MIPS_R_ZERO, MAX_TAIL_CALL_CNT);
if (stack_adjust)
emit_instr_long(ctx, daddiu, addiu,
MIPS_R_SP, MIPS_R_SP, -stack_adjust);
else
return 0;
store_offset = stack_adjust - sizeof(long);
if (ctx->flags & EBPF_SAVE_RA) {
emit_instr_long(ctx, sd, sw,
MIPS_R_RA, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S0) {
emit_instr_long(ctx, sd, sw,
MIPS_R_S0, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S1) {
emit_instr_long(ctx, sd, sw,
MIPS_R_S1, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S2) {
emit_instr_long(ctx, sd, sw,
MIPS_R_S2, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S3) {
emit_instr_long(ctx, sd, sw,
MIPS_R_S3, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S4) {
emit_instr_long(ctx, sd, sw,
MIPS_R_S4, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if ((ctx->flags & EBPF_SEEN_TC) && !(ctx->flags & EBPF_TCC_IN_V1))
emit_instr_long(ctx, daddu, addu,
MIPS_R_S4, MIPS_R_V1, MIPS_R_ZERO);
return 0;
}
static int build_int_epilogue(struct jit_ctx *ctx, int dest_reg)
{
const struct bpf_prog *prog = ctx->skf;
int stack_adjust = ctx->stack_size;
int store_offset = stack_adjust - sizeof(long);
enum reg_val_type td;
int r0 = MIPS_R_V0;
if (dest_reg == MIPS_R_RA) {
/* Don't let zero extended value escape. */
td = get_reg_val_type(ctx, prog->len, BPF_REG_0);
if (td == REG_64BIT)
emit_instr(ctx, sll, r0, r0, 0);
}
if (ctx->flags & EBPF_SAVE_RA) {
emit_instr_long(ctx, ld, lw,
MIPS_R_RA, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S0) {
emit_instr_long(ctx, ld, lw,
MIPS_R_S0, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S1) {
emit_instr_long(ctx, ld, lw,
MIPS_R_S1, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S2) {
emit_instr_long(ctx, ld, lw,
MIPS_R_S2, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S3) {
emit_instr_long(ctx, ld, lw,
MIPS_R_S3, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
if (ctx->flags & EBPF_SAVE_S4) {
emit_instr_long(ctx, ld, lw,
MIPS_R_S4, store_offset, MIPS_R_SP);
store_offset -= sizeof(long);
}
emit_instr(ctx, jr, dest_reg);
if (stack_adjust)
emit_instr_long(ctx, daddiu, addiu,
MIPS_R_SP, MIPS_R_SP, stack_adjust);
else
emit_instr(ctx, nop);
return 0;
}
static void gen_imm_to_reg(const struct bpf_insn *insn, int reg,
struct jit_ctx *ctx)
{
if (insn->imm >= S16_MIN && insn->imm <= S16_MAX) {
emit_instr(ctx, addiu, reg, MIPS_R_ZERO, insn->imm);
} else {
int lower = (s16)(insn->imm & 0xffff);
int upper = insn->imm - lower;
emit_instr(ctx, lui, reg, upper >> 16);
emit_instr(ctx, addiu, reg, reg, lower);
}
}
static int gen_imm_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
int idx)
{
int upper_bound, lower_bound;
int dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
switch (BPF_OP(insn->code)) {
case BPF_MOV:
case BPF_ADD:
upper_bound = S16_MAX;
lower_bound = S16_MIN;
break;
case BPF_SUB:
upper_bound = -(int)S16_MIN;
lower_bound = -(int)S16_MAX;
break;
case BPF_AND:
case BPF_OR:
case BPF_XOR:
upper_bound = 0xffff;
lower_bound = 0;
break;
case BPF_RSH:
case BPF_LSH:
case BPF_ARSH:
/* Shift amounts are truncated, no need for bounds */
upper_bound = S32_MAX;
lower_bound = S32_MIN;
break;
default:
return -EINVAL;
}
/*
* Immediate move clobbers the register, so no sign/zero
* extension needed.
*/
if (BPF_CLASS(insn->code) == BPF_ALU64 &&
BPF_OP(insn->code) != BPF_MOV &&
get_reg_val_type(ctx, idx, insn->dst_reg) == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
/* BPF_ALU | BPF_LSH doesn't need separate sign extension */
if (BPF_CLASS(insn->code) == BPF_ALU &&
BPF_OP(insn->code) != BPF_LSH &&
BPF_OP(insn->code) != BPF_MOV &&
get_reg_val_type(ctx, idx, insn->dst_reg) != REG_32BIT)
emit_instr(ctx, sll, dst, dst, 0);
if (insn->imm >= lower_bound && insn->imm <= upper_bound) {
/* single insn immediate case */
switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) {
case BPF_ALU64 | BPF_MOV:
emit_instr(ctx, daddiu, dst, MIPS_R_ZERO, insn->imm);
break;
case BPF_ALU64 | BPF_AND:
case BPF_ALU | BPF_AND:
emit_instr(ctx, andi, dst, dst, insn->imm);
break;
case BPF_ALU64 | BPF_OR:
case BPF_ALU | BPF_OR:
emit_instr(ctx, ori, dst, dst, insn->imm);
break;
case BPF_ALU64 | BPF_XOR:
case BPF_ALU | BPF_XOR:
emit_instr(ctx, xori, dst, dst, insn->imm);
break;
case BPF_ALU64 | BPF_ADD:
emit_instr(ctx, daddiu, dst, dst, insn->imm);
break;
case BPF_ALU64 | BPF_SUB:
emit_instr(ctx, daddiu, dst, dst, -insn->imm);
break;
case BPF_ALU64 | BPF_RSH:
emit_instr(ctx, dsrl_safe, dst, dst, insn->imm & 0x3f);
break;
case BPF_ALU | BPF_RSH:
emit_instr(ctx, srl, dst, dst, insn->imm & 0x1f);
break;
case BPF_ALU64 | BPF_LSH:
emit_instr(ctx, dsll_safe, dst, dst, insn->imm & 0x3f);
break;
case BPF_ALU | BPF_LSH:
emit_instr(ctx, sll, dst, dst, insn->imm & 0x1f);
break;
case BPF_ALU64 | BPF_ARSH:
emit_instr(ctx, dsra_safe, dst, dst, insn->imm & 0x3f);
break;
case BPF_ALU | BPF_ARSH:
emit_instr(ctx, sra, dst, dst, insn->imm & 0x1f);
break;
case BPF_ALU | BPF_MOV:
emit_instr(ctx, addiu, dst, MIPS_R_ZERO, insn->imm);
break;
case BPF_ALU | BPF_ADD:
emit_instr(ctx, addiu, dst, dst, insn->imm);
break;
case BPF_ALU | BPF_SUB:
emit_instr(ctx, addiu, dst, dst, -insn->imm);
break;
default:
return -EINVAL;
}
} else {
/* multi insn immediate case */
if (BPF_OP(insn->code) == BPF_MOV) {
gen_imm_to_reg(insn, dst, ctx);
} else {
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
switch (BPF_OP(insn->code) | BPF_CLASS(insn->code)) {
case BPF_ALU64 | BPF_AND:
case BPF_ALU | BPF_AND:
emit_instr(ctx, and, dst, dst, MIPS_R_AT);
break;
case BPF_ALU64 | BPF_OR:
case BPF_ALU | BPF_OR:
emit_instr(ctx, or, dst, dst, MIPS_R_AT);
break;
case BPF_ALU64 | BPF_XOR:
case BPF_ALU | BPF_XOR:
emit_instr(ctx, xor, dst, dst, MIPS_R_AT);
break;
case BPF_ALU64 | BPF_ADD:
emit_instr(ctx, daddu, dst, dst, MIPS_R_AT);
break;
case BPF_ALU64 | BPF_SUB:
emit_instr(ctx, dsubu, dst, dst, MIPS_R_AT);
break;
case BPF_ALU | BPF_ADD:
emit_instr(ctx, addu, dst, dst, MIPS_R_AT);
break;
case BPF_ALU | BPF_SUB:
emit_instr(ctx, subu, dst, dst, MIPS_R_AT);
break;
default:
return -EINVAL;
}
}
}
return 0;
}
static void emit_const_to_reg(struct jit_ctx *ctx, int dst, u64 value)
{
if (value >= 0xffffffffffff8000ull || value < 0x8000ull) {
emit_instr(ctx, daddiu, dst, MIPS_R_ZERO, (int)value);
} else if (value >= 0xffffffff80000000ull ||
(value < 0x80000000 && value > 0xffff)) {
emit_instr(ctx, lui, dst, (s32)(s16)(value >> 16));
emit_instr(ctx, ori, dst, dst, (unsigned int)(value & 0xffff));
} else {
int i;
bool seen_part = false;
int needed_shift = 0;
for (i = 0; i < 4; i++) {
u64 part = (value >> (16 * (3 - i))) & 0xffff;
if (seen_part && needed_shift > 0 && (part || i == 3)) {
emit_instr(ctx, dsll_safe, dst, dst, needed_shift);
needed_shift = 0;
}
if (part) {
if (i == 0 || (!seen_part && i < 3 && part < 0x8000)) {
emit_instr(ctx, lui, dst, (s32)(s16)part);
needed_shift = -16;
} else {
emit_instr(ctx, ori, dst,
seen_part ? dst : MIPS_R_ZERO,
(unsigned int)part);
}
seen_part = true;
}
if (seen_part)
needed_shift += 16;
}
}
}
static int emit_bpf_tail_call(struct jit_ctx *ctx, int this_idx)
{
int off, b_off;
int tcc_reg;
ctx->flags |= EBPF_SEEN_TC;
/*
* if (index >= array->map.max_entries)
* goto out;
*/
off = offsetof(struct bpf_array, map.max_entries);
emit_instr(ctx, lwu, MIPS_R_T5, off, MIPS_R_A1);
emit_instr(ctx, sltu, MIPS_R_AT, MIPS_R_T5, MIPS_R_A2);
b_off = b_imm(this_idx + 1, ctx);
emit_instr(ctx, bne, MIPS_R_AT, MIPS_R_ZERO, b_off);
/*
* if (TCC-- < 0)
* goto out;
*/
/* Delay slot */
tcc_reg = (ctx->flags & EBPF_TCC_IN_V1) ? MIPS_R_V1 : MIPS_R_S4;
emit_instr(ctx, daddiu, MIPS_R_T5, tcc_reg, -1);
b_off = b_imm(this_idx + 1, ctx);
emit_instr(ctx, bltz, tcc_reg, b_off);
/*
* prog = array->ptrs[index];
* if (prog == NULL)
* goto out;
*/
/* Delay slot */
emit_instr(ctx, dsll, MIPS_R_T8, MIPS_R_A2, 3);
emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, MIPS_R_A1);
off = offsetof(struct bpf_array, ptrs);
emit_instr(ctx, ld, MIPS_R_AT, off, MIPS_R_T8);
b_off = b_imm(this_idx + 1, ctx);
emit_instr(ctx, beq, MIPS_R_AT, MIPS_R_ZERO, b_off);
/* Delay slot */
emit_instr(ctx, nop);
/* goto *(prog->bpf_func + 4); */
off = offsetof(struct bpf_prog, bpf_func);
emit_instr(ctx, ld, MIPS_R_T9, off, MIPS_R_AT);
/* All systems are go... propagate TCC */
emit_instr(ctx, daddu, MIPS_R_V1, MIPS_R_T5, MIPS_R_ZERO);
/* Skip first instruction (TCC initialization) */
emit_instr(ctx, daddiu, MIPS_R_T9, MIPS_R_T9, 4);
return build_int_epilogue(ctx, MIPS_R_T9);
}
static bool is_bad_offset(int b_off)
{
return b_off > 0x1ffff || b_off < -0x20000;
}
/* Returns the number of insn slots consumed. */
static int build_one_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
int this_idx, int exit_idx)
{
int src, dst, r, td, ts, mem_off, b_off;
bool need_swap, did_move, cmp_eq;
unsigned int target = 0;
u64 t64;
s64 t64s;
int bpf_op = BPF_OP(insn->code);
if (IS_ENABLED(CONFIG_32BIT) && ((BPF_CLASS(insn->code) == BPF_ALU64)
|| (bpf_op == BPF_DW)))
return -EINVAL;
switch (insn->code) {
case BPF_ALU64 | BPF_ADD | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_SUB | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_OR | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_AND | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_LSH | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_RSH | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_XOR | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_ARSH | BPF_K: /* ALU64_IMM */
case BPF_ALU64 | BPF_MOV | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_MOV | BPF_K: /* ALU32_IMM */
case BPF_ALU | BPF_ADD | BPF_K: /* ALU32_IMM */
case BPF_ALU | BPF_SUB | BPF_K: /* ALU32_IMM */
case BPF_ALU | BPF_OR | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_AND | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_LSH | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_RSH | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_XOR | BPF_K: /* ALU64_IMM */
case BPF_ALU | BPF_ARSH | BPF_K: /* ALU64_IMM */
r = gen_imm_insn(insn, ctx, this_idx);
if (r < 0)
return r;
break;
case BPF_ALU64 | BPF_MUL | BPF_K: /* ALU64_IMM */
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
if (insn->imm == 1) /* Mult by 1 is a nop */
break;
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
if (MIPS_ISA_REV >= 6) {
emit_instr(ctx, dmulu, dst, dst, MIPS_R_AT);
} else {
emit_instr(ctx, dmultu, MIPS_R_AT, dst);
emit_instr(ctx, mflo, dst);
}
break;
case BPF_ALU64 | BPF_NEG | BPF_K: /* ALU64_IMM */
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
emit_instr(ctx, dsubu, dst, MIPS_R_ZERO, dst);
break;
case BPF_ALU | BPF_MUL | BPF_K: /* ALU_IMM */
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
if (td == REG_64BIT) {
/* sign extend */
emit_instr(ctx, sll, dst, dst, 0);
}
if (insn->imm == 1) /* Mult by 1 is a nop */
break;
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
if (MIPS_ISA_REV >= 6) {
emit_instr(ctx, mulu, dst, dst, MIPS_R_AT);
} else {
emit_instr(ctx, multu, dst, MIPS_R_AT);
emit_instr(ctx, mflo, dst);
}
break;
case BPF_ALU | BPF_NEG | BPF_K: /* ALU_IMM */
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
if (td == REG_64BIT) {
/* sign extend */
emit_instr(ctx, sll, dst, dst, 0);
}
emit_instr(ctx, subu, dst, MIPS_R_ZERO, dst);
break;
case BPF_ALU | BPF_DIV | BPF_K: /* ALU_IMM */
case BPF_ALU | BPF_MOD | BPF_K: /* ALU_IMM */
if (insn->imm == 0)
return -EINVAL;
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
if (td == REG_64BIT)
/* sign extend */
emit_instr(ctx, sll, dst, dst, 0);
if (insn->imm == 1) {
/* div by 1 is a nop, mod by 1 is zero */
if (bpf_op == BPF_MOD)
emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO);
break;
}
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
if (MIPS_ISA_REV >= 6) {
if (bpf_op == BPF_DIV)
emit_instr(ctx, divu_r6, dst, dst, MIPS_R_AT);
else
emit_instr(ctx, modu, dst, dst, MIPS_R_AT);
break;
}
emit_instr(ctx, divu, dst, MIPS_R_AT);
if (bpf_op == BPF_DIV)
emit_instr(ctx, mflo, dst);
else
emit_instr(ctx, mfhi, dst);
break;
case BPF_ALU64 | BPF_DIV | BPF_K: /* ALU_IMM */
case BPF_ALU64 | BPF_MOD | BPF_K: /* ALU_IMM */
if (insn->imm == 0)
return -EINVAL;
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
if (insn->imm == 1) {
/* div by 1 is a nop, mod by 1 is zero */
if (bpf_op == BPF_MOD)
emit_instr(ctx, addu, dst, MIPS_R_ZERO, MIPS_R_ZERO);
break;
}
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
if (MIPS_ISA_REV >= 6) {
if (bpf_op == BPF_DIV)
emit_instr(ctx, ddivu_r6, dst, dst, MIPS_R_AT);
else
emit_instr(ctx, modu, dst, dst, MIPS_R_AT);
break;
}
emit_instr(ctx, ddivu, dst, MIPS_R_AT);
if (bpf_op == BPF_DIV)
emit_instr(ctx, mflo, dst);
else
emit_instr(ctx, mfhi, dst);
break;
case BPF_ALU64 | BPF_MOV | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_ADD | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_SUB | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_XOR | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_OR | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_AND | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_MUL | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_DIV | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_MOD | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_LSH | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_RSH | BPF_X: /* ALU64_REG */
case BPF_ALU64 | BPF_ARSH | BPF_X: /* ALU64_REG */
src = ebpf_to_mips_reg(ctx, insn, src_reg);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (src < 0 || dst < 0)
return -EINVAL;
if (get_reg_val_type(ctx, this_idx, insn->dst_reg) == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
did_move = false;
if (insn->src_reg == BPF_REG_10) {
if (bpf_op == BPF_MOV) {
emit_instr(ctx, daddiu, dst, MIPS_R_SP, MAX_BPF_STACK);
did_move = true;
} else {
emit_instr(ctx, daddiu, MIPS_R_AT, MIPS_R_SP, MAX_BPF_STACK);
src = MIPS_R_AT;
}
} else if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) {
int tmp_reg = MIPS_R_AT;
if (bpf_op == BPF_MOV) {
tmp_reg = dst;
did_move = true;
}
emit_instr(ctx, daddu, tmp_reg, src, MIPS_R_ZERO);
emit_instr(ctx, dinsu, tmp_reg, MIPS_R_ZERO, 32, 32);
src = MIPS_R_AT;
}
switch (bpf_op) {
case BPF_MOV:
if (!did_move)
emit_instr(ctx, daddu, dst, src, MIPS_R_ZERO);
break;
case BPF_ADD:
emit_instr(ctx, daddu, dst, dst, src);
break;
case BPF_SUB:
emit_instr(ctx, dsubu, dst, dst, src);
break;
case BPF_XOR:
emit_instr(ctx, xor, dst, dst, src);
break;
case BPF_OR:
emit_instr(ctx, or, dst, dst, src);
break;
case BPF_AND:
emit_instr(ctx, and, dst, dst, src);
break;
case BPF_MUL:
if (MIPS_ISA_REV >= 6) {
emit_instr(ctx, dmulu, dst, dst, src);
} else {
emit_instr(ctx, dmultu, dst, src);
emit_instr(ctx, mflo, dst);
}
break;
case BPF_DIV:
case BPF_MOD:
if (MIPS_ISA_REV >= 6) {
if (bpf_op == BPF_DIV)
emit_instr(ctx, ddivu_r6,
dst, dst, src);
else
emit_instr(ctx, modu, dst, dst, src);
break;
}
emit_instr(ctx, ddivu, dst, src);
if (bpf_op == BPF_DIV)
emit_instr(ctx, mflo, dst);
else
emit_instr(ctx, mfhi, dst);
break;
case BPF_LSH:
emit_instr(ctx, dsllv, dst, dst, src);
break;
case BPF_RSH:
emit_instr(ctx, dsrlv, dst, dst, src);
break;
case BPF_ARSH:
emit_instr(ctx, dsrav, dst, dst, src);
break;
default:
pr_err("ALU64_REG NOT HANDLED\n");
return -EINVAL;
}
break;
case BPF_ALU | BPF_MOV | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_ADD | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_SUB | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_XOR | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_OR | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_AND | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_MUL | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_DIV | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_MOD | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_LSH | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_RSH | BPF_X: /* ALU_REG */
case BPF_ALU | BPF_ARSH | BPF_X: /* ALU_REG */
src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (src < 0 || dst < 0)
return -EINVAL;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
if (td == REG_64BIT) {
/* sign extend */
emit_instr(ctx, sll, dst, dst, 0);
}
did_move = false;
ts = get_reg_val_type(ctx, this_idx, insn->src_reg);
if (ts == REG_64BIT) {
int tmp_reg = MIPS_R_AT;
if (bpf_op == BPF_MOV) {
tmp_reg = dst;
did_move = true;
}
/* sign extend */
emit_instr(ctx, sll, tmp_reg, src, 0);
src = MIPS_R_AT;
}
switch (bpf_op) {
case BPF_MOV:
if (!did_move)
emit_instr(ctx, addu, dst, src, MIPS_R_ZERO);
break;
case BPF_ADD:
emit_instr(ctx, addu, dst, dst, src);
break;
case BPF_SUB:
emit_instr(ctx, subu, dst, dst, src);
break;
case BPF_XOR:
emit_instr(ctx, xor, dst, dst, src);
break;
case BPF_OR:
emit_instr(ctx, or, dst, dst, src);
break;
case BPF_AND:
emit_instr(ctx, and, dst, dst, src);
break;
case BPF_MUL:
emit_instr(ctx, mul, dst, dst, src);
break;
case BPF_DIV:
case BPF_MOD:
if (MIPS_ISA_REV >= 6) {
if (bpf_op == BPF_DIV)
emit_instr(ctx, divu_r6, dst, dst, src);
else
emit_instr(ctx, modu, dst, dst, src);
break;
}
emit_instr(ctx, divu, dst, src);
if (bpf_op == BPF_DIV)
emit_instr(ctx, mflo, dst);
else
emit_instr(ctx, mfhi, dst);
break;
case BPF_LSH:
emit_instr(ctx, sllv, dst, dst, src);
break;
case BPF_RSH:
emit_instr(ctx, srlv, dst, dst, src);
break;
case BPF_ARSH:
emit_instr(ctx, srav, dst, dst, src);
break;
default:
pr_err("ALU_REG NOT HANDLED\n");
return -EINVAL;
}
break;
case BPF_JMP | BPF_EXIT:
if (this_idx + 1 < exit_idx) {
b_off = b_imm(exit_idx, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_instr(ctx, beq, MIPS_R_ZERO, MIPS_R_ZERO, b_off);
emit_instr(ctx, nop);
}
break;
case BPF_JMP | BPF_JEQ | BPF_K: /* JMP_IMM */
case BPF_JMP | BPF_JNE | BPF_K: /* JMP_IMM */
cmp_eq = (bpf_op == BPF_JEQ);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok);
if (dst < 0)
return dst;
if (insn->imm == 0) {
src = MIPS_R_ZERO;
} else {
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
src = MIPS_R_AT;
}
goto jeq_common;
case BPF_JMP | BPF_JEQ | BPF_X: /* JMP_REG */
case BPF_JMP | BPF_JNE | BPF_X:
case BPF_JMP | BPF_JSLT | BPF_X:
case BPF_JMP | BPF_JSLE | BPF_X:
case BPF_JMP | BPF_JSGT | BPF_X:
case BPF_JMP | BPF_JSGE | BPF_X:
case BPF_JMP | BPF_JLT | BPF_X:
case BPF_JMP | BPF_JLE | BPF_X:
case BPF_JMP | BPF_JGT | BPF_X:
case BPF_JMP | BPF_JGE | BPF_X:
case BPF_JMP | BPF_JSET | BPF_X:
src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (src < 0 || dst < 0)
return -EINVAL;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
ts = get_reg_val_type(ctx, this_idx, insn->src_reg);
if (td == REG_32BIT && ts != REG_32BIT) {
emit_instr(ctx, sll, MIPS_R_AT, src, 0);
src = MIPS_R_AT;
} else if (ts == REG_32BIT && td != REG_32BIT) {
emit_instr(ctx, sll, MIPS_R_AT, dst, 0);
dst = MIPS_R_AT;
}
if (bpf_op == BPF_JSET) {
emit_instr(ctx, and, MIPS_R_AT, dst, src);
cmp_eq = false;
dst = MIPS_R_AT;
src = MIPS_R_ZERO;
} else if (bpf_op == BPF_JSGT || bpf_op == BPF_JSLE) {
emit_instr(ctx, dsubu, MIPS_R_AT, dst, src);
if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) {
b_off = b_imm(exit_idx, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
if (bpf_op == BPF_JSGT)
emit_instr(ctx, blez, MIPS_R_AT, b_off);
else
emit_instr(ctx, bgtz, MIPS_R_AT, b_off);
emit_instr(ctx, nop);
return 2; /* We consumed the exit. */
}
b_off = b_imm(this_idx + insn->off + 1, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
if (bpf_op == BPF_JSGT)
emit_instr(ctx, bgtz, MIPS_R_AT, b_off);
else
emit_instr(ctx, blez, MIPS_R_AT, b_off);
emit_instr(ctx, nop);
break;
} else if (bpf_op == BPF_JSGE || bpf_op == BPF_JSLT) {
emit_instr(ctx, slt, MIPS_R_AT, dst, src);
cmp_eq = bpf_op == BPF_JSGE;
dst = MIPS_R_AT;
src = MIPS_R_ZERO;
} else if (bpf_op == BPF_JGT || bpf_op == BPF_JLE) {
/* dst or src could be AT */
emit_instr(ctx, dsubu, MIPS_R_T8, dst, src);
emit_instr(ctx, sltu, MIPS_R_AT, dst, src);
/* SP known to be non-zero, movz becomes boolean not */
if (MIPS_ISA_REV >= 6) {
emit_instr(ctx, seleqz, MIPS_R_T9,
MIPS_R_SP, MIPS_R_T8);
} else {
emit_instr(ctx, movz, MIPS_R_T9,
MIPS_R_SP, MIPS_R_T8);
emit_instr(ctx, movn, MIPS_R_T9,
MIPS_R_ZERO, MIPS_R_T8);
}
emit_instr(ctx, or, MIPS_R_AT, MIPS_R_T9, MIPS_R_AT);
cmp_eq = bpf_op == BPF_JGT;
dst = MIPS_R_AT;
src = MIPS_R_ZERO;
} else if (bpf_op == BPF_JGE || bpf_op == BPF_JLT) {
emit_instr(ctx, sltu, MIPS_R_AT, dst, src);
cmp_eq = bpf_op == BPF_JGE;
dst = MIPS_R_AT;
src = MIPS_R_ZERO;
} else { /* JNE/JEQ case */
cmp_eq = (bpf_op == BPF_JEQ);
}
jeq_common:
/*
* If the next insn is EXIT and we are jumping arround
* only it, invert the sense of the compare and
* conditionally jump to the exit. Poor man's branch
* chaining.
*/
if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) {
b_off = b_imm(exit_idx, ctx);
if (is_bad_offset(b_off)) {
target = j_target(ctx, exit_idx);
if (target == (unsigned int)-1)
return -E2BIG;
cmp_eq = !cmp_eq;
b_off = 4 * 3;
if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) {
ctx->offsets[this_idx] |= OFFSETS_B_CONV;
ctx->long_b_conversion = 1;
}
}
if (cmp_eq)
emit_instr(ctx, bne, dst, src, b_off);
else
emit_instr(ctx, beq, dst, src, b_off);
emit_instr(ctx, nop);
if (ctx->offsets[this_idx] & OFFSETS_B_CONV) {
emit_instr(ctx, j, target);
emit_instr(ctx, nop);
}
return 2; /* We consumed the exit. */
}
b_off = b_imm(this_idx + insn->off + 1, ctx);
if (is_bad_offset(b_off)) {
target = j_target(ctx, this_idx + insn->off + 1);
if (target == (unsigned int)-1)
return -E2BIG;
cmp_eq = !cmp_eq;
b_off = 4 * 3;
if (!(ctx->offsets[this_idx] & OFFSETS_B_CONV)) {
ctx->offsets[this_idx] |= OFFSETS_B_CONV;
ctx->long_b_conversion = 1;
}
}
if (cmp_eq)
emit_instr(ctx, beq, dst, src, b_off);
else
emit_instr(ctx, bne, dst, src, b_off);
emit_instr(ctx, nop);
if (ctx->offsets[this_idx] & OFFSETS_B_CONV) {
emit_instr(ctx, j, target);
emit_instr(ctx, nop);
}
break;
case BPF_JMP | BPF_JSGT | BPF_K: /* JMP_IMM */
case BPF_JMP | BPF_JSGE | BPF_K: /* JMP_IMM */
case BPF_JMP | BPF_JSLT | BPF_K: /* JMP_IMM */
case BPF_JMP | BPF_JSLE | BPF_K: /* JMP_IMM */
cmp_eq = (bpf_op == BPF_JSGE);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok);
if (dst < 0)
return dst;
if (insn->imm == 0) {
if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) {
b_off = b_imm(exit_idx, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
switch (bpf_op) {
case BPF_JSGT:
emit_instr(ctx, blez, dst, b_off);
break;
case BPF_JSGE:
emit_instr(ctx, bltz, dst, b_off);
break;
case BPF_JSLT:
emit_instr(ctx, bgez, dst, b_off);
break;
case BPF_JSLE:
emit_instr(ctx, bgtz, dst, b_off);
break;
}
emit_instr(ctx, nop);
return 2; /* We consumed the exit. */
}
b_off = b_imm(this_idx + insn->off + 1, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
switch (bpf_op) {
case BPF_JSGT:
emit_instr(ctx, bgtz, dst, b_off);
break;
case BPF_JSGE:
emit_instr(ctx, bgez, dst, b_off);
break;
case BPF_JSLT:
emit_instr(ctx, bltz, dst, b_off);
break;
case BPF_JSLE:
emit_instr(ctx, blez, dst, b_off);
break;
}
emit_instr(ctx, nop);
break;
}
/*
* only "LT" compare available, so we must use imm + 1
* to generate "GT" and imm -1 to generate LE
*/
if (bpf_op == BPF_JSGT)
t64s = insn->imm + 1;
else if (bpf_op == BPF_JSLE)
t64s = insn->imm + 1;
else
t64s = insn->imm;
cmp_eq = bpf_op == BPF_JSGT || bpf_op == BPF_JSGE;
if (t64s >= S16_MIN && t64s <= S16_MAX) {
emit_instr(ctx, slti, MIPS_R_AT, dst, (int)t64s);
src = MIPS_R_AT;
dst = MIPS_R_ZERO;
goto jeq_common;
}
emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s);
emit_instr(ctx, slt, MIPS_R_AT, dst, MIPS_R_AT);
src = MIPS_R_AT;
dst = MIPS_R_ZERO;
goto jeq_common;
case BPF_JMP | BPF_JGT | BPF_K:
case BPF_JMP | BPF_JGE | BPF_K:
case BPF_JMP | BPF_JLT | BPF_K:
case BPF_JMP | BPF_JLE | BPF_K:
cmp_eq = (bpf_op == BPF_JGE);
dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok);
if (dst < 0)
return dst;
/*
* only "LT" compare available, so we must use imm + 1
* to generate "GT" and imm -1 to generate LE
*/
if (bpf_op == BPF_JGT)
t64s = (u64)(u32)(insn->imm) + 1;
else if (bpf_op == BPF_JLE)
t64s = (u64)(u32)(insn->imm) + 1;
else
t64s = (u64)(u32)(insn->imm);
cmp_eq = bpf_op == BPF_JGT || bpf_op == BPF_JGE;
emit_const_to_reg(ctx, MIPS_R_AT, (u64)t64s);
emit_instr(ctx, sltu, MIPS_R_AT, dst, MIPS_R_AT);
src = MIPS_R_AT;
dst = MIPS_R_ZERO;
goto jeq_common;
case BPF_JMP | BPF_JSET | BPF_K: /* JMP_IMM */
dst = ebpf_to_mips_reg(ctx, insn, dst_reg_fp_ok);
if (dst < 0)
return dst;
if (ctx->use_bbit_insns && hweight32((u32)insn->imm) == 1) {
if ((insn + 1)->code == (BPF_JMP | BPF_EXIT) && insn->off == 1) {
b_off = b_imm(exit_idx, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_instr(ctx, bbit0, dst, ffs((u32)insn->imm) - 1, b_off);
emit_instr(ctx, nop);
return 2; /* We consumed the exit. */
}
b_off = b_imm(this_idx + insn->off + 1, ctx);
if (is_bad_offset(b_off))
return -E2BIG;
emit_instr(ctx, bbit1, dst, ffs((u32)insn->imm) - 1, b_off);
emit_instr(ctx, nop);
break;
}
t64 = (u32)insn->imm;
emit_const_to_reg(ctx, MIPS_R_AT, t64);
emit_instr(ctx, and, MIPS_R_AT, dst, MIPS_R_AT);
src = MIPS_R_AT;
dst = MIPS_R_ZERO;
cmp_eq = false;
goto jeq_common;
case BPF_JMP | BPF_JA:
/*
* Prefer relative branch for easier debugging, but
* fall back if needed.
*/
b_off = b_imm(this_idx + insn->off + 1, ctx);
if (is_bad_offset(b_off)) {
target = j_target(ctx, this_idx + insn->off + 1);
if (target == (unsigned int)-1)
return -E2BIG;
emit_instr(ctx, j, target);
} else {
emit_instr(ctx, b, b_off);
}
emit_instr(ctx, nop);
break;
case BPF_LD | BPF_DW | BPF_IMM:
if (insn->src_reg != 0)
return -EINVAL;
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
t64 = ((u64)(u32)insn->imm) | ((u64)(insn + 1)->imm << 32);
emit_const_to_reg(ctx, dst, t64);
return 2; /* Double slot insn */
case BPF_JMP | BPF_CALL:
ctx->flags |= EBPF_SAVE_RA;
t64s = (s64)insn->imm + (long)__bpf_call_base;
emit_const_to_reg(ctx, MIPS_R_T9, (u64)t64s);
emit_instr(ctx, jalr, MIPS_R_RA, MIPS_R_T9);
/* delay slot */
emit_instr(ctx, nop);
break;
case BPF_JMP | BPF_TAIL_CALL:
if (emit_bpf_tail_call(ctx, this_idx))
return -EINVAL;
break;
case BPF_ALU | BPF_END | BPF_FROM_BE:
case BPF_ALU | BPF_END | BPF_FROM_LE:
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
td = get_reg_val_type(ctx, this_idx, insn->dst_reg);
if (insn->imm == 64 && td == REG_32BIT)
emit_instr(ctx, dinsu, dst, MIPS_R_ZERO, 32, 32);
if (insn->imm != 64 && td == REG_64BIT) {
/* sign extend */
emit_instr(ctx, sll, dst, dst, 0);
}
#ifdef __BIG_ENDIAN
need_swap = (BPF_SRC(insn->code) == BPF_FROM_LE);
#else
need_swap = (BPF_SRC(insn->code) == BPF_FROM_BE);
#endif
if (insn->imm == 16) {
if (need_swap)
emit_instr(ctx, wsbh, dst, dst);
emit_instr(ctx, andi, dst, dst, 0xffff);
} else if (insn->imm == 32) {
if (need_swap) {
emit_instr(ctx, wsbh, dst, dst);
emit_instr(ctx, rotr, dst, dst, 16);
}
} else { /* 64-bit*/
if (need_swap) {
emit_instr(ctx, dsbh, dst, dst);
emit_instr(ctx, dshd, dst, dst);
}
}
break;
case BPF_ST | BPF_NOSPEC: /* speculation barrier */
break;
case BPF_ST | BPF_B | BPF_MEM:
case BPF_ST | BPF_H | BPF_MEM:
case BPF_ST | BPF_W | BPF_MEM:
case BPF_ST | BPF_DW | BPF_MEM:
if (insn->dst_reg == BPF_REG_10) {
ctx->flags |= EBPF_SEEN_FP;
dst = MIPS_R_SP;
mem_off = insn->off + MAX_BPF_STACK;
} else {
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
mem_off = insn->off;
}
gen_imm_to_reg(insn, MIPS_R_AT, ctx);
switch (BPF_SIZE(insn->code)) {
case BPF_B:
emit_instr(ctx, sb, MIPS_R_AT, mem_off, dst);
break;
case BPF_H:
emit_instr(ctx, sh, MIPS_R_AT, mem_off, dst);
break;
case BPF_W:
emit_instr(ctx, sw, MIPS_R_AT, mem_off, dst);
break;
case BPF_DW:
emit_instr(ctx, sd, MIPS_R_AT, mem_off, dst);
break;
}
break;
case BPF_LDX | BPF_B | BPF_MEM:
case BPF_LDX | BPF_H | BPF_MEM:
case BPF_LDX | BPF_W | BPF_MEM:
case BPF_LDX | BPF_DW | BPF_MEM:
if (insn->src_reg == BPF_REG_10) {
ctx->flags |= EBPF_SEEN_FP;
src = MIPS_R_SP;
mem_off = insn->off + MAX_BPF_STACK;
} else {
src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp);
if (src < 0)
return src;
mem_off = insn->off;
}
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
switch (BPF_SIZE(insn->code)) {
case BPF_B:
emit_instr(ctx, lbu, dst, mem_off, src);
break;
case BPF_H:
emit_instr(ctx, lhu, dst, mem_off, src);
break;
case BPF_W:
emit_instr(ctx, lw, dst, mem_off, src);
break;
case BPF_DW:
emit_instr(ctx, ld, dst, mem_off, src);
break;
}
break;
case BPF_STX | BPF_B | BPF_MEM:
case BPF_STX | BPF_H | BPF_MEM:
case BPF_STX | BPF_W | BPF_MEM:
case BPF_STX | BPF_DW | BPF_MEM:
case BPF_STX | BPF_W | BPF_ATOMIC:
case BPF_STX | BPF_DW | BPF_ATOMIC:
if (insn->dst_reg == BPF_REG_10) {
ctx->flags |= EBPF_SEEN_FP;
dst = MIPS_R_SP;
mem_off = insn->off + MAX_BPF_STACK;
} else {
dst = ebpf_to_mips_reg(ctx, insn, dst_reg);
if (dst < 0)
return dst;
mem_off = insn->off;
}
src = ebpf_to_mips_reg(ctx, insn, src_reg_no_fp);
if (src < 0)
return src;
if (BPF_MODE(insn->code) == BPF_ATOMIC) {
if (insn->imm != BPF_ADD) {
pr_err("ATOMIC OP %02x NOT HANDLED\n", insn->imm);
return -EINVAL;
}
/*
* If mem_off does not fit within the 9 bit ll/sc
* instruction immediate field, use a temp reg.
*/
if (MIPS_ISA_REV >= 6 &&
(mem_off >= BIT(8) || mem_off < -BIT(8))) {
emit_instr(ctx, daddiu, MIPS_R_T6,
dst, mem_off);
mem_off = 0;
dst = MIPS_R_T6;
}
switch (BPF_SIZE(insn->code)) {
case BPF_W:
if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) {
emit_instr(ctx, sll, MIPS_R_AT, src, 0);
src = MIPS_R_AT;
}
emit_instr(ctx, ll, MIPS_R_T8, mem_off, dst);
emit_instr(ctx, addu, MIPS_R_T8, MIPS_R_T8, src);
emit_instr(ctx, sc, MIPS_R_T8, mem_off, dst);
/*
* On failure back up to LL (-4
* instructions of 4 bytes each
*/
emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4);
emit_instr(ctx, nop);
break;
case BPF_DW:
if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) {
emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO);
emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32);
src = MIPS_R_AT;
}
emit_instr(ctx, lld, MIPS_R_T8, mem_off, dst);
emit_instr(ctx, daddu, MIPS_R_T8, MIPS_R_T8, src);
emit_instr(ctx, scd, MIPS_R_T8, mem_off, dst);
emit_instr(ctx, beq, MIPS_R_T8, MIPS_R_ZERO, -4 * 4);
emit_instr(ctx, nop);
break;
}
} else { /* BPF_MEM */
switch (BPF_SIZE(insn->code)) {
case BPF_B:
emit_instr(ctx, sb, src, mem_off, dst);
break;
case BPF_H:
emit_instr(ctx, sh, src, mem_off, dst);
break;
case BPF_W:
emit_instr(ctx, sw, src, mem_off, dst);
break;
case BPF_DW:
if (get_reg_val_type(ctx, this_idx, insn->src_reg) == REG_32BIT) {
emit_instr(ctx, daddu, MIPS_R_AT, src, MIPS_R_ZERO);
emit_instr(ctx, dinsu, MIPS_R_AT, MIPS_R_ZERO, 32, 32);
src = MIPS_R_AT;
}
emit_instr(ctx, sd, src, mem_off, dst);
break;
}
}
break;
default:
pr_err("NOT HANDLED %d - (%02x)\n",
this_idx, (unsigned int)insn->code);
return -EINVAL;
}
return 1;
}
#define RVT_VISITED_MASK 0xc000000000000000ull
#define RVT_FALL_THROUGH 0x4000000000000000ull
#define RVT_BRANCH_TAKEN 0x8000000000000000ull
#define RVT_DONE (RVT_FALL_THROUGH | RVT_BRANCH_TAKEN)
static int build_int_body(struct jit_ctx *ctx)
{
const struct bpf_prog *prog = ctx->skf;
const struct bpf_insn *insn;
int i, r;
for (i = 0; i < prog->len; ) {
insn = prog->insnsi + i;
if ((ctx->reg_val_types[i] & RVT_VISITED_MASK) == 0) {
/* dead instruction, don't emit it. */
i++;
continue;
}
if (ctx->target == NULL)
ctx->offsets[i] = (ctx->offsets[i] & OFFSETS_B_CONV) | (ctx->idx * 4);
r = build_one_insn(insn, ctx, i, prog->len);
if (r < 0)
return r;
i += r;
}
/* epilogue offset */
if (ctx->target == NULL)
ctx->offsets[i] = ctx->idx * 4;
/*
* All exits have an offset of the epilogue, some offsets may
* not have been set due to banch-around threading, so set
* them now.
*/
if (ctx->target == NULL)
for (i = 0; i < prog->len; i++) {
insn = prog->insnsi + i;
if (insn->code == (BPF_JMP | BPF_EXIT))
ctx->offsets[i] = ctx->idx * 4;
}
return 0;
}
/* return the last idx processed, or negative for error */
static int reg_val_propagate_range(struct jit_ctx *ctx, u64 initial_rvt,
int start_idx, bool follow_taken)
{
const struct bpf_prog *prog = ctx->skf;
const struct bpf_insn *insn;
u64 exit_rvt = initial_rvt;
u64 *rvt = ctx->reg_val_types;
int idx;
int reg;
for (idx = start_idx; idx < prog->len; idx++) {
rvt[idx] = (rvt[idx] & RVT_VISITED_MASK) | exit_rvt;
insn = prog->insnsi + idx;
switch (BPF_CLASS(insn->code)) {
case BPF_ALU:
switch (BPF_OP(insn->code)) {
case BPF_ADD:
case BPF_SUB:
case BPF_MUL:
case BPF_DIV:
case BPF_OR:
case BPF_AND:
case BPF_LSH:
case BPF_RSH:
case BPF_NEG:
case BPF_MOD:
case BPF_XOR:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
break;
case BPF_MOV:
if (BPF_SRC(insn->code)) {
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
} else {
/* IMM to REG move*/
if (insn->imm >= 0)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
else
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
}
break;
case BPF_END:
if (insn->imm == 64)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
else if (insn->imm == 32)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
else /* insn->imm == 16 */
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
break;
}
rvt[idx] |= RVT_DONE;
break;
case BPF_ALU64:
switch (BPF_OP(insn->code)) {
case BPF_MOV:
if (BPF_SRC(insn->code)) {
/* REG to REG move*/
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
} else {
/* IMM to REG move*/
if (insn->imm >= 0)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
else
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT);
}
break;
default:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
}
rvt[idx] |= RVT_DONE;
break;
case BPF_LD:
switch (BPF_SIZE(insn->code)) {
case BPF_DW:
if (BPF_MODE(insn->code) == BPF_IMM) {
s64 val;
val = (s64)((u32)insn->imm | ((u64)(insn + 1)->imm << 32));
if (val > 0 && val <= S32_MAX)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
else if (val >= S32_MIN && val <= S32_MAX)
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT_32BIT);
else
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
rvt[idx] |= RVT_DONE;
idx++;
} else {
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
}
break;
case BPF_B:
case BPF_H:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
break;
case BPF_W:
if (BPF_MODE(insn->code) == BPF_IMM)
set_reg_val_type(&exit_rvt, insn->dst_reg,
insn->imm >= 0 ? REG_32BIT_POS : REG_32BIT);
else
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
break;
}
rvt[idx] |= RVT_DONE;
break;
case BPF_LDX:
switch (BPF_SIZE(insn->code)) {
case BPF_DW:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_64BIT);
break;
case BPF_B:
case BPF_H:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT_POS);
break;
case BPF_W:
set_reg_val_type(&exit_rvt, insn->dst_reg, REG_32BIT);
break;
}
rvt[idx] |= RVT_DONE;
break;
case BPF_JMP:
switch (BPF_OP(insn->code)) {
case BPF_EXIT:
rvt[idx] = RVT_DONE | exit_rvt;
rvt[prog->len] = exit_rvt;
return idx;
case BPF_JA:
rvt[idx] |= RVT_DONE;
idx += insn->off;
break;
case BPF_JEQ:
case BPF_JGT:
case BPF_JGE:
case BPF_JLT:
case BPF_JLE:
case BPF_JSET:
case BPF_JNE:
case BPF_JSGT:
case BPF_JSGE:
case BPF_JSLT:
case BPF_JSLE:
if (follow_taken) {
rvt[idx] |= RVT_BRANCH_TAKEN;
idx += insn->off;
follow_taken = false;
} else {
rvt[idx] |= RVT_FALL_THROUGH;
}
break;
case BPF_CALL:
set_reg_val_type(&exit_rvt, BPF_REG_0, REG_64BIT);
/* Upon call return, argument registers are clobbered. */
for (reg = BPF_REG_0; reg <= BPF_REG_5; reg++)
set_reg_val_type(&exit_rvt, reg, REG_64BIT);
rvt[idx] |= RVT_DONE;
break;
default:
WARN(1, "Unhandled BPF_JMP case.\n");
rvt[idx] |= RVT_DONE;
break;
}
break;
default:
rvt[idx] |= RVT_DONE;
break;
}
}
return idx;
}
/*
* Track the value range (i.e. 32-bit vs. 64-bit) of each register at
* each eBPF insn. This allows unneeded sign and zero extension
* operations to be omitted.
*
* Doesn't handle yet confluence of control paths with conflicting
* ranges, but it is good enough for most sane code.
*/
static int reg_val_propagate(struct jit_ctx *ctx)
{
const struct bpf_prog *prog = ctx->skf;
u64 exit_rvt;
int reg;
int i;
/*
* 11 registers * 3 bits/reg leaves top bits free for other
* uses. Bit-62..63 used to see if we have visited an insn.
*/
exit_rvt = 0;
/* Upon entry, argument registers are 64-bit. */
for (reg = BPF_REG_1; reg <= BPF_REG_5; reg++)
set_reg_val_type(&exit_rvt, reg, REG_64BIT);
/*
* First follow all conditional branches on the fall-through
* edge of control flow..
*/
reg_val_propagate_range(ctx, exit_rvt, 0, false);
restart_search:
/*
* Then repeatedly find the first conditional branch where
* both edges of control flow have not been taken, and follow
* the branch taken edge. We will end up restarting the
* search once per conditional branch insn.
*/
for (i = 0; i < prog->len; i++) {
u64 rvt = ctx->reg_val_types[i];
if ((rvt & RVT_VISITED_MASK) == RVT_DONE ||
(rvt & RVT_VISITED_MASK) == 0)
continue;
if ((rvt & RVT_VISITED_MASK) == RVT_FALL_THROUGH) {
reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, true);
} else { /* RVT_BRANCH_TAKEN */
WARN(1, "Unexpected RVT_BRANCH_TAKEN case.\n");
reg_val_propagate_range(ctx, rvt & ~RVT_VISITED_MASK, i, false);
}
goto restart_search;
}
/*
* Eventually all conditional branches have been followed on
* both branches and we are done. Any insn that has not been
* visited at this point is dead.
*/
return 0;
}
static void jit_fill_hole(void *area, unsigned int size)
{
u32 *p;
/* We are guaranteed to have aligned memory. */
for (p = area; size >= sizeof(u32); size -= sizeof(u32))
uasm_i_break(&p, BRK_BUG); /* Increments p */
}
struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
{
struct bpf_prog *orig_prog = prog;
bool tmp_blinded = false;
struct bpf_prog *tmp;
struct bpf_binary_header *header = NULL;
struct jit_ctx ctx;
unsigned int image_size;
u8 *image_ptr;
if (!prog->jit_requested)
return prog;
tmp = bpf_jit_blind_constants(prog);
/* If blinding was requested and we failed during blinding,
* we must fall back to the interpreter.
*/
if (IS_ERR(tmp))
return orig_prog;
if (tmp != prog) {
tmp_blinded = true;
prog = tmp;
}
memset(&ctx, 0, sizeof(ctx));
preempt_disable();
switch (current_cpu_type()) {
case CPU_CAVIUM_OCTEON:
case CPU_CAVIUM_OCTEON_PLUS:
case CPU_CAVIUM_OCTEON2:
case CPU_CAVIUM_OCTEON3:
ctx.use_bbit_insns = 1;
break;
default:
ctx.use_bbit_insns = 0;
}
preempt_enable();
ctx.offsets = kcalloc(prog->len + 1, sizeof(*ctx.offsets), GFP_KERNEL);
if (ctx.offsets == NULL)
goto out_err;
ctx.reg_val_types = kcalloc(prog->len + 1, sizeof(*ctx.reg_val_types), GFP_KERNEL);
if (ctx.reg_val_types == NULL)
goto out_err;
ctx.skf = prog;
if (reg_val_propagate(&ctx))
goto out_err;
/*
* First pass discovers used resources and instruction offsets
* assuming short branches are used.
*/
if (build_int_body(&ctx))
goto out_err;
/*
* If no calls are made (EBPF_SAVE_RA), then tail call count
* in $v1, else we must save in n$s4.
*/
if (ctx.flags & EBPF_SEEN_TC) {
if (ctx.flags & EBPF_SAVE_RA)
ctx.flags |= EBPF_SAVE_S4;
else
ctx.flags |= EBPF_TCC_IN_V1;
}
/*
* Second pass generates offsets, if any branches are out of
* range a jump-around long sequence is generated, and we have
* to try again from the beginning to generate the new
* offsets. This is done until no additional conversions are
* necessary.
*/
do {
ctx.idx = 0;
ctx.gen_b_offsets = 1;
ctx.long_b_conversion = 0;
if (gen_int_prologue(&ctx))
goto out_err;
if (build_int_body(&ctx))
goto out_err;
if (build_int_epilogue(&ctx, MIPS_R_RA))
goto out_err;
} while (ctx.long_b_conversion);
image_size = 4 * ctx.idx;
header = bpf_jit_binary_alloc(image_size, &image_ptr,
sizeof(u32), jit_fill_hole);
if (header == NULL)
goto out_err;
ctx.target = (u32 *)image_ptr;
/* Third pass generates the code */
ctx.idx = 0;
if (gen_int_prologue(&ctx))
goto out_err;
if (build_int_body(&ctx))
goto out_err;
if (build_int_epilogue(&ctx, MIPS_R_RA))
goto out_err;
/* Update the icache */
flush_icache_range((unsigned long)ctx.target,
(unsigned long)&ctx.target[ctx.idx]);
if (bpf_jit_enable > 1)
/* Dump JIT code */
bpf_jit_dump(prog->len, image_size, 2, ctx.target);
bpf_jit_binary_lock_ro(header);
prog->bpf_func = (void *)ctx.target;
prog->jited = 1;
prog->jited_len = image_size;
out_normal:
if (tmp_blinded)
bpf_jit_prog_release_other(prog, prog == orig_prog ?
tmp : orig_prog);
kfree(ctx.offsets);
kfree(ctx.reg_val_types);
return prog;
out_err:
prog = orig_prog;
if (header)
bpf_jit_binary_free(header);
goto out_normal;
}
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7