Packaging eBPF Programs in a Linux Distribution

# Packaging eBPF Programs in a Linux Distribution

## Challenges & Solutions

.logos[
<img src="assets/fedora-logo.svg" alt="Fedora" class="fedora-logo">
<img src="assets/ebpf-logo.png" alt="eBPF" class="ebpf-logo">
]

---

# About This Talk

1. **The Hook** — eBPF is everywhere
2. **The Headache** — Why packaging fails
3. **The Solution** — CO-RE to the rescue
4. **Real World** — Fedora patterns
5. **Edge Cases** — The gotchas
6. **The Checklist** — Your takeaway
]

.right-column[
.arch-diagram[
```
    ┌─────────────────────┐
    │   Your eBPF Tool    │
    │   (RPM Package)     │
    └──────────┬──────────┘
               │
    ┌──────────▼──────────┐
    │    libbpf.so        │
    │  (System Library)   │
    └──────────┬──────────┘
               │ CO-RE Magic
    ┌──────────▼──────────┐
    │   Linux Kernel      │
    │   /sys/kernel/btf   │
    └─────────────────────┘
```
]
]

???

**Walk through the narrative arc briefly.**

"We'll start with WHY this is hard, look at HOW we're solving it in Fedora, and end with a practical checklist you can take home."

**Point to the ASCII diagram:**

"This is the stack we're dealing with — your tool, libbpf, and the kernel. The magic happens in that middle layer."

---

# The Hook

## eBPF is here — and it's a packaging nightmare

???

**Pause briefly. Let the audience read.**

---

# eBPF: No Longer Optional

.columns.col-2[
.left-column[
### It's Everywhere Now

- **systemd** — oomd, cgroup tracking
- **Networking** — Cilium, Calico, XDP
- **Security** — Falco, Tetragon, LSM
- **Observability** — bpftrace, Pixie
- **Storage** — io_uring integration
]

<br/>

???

**Key message: This isn't niche anymore.**

"Five years ago, eBPF was a tool for kernel developers and observability geeks. Today? It's in systemd. It's in your firewall. If Cilium is your CNI, it's how your Kubernetes pods talk to each other."

**Point to the infinity symbol:**

"The use cases are exploding. And that means we — as distribution maintainers — can no longer ignore it."

---

# The Packaging Paradigm

.flow-diagram[
.flow-box[Source] .flow-arrow[→] .flow-box[Compile] .flow-arrow[→] .flow-box[Binary] .flow-arrow[→] .flow-box.highlight[Ship ✅]
]

### eBPF Software

.flow-diagram[
.flow-box[Source] .flow-arrow[→] .flow-box[Compile] .flow-arrow[→] .flow-box.orange[???] .flow-arrow[→] .flow-box[Ship ❓]
]

.warning-box[
**Dependencies:** The KERNEL ITSELF
**Works on:** Maybe? Depends on kernel version...
]
]

.right-40[
<br/><br/><br/>
.quote-box[
**The kernel is our "library"** — but it has no stable ABI for eBPF programs.
]
]

???

**Build the contrast using the flow diagrams:**

"Look at traditional software — the top diagram. Source, compile, binary, ship. Green checkmark. Dependencies are libc, libssl — stable stuff."

"Now look at eBPF below it. Source, compile... question marks. Orange warning box. Our 'library' is the kernel itself. And the kernel has no stable ABI for eBPF programs."

---

# The Key Insight

.center[
.quote-box[
"We are essentially packaging **runtime-injected kernel modules**,
but expecting them to behave like **userspace applications**."
]
]

<br/>

.card-grid[
.card[
### 📦 Users Expect
`dnf install tool` → it works
Survives kernel updates
Respects SELinux
]
.card[
### 💥 Reality
Kernel struct layouts change
BTF version mismatches
Capability and policy issues
]
]

???

**This is your memorable quote. Pause and emphasize it:**

"We are essentially packaging runtime-injected kernel modules, but expecting them to behave like userspace applications."

**Let that sink in.**

"This is the fundamental tension we're dealing with."

---

# The Stakes Are High

- System updates break observability
- Security enforcement stops silently
- Networking tools fail mysteriously
- Users lose trust in distros
]

- eBPF becomes "boring" infrastructure
- Install once, works forever
- Kernel updates are seamless
- Distros become eBPF-ready
]
]

???

**Two sides of the coin:**

"Get this wrong: kernel update breaks your observability, security tools fail silently, users blame the distro."

"Get it right: eBPF becomes boring infrastructure. Install, forget, it just works."

**Transition:**

"So what exactly makes this so hard? Let's look at the headaches."

---

# The Headache

## Why Traditional Packaging Fails eBPF

???

**Pause briefly. Let the audience read.**

---

# Headache #1: The BCC Trap

### The "Old Way" — Compile at Runtime

- Ships C source code
- Compiles **on target machine**
- Requires: `clang`, `llvm`, `kernel-headers`
- If headers ≠ running kernel: 💥
]

- Pre-compiled binaries
- Minimal runtime dependencies
- Works across kernel versions
- No build tools on production
]
]

???

**Explain the old world:**

"BCC — the Berkeley eBPF Compiler Collection — was groundbreaking. But it has a fundamental problem for distros: it compiles C code AT RUNTIME on the target machine."

"That means you need clang, LLVM, and kernel headers on your production server. Hundreds of megabytes of build tools just to run a simple tracing tool."

"And if those headers don't match the running kernel EXACTLY? It crashes."

---

# Headache #2: The ABI Problem

### Kernel structures change between versions

```c
struct task_struct {
    // Kernel 5.15                    // Kernel 6.8
    volatile long state;              // removed!
    void *stack;                      void *stack;
    // ...                            // ...
    pid_t pid;  // offset: 16?        pid_t pid;  // offset: 24?
};
```

.warning-box[
A program compiled for kernel 5.15 reading `pid` at offset 16 will **read garbage** on kernel 6.8 where `pid` is at offset 24.
]

???

**Walk through the code example:**

"This is task_struct — the kernel's representation of a process. Notice the `pid` field."

"Between kernel 5.15 and 6.8, fields were added, removed, reordered. The `pid` field moved."

"If your eBPF program was compiled to read `pid` at offset 16, and the kernel has it at offset 24? You're reading garbage."

---

# Headache #3: The Tooling Matrix

### It's not one version — it's a compatibility matrix

| Component | Version Matters | Breaks When... |
|-----------|----------------|----------------|
| **Application** | ✅ | API changes |
| **libbpf** | ✅ | Loader logic, CO-RE support |
| **bpftool** | ✅ | Skeleton format changes |
| **Clang/LLVM** | ✅ | Bytecode generation |
| **Kernel** | ✅ | BTF format, BPF features |

???

"It's not one version you're managing. It's a matrix. Application version, libbpf version, bpftool version, clang version, kernel version."

"Any of these can break compatibility with any other."

**Transition:**

"This sounds like a nightmare. Fortunately, there's a solution, and it's called CO-RE."
---

# The Solution

## CO-RE and the Fedora Approach

???

**Pause briefly. Let the audience read.**

---

# CO-RE: Compile Once, Run Everywhere

.flow-diagram.centered[
.flow-box[Source] .flow-arrow[→] .flow-box[Clang + BTF] .flow-arrow[→] .flow-box[.bpf.o] .flow-arrow[→] .flow-box.highlight[libbpf] .flow-arrow[→] .flow-box.orange[Kernel]
]

.solution[
**libbpf** reads kernel BTF at load time and rewrites field offsets.
One binary works across kernel versions!
]

.footnote[Reference: [nakryiko.com/posts/bpf-portability-and-co-re](https://nakryiko.com/posts/bpf-portability-and-co-re/)]

???

**The key concept:**

"CO-RE is to eBPF what dynamic linking is to userspace. Instead of hardcoding offsets, we record WHAT we need — 'give me the pid field from task_struct' — and libbpf figures out WHERE it is at load time."

**Walk through the flow diagram:**

"Source code, clang compiles it with BTF metadata, we get a .bpf.o file with relocations, libbpf patches the offsets at load time, program runs in kernel."

---

# How CO-RE Actually Works

```c
// Your code uses CO-RE macros
pid_t p = BPF_CORE_READ(task, pid);
```

**Clang records:**
- "Need field `pid` from `task_struct`"
- NOT a hardcoded offset!
- Stored as BTF relocation
]

.info-box[
1. libbpf reads program BTF
2. libbpf reads `/sys/kernel/btf/vmlinux`
3. Finds `task_struct` in kernel BTF
4. Looks up `pid` field → offset 24
5. Patches bytecode instruction
6. Loads patched program into kernel
]
]

.solution[
**Result:** Same `.bpf.o` file works on Fedora 39, 40, 41, RHEL 9, any BTF-enabled kernel!
]

???

**Two columns — explain both:**

"At compile time, you write BPF_CORE_READ. Clang doesn't hardcode an offset. It records 'I need the pid field from task_struct.'"

"At load time, libbpf reads your program's BTF, reads the kernel's BTF from /sys/kernel/btf/vmlinux, finds the actual offset, patches your bytecode, then loads it."

**Result:**

"Same binary file works on Fedora 39, 40, 41, RHEL 9, any BTF-enabled kernel."

---

# The CO-RE Magic — Visualized

.core-diagram[
<div class="core-step">
<div class="core-box source">
<strong>YOUR SOURCE CODE</strong><br/>
<code>pid_t pid = BPF_CORE_READ(task, pid);</code>
</div>
<div class="core-arrow">↓ clang -target bpf</div>
</div>
<div class="core-step">
<div class="core-box compiled">
<strong>COMPILED .bpf.o FILE</strong><br/>
Bytecode: "read 4 bytes from R1 + <em>OFFSET</em>"<br/>
BTF: "OFFSET = field 'pid' in 'task_struct'"
</div>
<div class="core-arrow">↓ libbpf loads</div>
</div>
<div class="core-step core-merge">
<div class="core-box btf">Program BTF<br/>"need task→pid"</div>
<div class="core-arrow-h">→</div>
<div class="core-box btf kernel">Kernel BTF<br/>"pid is at +24"</div>
</div>
<div class="core-step">
<div class="core-arrow">↓</div>
<div class="core-box success">
<strong>PATCHED & LOADED</strong><br/>
Bytecode: "read 4 bytes from R1 + <strong>24</strong>" ✓
</div>
</div>
]

???

**Walk through the diagram (styled boxes showing the flow):**

"Look at the flow: Your source code at the top uses BPF_CORE_READ. Clang compiles it and records BTF metadata — 'I need the pid field'."

"At load time, libbpf takes your program's BTF and matches it against the kernel's BTF. The kernel says 'pid is at offset 24'. libbpf patches the bytecode. Done."

"The green box at the bottom shows success — the correct offset is now in the bytecode."

---

# Build vs Runtime Dependencies

### The Critical Separation

| Phase | Dependencies | In Package? |
|-------|--------------|-------------|
| **Build** | clang, llvm, bpftool, libbpf-devel | ❌ BuildRequires |
| **Runtime** | libbpf.so only | ✅ Requires |

<br/>

If a CVE is found in libbpf's loader, we patch `libbpf.so` **once** and ALL eBPF tools are fixed — bpftrace, your custom tools, everything.
]

???

**Critical point:**

"Build time: you need the full toolchain. Runtime: you ONLY need libbpf."

**Policy point:**

"In Fedora, we dynamically link against system libbpf. Never bundle it. Why? If there's a CVE in libbpf's loader, we patch libbpf ONCE and all eBPF tools are fixed."

---

# Fedora's BTF Foundation

### The Infrastructure Is Ready

```bash
# Every Fedora kernel ships BTF (since Fedora 30!)
$ ls -la /sys/kernel/btf/vmlinux
-r--r--r--. 1 root root 5765067 Jan 15 10:00 /sys/kernel/btf/vmlinux

# Inspect it with bpftool
$ bpftool btf dump file /sys/kernel/btf/vmlinux | head -5
[1] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64
[2] CONST '(anon)' type_id=1
[3] ARRAY '(anon)' type_id=1 index_type_id=18 nr_elems=2
```

.footnote[Fedora Change: [fedoraproject.org/wiki/Changes/BTF](https://fedoraproject.org/wiki/Changes/BTF)]

???

**Good news:**

"Fedora has shipped BTF-enabled kernels since Fedora 30 — that's 2019. Every kernel has /sys/kernel/btf/vmlinux. CO-RE just works out of the box."

---

# The vmlinux.h Challenge

- Ship a static `vmlinux.h` in tarball
- Use `/sys/kernel/btf/vmlinux`

*Build host kernel ≠ target distro kernel!*
]

Use `kernel-core` (image) & `kernel-devel` (scripts):
```bash
BuildRequires: kernel-core, kernel-devel

# In %build:
KVER=$(rpm -q --qf '%%{VERSION}-%%{RELEASE}.%%{ARCH}' \
  kernel-core | head -1)

# Extract vmlinuz -> vmlinux & dump BTF
/usr/src/kernels/$KVER/scripts/extract-vmlinux \
  /lib/modules/$KVER/vmlinuz > vmlinux_local
bpftool btf dump file vmlinux_local format c > vmlinux.h
```
]
]

.info-box[
**Why this works:** The `kernel-devel` package provides the BTF for the *target* kernel, not the build host's running kernel.
]

???

**Common mistake:**

"You need vmlinux.h to build CO-RE programs. The WRONG approach is shipping a static vmlinux.h in your source tarball."

"The RIGHT approach is generating it at build time. But wait — we don't just dump it from the running kernel. We extract it from the **kernel-core** package image to be safe."

---

# The Build Environment Trap

.warning-box[
In Koji/mock, `/sys` reflects the **build node's kernel**, not the target distro's kernel!
]

- Build node: **RHEL 9** (Kernel 5.14)
- Target: **Fedora Rawhide** (Kernel 6.10)
- Code uses `struct sched_ext_ops` added in 6.0
- `vmlinux.h` from 5.14 → **missing definition**
- ** Clang fails!** 💥
]

```bash
BuildRequires: kernel-core, kernel-devel

# 1. Get exact kernel version from package
KVER=$(rpm -q --qf '%%{VERSION}-%%{RELEASE}.%%{ARCH}' \
  kernel-core | head -1)

# 2. Extract the compressed kernel image
/usr/src/kernels/$KVER/scripts/extract-vmlinux \
  /lib/modules/$KVER/vmlinuz > vmlinux_local

# 3. Generate header from UNCOMPRESSED image
bpftool btf dump file vmlinux_local format c > vmlinux.h
```
]
]

.info-box[
**CO-RE makes this safe:** A `vmlinux.h` from a newer kernel is safe as long as your program only uses fields that exist on the target kernel - CO-RE will relocate offsets, but it cannot invent missing types.
]

???

**Explain the build trap:**

"In Koji/mock, /sys reflects the BUILD NODE's kernel, not the target distro's kernel!"

"If your build node runs RHEL 9 (5.14), but you're building for Fedora Rawhide (6.10), clang will fail if you use the host headers."

"Solution: We pull the compressed `vmlinuz` from the target `kernel-core` package, use the extraction script from `kernel-devel`, and generate our own fresh BTF. It's bulletproof."

---

# Real World

## Fedora Packaging Patterns

???

**Pause briefly. Let the audience read.**

---

# Real Example: "Pure CO-RE"

### Meet `process-snoop`

We want to build a tool that logs every command executed on the system.

.columns.col-2[
.left-column[
#### ❌ The "Scripting" Way
*(bpftrace, python-bcc)*

- Requires **Clang/LLVM** at runtime
- Compiles on the fly
- Heavy footprint (hundreds of MBs)
- Fragile if headers are missing
]

- **0 Runtime Dependencies** (except glibc/libbpf)
- Tiny binary (~20KB)
- Pre-compiled BPF bytecode
- Uses **CO-RE** to adapt to kernels
]
]

.info-box[
**Goal:** Build on the build server, run on *any* target kernel. Code available at: https://github.com/mikelolasagasti/fosdem-process-snoop
]

???

**Introduce the new demo-app:**

"Let's look at `process-snoop`. This is the 'Distro Way'."

"Unlike `bpftrace` or python scripts, which drag in a 300MB LLVM dependency to compile at runtime, this tool has **zero** heavy runtime dependencies."

"It ships as a tiny, pre-compiled binary. It uses libbpf to load, and CO-RE to adapt. This is what we want to see in Fedora."

---

# The Code: Kernel Side - agent.bpf.c

```c
#include "vmlinux.h"  // <--- Generated from kernel-core!
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} rb SEC(".maps");

SEC("tp/syscalls/sys_enter_execve")
int handle_exec(void *ctx) {
    struct event *e;

// 1. Reserve space in Ring Buffer
    e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);
    if (!e) return 0;

// 2. Capture data
    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

// 3. Send to userspace
    bpf_ringbuf_submit(e, 0);
    return 0;
}

```

???

**Point to the first line:**

"The magic starts at line 1: `#include vmlinux.h`."

"This file doesn't exist in the source repo. We generate it during the RPM build. This allows our code to use kernel types (`task_struct`, etc.) without needing kernel headers installed on the target machine."

---

# The Packaging: "Gold Standard"

### `process-snoop.spec`

```spec
Name:       process-snoop
# 1. BUILD with full toolchain
BuildRequires: clang, llvm, bpftool, libbpf-devel
# 2. Kernel source (Core for Image, Devel for Scripts)
BuildRequires: kernel-core, kernel-devel
# 3. RUNTIME: Only the library! No Clang!
Requires:   libbpf >= 1.3

%build
# Query installed kernel-core and extract vmlinuz
KVER=$(rpm -q --qf '%%{V}-%%{R}.%%{A}' kernel-core | head -1)
/usr/src/kernels/$KVER/scripts/extract-vmlinux \
    /lib/modules/$KVER/vmlinuz > vmlinux_local
# Generate vmlinux.h & Compile
bpftool btf dump file vmlinux_local format c > vmlinux.h
clang -g -O2 -target bpf -c agent.bpf.c -o agent.bpf.o
bpftool gen skeleton agent.bpf.o > agent.skel.h
gcc -O2 -g main.c -o process-snoop -lbpf -lelf -lz
```

???

**1. The Dependencies (BuildRequires)**
"Start at the top. We need the full toolchain (`clang`, `llvm`), but crucially we need **kernel inputs**:
* `kernel-core`: Provides the actual compressed kernel image.
* `kernel-devel`: Provides the scripts to decompress it."

**2. The Build Pipeline (%build)**
"The `%build` section is a pipeline. Let's trace it line-by-line:
* **Get Version:** We query `rpm` to find the exact kernel version installed in the build root.
* **Decompress:** We run `extract-vmlinux` to turn the compressed `vmlinuz` into a raw ELF file `bpftool` can read.
* **Gen Headers:** We dump the BTF from that raw file to create `vmlinux.h`. This gives us the target kernel's memory layout.
* **Compile BPF:** We use `clang -target bpf` to build the bytecode.
* **Gen Skeleton:** `bpftool` generates a C header (`agent.skel.h`) so our userspace code can easily load the BPF program.
* **Compile App:** Finally, `gcc` builds the userspace loader, linking against `libbpf`."

**3. The Payoff (Runtime)**
"Now look at the `Requires`. Just `libbpf`. No `clang`, no python, no kernel headers required on the user's machine. This is the goal: a clean, tiny, standalone package."

---

# Systemd Integration

### The Complete Unit File Pattern

```ini
[Unit]
Description=My eBPF Observability Tool
After=network.target sys-fs-bpf.mount
Requires=sys-fs-bpf.mount
[Service]
Type=notify
ExecStart=/usr/bin/my-ebpf-tool
Restart=on-failure
RestartSec=5
# === SECURITY HARDENING ===
# Drop CAP_NET_ADMIN if not using XDP / TC
CapabilityBoundingSet=CAP_BPF CAP_PERFMON CAP_NET_ADMIN
AmbientCapabilities=CAP_BPF CAP_PERFMON CAP_NET_ADMIN
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/sys/fs/bpf/%N
[Install]
WantedBy=multi-user.target
```

???

**Walk through the unit file:**

"After sys-fs-bpf.mount — you need bpffs mounted. Type notify for proper readiness signaling."

"The security section is critical: CapabilityBoundingSet, AmbientCapabilities, NoNewPrivileges, ProtectSystem."

---

# Reproducible Builds

### The Challenges

.left-column[
**What affects reproducibility:**
- Clang/LLVM version
- libbpf version at build
- vmlinux.h source
- Build flags
- Timestamps in BTF
]

```spec
# don't Pin versions explicitly
# BuildRequires: clang = 17.0.6
# BuildRequires: libbpf-devel = 1.3.0

# Avoid strict pinning (Fedora Policy)
BuildRequires: clang >= 17
BuildRequires: libbpf-devel >= 1.3

# Consistent build flags
%global ebpf_cflags -O2 -g \
  -target bpf \
  -D__TARGET_ARCH_%{_target_cpu}
```
]

.info-box[
Test builds across Fedora versions in CI. Same source should produce functionally equivalent bytecode.
]

???

"Reproducibility is hard with BPF because Clang versions change the bytecode generation."

"While you might be tempted to pin specific Clang versions (like `clang = 17.0.6`), Fedora policy actually discourages this because it blocks OS upgrades."

"Instead, use loose dependencies (`clang >= 17`) and rely on your CI to detect if a new compiler version breaks your BPF object."

---

# Edge Cases

## The Gotchas That Keep Packagers Up at Night

???

**Pause briefly. Let the audience read.**

---

# Edge Case #1: Pinned Maps & Upgrades

### The Scenario

```bash
# Your tool pins persistent state to bpffs
$ ls /sys/fs/bpf/my-firewall/
connection_table    # 50,000 active connections
rule_map            # Firewall rules
```

### The Problem: `dnf update my-ebpf-tool`

.flow-diagram.centered[
.flow-box[Old service stops] .flow-arrow[→] .flow-box.orange[Maps still pinned!] .flow-arrow[→] .flow-box[New service starts] .flow-arrow[→] .flow-box[???]
]

.warning-box[
Does the new binary reuse the old map? Is the schema compatible? What if a field was added?
]

???

**The scenario:**

"Your tool pins a map to bpffs — maybe firewall rules, connection state. User runs dnf update. Your service restarts."

**The problem:**

"Does the new binary reuse the old map? What if the schema changed?"

---

# Pinned Maps: The Solution

### This is an Upstream Requirement

```c
// Safe map handling pattern
int map_fd = bpf_obj_get("/sys/fs/bpf/my-app/my_map");

if (map_fd >= 0) {
    // Map exists - validate and reuse
    struct bpf_map_info info = {};
    __u32 len = sizeof(info);
    bpf_obj_get_info_by_fd(map_fd, &info, &len);

if (info.value_size == expected_size) {
        reuse_existing_map(map_fd);  // ✅ Adopt existing data
    } else {
        migrate_map_data(map_fd);    // 🔄 Schema migration
    }
} else {
    create_and_pin_new_map();        // 🆕 Fresh start
}
```

???

**The solution:**

"This is NOT an RPM problem. The upstream code must handle existing maps. The packager's job is to VERIFY the upstream does this correctly."

"Show the code pattern: check if map exists, validate schema, migrate if needed, or create fresh."

---

# Edge Case #2: Privilege Models

### The Capability Matrix

| Capability | What It Allows |
|------------|----------------|
| `CAP_BPF` | Load BPF programs, create maps |
| `CAP_PERFMON` | Attach to perf events, kprobes, uprobes |
| `CAP_NET_ADMIN` | XDP, TC, socket filters |
| .strike[`CAP_SYS_ADMIN`] | .strike[Legacy — avoid!] |

<br/>

.info-box[
**Fedora default:** `kernel.unprivileged_bpf_disabled = 2`
Unprivileged users cannot load BPF programs.
]

???

"Don't use CAP_SYS_ADMIN. Use the specific capabilities: CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN."

"Fedora default is unprivileged_bpf_disabled=2 — unprivileged users cannot load BPF programs."

---

# Privilege Management Strategies

.card-grid[
.card[
### 🔧 Systemd Service
```ini
CapabilityBoundingSet=CAP_BPF
AmbientCapabilities=CAP_BPF
NoNewPrivileges=yes
```
*Recommended for daemons*
]
.card[
### 🚀 Boot-time Loader
Load eBPF at boot with privileges, then drop them. Unprivileged daemon uses pinned maps.

*For security-sensitive tools*
]
.card[
### 👤 File Capabilities
```bash
setcap cap_bpf,cap_perfmon=ep \
  /usr/bin/my-tool
```
*For CLI tools*
]
.card[
### 🔐 Polkit Integration
Interactive privilege escalation for desktop tools.

*For user-facing apps*
]
]

???

**Four strategies:**

"Systemd capabilities for daemons, boot-time loader for security tools, file capabilities for CLI tools, polkit for desktop apps."

---

# Edge Case #3: SELinux

### Fedora Runs in Enforcing Mode

```bash
$ ausearch -m avc -ts recent | grep bpf
type=AVC msg=audit(...): avc:  denied  { prog_load } for  pid=1234
  comm="my-tool" scontext=system_u:system_r:my_tool_t:s0
  tcontext=system_u:system_r:my_tool_t:s0 tclass=bpf permissive=0
```

.warning-box[
If your tool runs in a confined domain, BPF operations will be **denied by default**.
]

???

"Fedora runs in enforcing mode. BPF operations WILL be denied for confined domains."

"Check ausearch for AVC denials related to bpf — that's how you know SELinux is blocking your tool."

---

# SELinux Policy Module

### Ship a Companion Policy

```te
# my-ebpf-tool.te
policy_module(my_ebpf_tool, 1.0)

require {
    type my_tool_t;
    type bpf_t;
    type bpffs_t;
}

# Allow BPF operations
allow my_tool_t self:bpf { map_create map_read map_write
                            prog_load prog_run };

# Allow access to bpffs
allow my_tool_t bpffs_t:dir { add_name create read write };
allow my_tool_t bpffs_t:file { create read write unlink };
```

???

**Show the policy snippet:**

"Allow bpf operations on self, allow access to bpffs. Package the compiled .pp and install in post."

"You'll probably need to ship a companion SELinux policy module for confined domains."

---

# Edge Case #4: Kernel Feature Detection

### Not All Kernels Have All Features

```c
// Runtime feature detection
static bool has_ringbuf_support(void) {
    int fd = bpf_create_map(BPF_MAP_TYPE_RINGBUF, 0, 0, 4096, 0);
    if (fd >= 0) {
        close(fd);
        return true;
    }
    return false;
}

// Graceful degradation
if (has_ringbuf_support()) {
    use_ringbuf();      // Kernel 5.8+
} else {
    use_perf_buffer();  // Fallback
}
```

.info-box[
Document minimum kernel version. Use `Requires: kernel >= 6.1` when features are mandatory.
]

???

"Not all kernels have all features. Ringbuf is 5.8+. Some features are even newer."

"Implement runtime detection and graceful degradation. The code example shows how to probe for ringbuf support and fall back to perf_buffer."

---

# Edge Case #5: Kernel Updates

### What Happens When the Kernel Updates?

| Scenario | Outcome |
|----------|---------|
| CO-RE handles layout change | ✅ Transparent, just works |
| Missing BTF type | ⚠️ Program fails to load |
| Removed kernel feature | ⚠️ Load fails, needs code change |
| Map schema incompatible | ❌ Data loss possible |

<br/>

.solution[
**Mitigation:**
- Test against multiple kernel versions in CI
- Implement graceful degradation
- Use CO-RE optional features (`bpf_core_field_exists`)
]

???

"What happens when the kernel updates? CO-RE handles most layout changes transparently. But missing BTF types or removed features can still break things."

"Mitigation: test against multiple kernels, implement graceful degradation, use bpf_core_field_exists for optional features."

---

# The Checklist

## What Every Packager Needs to Know

???

**Pause briefly. Let the audience read.**

---

# The Packager's Checklist

.columns.col-2[
.left-column[
### Before Packaging

<ul class="checklist">
<li>Upstream supports CO-RE?<br/><span class="small">If not: reject or patch</span></li>
<li>libbpf dynamically linked?<br/><span class="small">No bundled/vendored copies</span></li>
<li>GPL-compatible license?<br/><span class="small">Kernel verifier enforces this!</span></li>
<li>vmlinux.h generated at build?<br/><span class="small">Not shipped in tarball</span></li>
</ul>
]

<ul class="checklist">
<li>Minimal capabilities in unit<br/><span class="small">CAP_BPF, not CAP_SYS_ADMIN</span></li>
<li>SELinux policy if needed<br/><span class="small">Test in enforcing mode!</span></li>
<li>Kernel requirements documented<br/><span class="small">Min version, required features</span></li>
<li>Pinned map handling verified<br/><span class="small">Upgrades don't lose state</span></li>
<li>Multi-kernel CI testing<br/><span class="small">Test F40, F41, Rawhide or EPEL</span></li>
</ul>
]
]

???

**Walk through each checkbox:**

"Before packaging: Does upstream support CO-RE? If not, reject or help them migrate. Is libbpf dynamically linked? GPL-compatible license? vmlinux.h generated at build?"

"In your package: Minimal capabilities, SELinux policy if needed, kernel requirements documented, pinned map handling verified, multi-kernel CI."

---

# Quick Reference Card

```spec
BuildRequires: clang, llvm, bpftool, libbpf-devel
BuildRequires: kernel-core, kernel-devel

# Runtime
Requires: libbpf >= 1.3

%build
KVER=$(rpm -q --qf '%%{V}-%%{R}.%%{A}' kernel-core | head -1)
# Extract compressed kernel
/usr/src/kernels/$KVER/scripts/extract-vmlinux \
  /lib/modules/$KVER/vmlinuz > vmlinux_local
# Generate header
bpftool btf dump file vmlinux_local format c > vmlinux.h

%files
# Pro Tip: Allow non-root usage!
%caps(cap_bpf,cap_perfmon=ep) %{_bindir}/my-tool
```
]

```ini
[Service]
ExecStart=/usr/bin/my-tool

# Minimal privileges
CapabilityBoundingSet=CAP_BPF CAP_PERFMON
AmbientCapabilities=CAP_BPF CAP_PERFMON
NoNewPrivileges=yes

# Filesystem protection
ProtectSystem=strict
ReadWritePaths=/sys/fs/bpf
```
]

???

"This is your screenshot slide. The essentials for spec file and systemd unit."

"**Pro Tip:** Look at the `%caps` line in the spec file. This allows users to run your tool *without* sudo, by granting just the `CAP_BPF` capability to the binary file itself. Much safer than running as root!"

---

# Future: Packaging Macros

### What We're Working On in the eBPF SIG

```spec
# Proposed Fedora macros for eBPF packages

%ebpf_build
# - Generates vmlinux.h
# - Sets correct CFLAGS
# - Builds .bpf.o files

%ebpf_install
# - Installs .bpf.o to correct location
# - Sets up bpffs directories

%ebpf_selinux my_tool
# - Compiles and installs SELinux policy
```

???

"We're working on Fedora macros to make this easier. %ebpf_build, %ebpf_install, %ebpf_selinux. Help wanted!"

---
# Future: BPFToken support in Fedora

We're also proposiing a change for Fedora 45 about having a default `ebpf` group
which would allow non root and much more permission granularity for bpf
programs.

More info about this feature and propsal was presented in yesterday session at
https://fosdem.org/2026/schedule/event/3LLHG9-bpf-tokens-safe-userspace-ebpf/
---
# Join the Fedora eBPF SIG

.columns.col-2[
.left-column[
### Get Involved

- **Wiki:** [fedoraproject.org/wiki/SIGs/eBPF](https://fedoraproject.org/wiki/SIGs/eBPF)
- **Matrix:** `#ebpf:fedoraproject.org`
- **Mailing List:** `devel@lists.fedoraproject.org`

### We Need Help With

- Packaging guidelines documentation
- Fedora macros for eBPF
- CI/testing patterns
- SELinux policy templates
- Package reviews
]

<br/>

Let's make it **boring**
and **reliable** to install.
]
]
]

???

"Come help us build these standards. Wiki link, Matrix channel, mailing list. We need help with guidelines, macros, CI patterns, policy templates."

**Memorable close:**

"eBPF is the future of Linux internals. Let's make it boring and reliable to install."

---

# Resources

.columns.col-2[
.left-column[
### Essential Reading

- **CO-RE Guide**
  [nakryiko.com/posts/bpf-portability-and-co-re](https://nakryiko.com/posts/bpf-portability-and-co-re/)

- **libbpf**
  [github.com/libbpf/libbpf](https://github.com/libbpf/libbpf)

- **bpftool**
  [github.com/libbpf/bpftool](https://github.com/libbpf/bpftool)

- **bpftrace**
  [github.com/bpftrace/bpftrace](https://github.com/bpftrace/bpftrace)
]

- **eBPF SIG**
  [fedoraproject.org/wiki/SIGs/eBPF](https://fedoraproject.org/wiki/SIGs/eBPF)

- **BTF Support**
  [fedoraproject.org/wiki/Changes/BTF](https://fedoraproject.org/wiki/Changes/BTF)

- **Systemd Guidelines**
  [docs.fedoraproject.org/.../Systemd/](https://docs.fedoraproject.org/en-US/packaging-guidelines/Systemd/)

- **Packaging Guidelines**
  [docs.fedoraproject.org/.../packaging-guidelines/](https://docs.fedoraproject.org/en-US/packaging-guidelines/)
]
]

???

"All these links will be in the published slides. The Andrii Nakryiko blog post on CO-RE is essential reading."

---

# Questions?

<br/>

.contact[
🐦 @dmellado · @mikelolasagasti <br/>
🐙 github.com/danielmellado · github.com/mikelolasagasti
]

<br/>

.small[
Slides: `https://github.com/danielmellado/fosdem26-ebpf-packaging` <br/>
Code: `https://github.com/mikelolasagasti/fosdem-process-snoop`
]

???

**Open for Q&A.**

**Common questions to prepare for:**

- "What about RHEL?" → Yes! RHEL 8.2+ and RHEL 9 ship BTF-enabled kernels. CO-RE works the same way.

- "What if upstream doesn't support CO-RE?" → Help them migrate from BCC to libbpf/CO-RE, or reject the package.

- "Do I need to ship vmlinux.h?" → No! Generate it at build time. Never include it in the source tarball.

- "What about static linking libbpf?" → Fedora policy discourages it. Dynamic linking allows single-point security patches.

- "How do I test across kernels?" → Podman/VMs with different Fedora versions. We're working on CI patterns in the eBPF SIG.

- "What's the GPL requirement about?" → The kernel BPF verifier checks the license. eBPF programs using kernel helpers MUST be GPL-compatible.

---

# Thank You!

.contact[
<br/>
.quote-box[
"We are essentially packaging runtime-injected kernel modules,
but expecting them to behave like userspace applications."
]

<br/>

**FOSDEM 2026** · Distributions Devroom · Brussels
]

???

**Leave this slide up during Q&A.**

**Key quotes to remember:**

1. "We are essentially packaging runtime-injected kernel modules, but expecting them to behave like userspace applications."

2. "CO-RE is to eBPF what dynamic linking is to userspace programs."

3. "If a CVE is found in libbpf's loader, we patch libbpf ONCE and all eBPF tools are fixed."

4. "eBPF is the future of Linux internals. Let's make it boring and reliable to install."