Linux kernel's extended BPF (eBPF) has become a stand-out open-source tool for monitoring and manipulating cloud-based Linux containers at a deep level and one of eBPF’s biggest fans is Amazon Web Services (AWS).
"At AWS, we love eBPF,” said Jeremy Cowan, AWS developer advocate manager at CloudNativeSecurityCon in Seattle.
What is eBPF?
Why does everyone love eBPF? Because it's so darn useful. But first, here’s what it does.
EBPF allows cloud network architects to run programs in the Linux kernel without changing the kernel source code or adding additional modules. There it works as a lightweight, sandbox virtual machine (VM).
“By allowing sandboxed programs to run within the operating system, eBPF enables developers to create eBPF programs that add capabilities to the operating system at runtime,” Thomas Graf, Isovalent's CTO and co-founder and chair of the eBPF Governing Board, explained, adding that this allows for a wide array of use cases, including next-generation networking, observability, and security functionality.
In the case of AWS Lambda (a compute service that lets you run code without provisioning or managing servers), eBPF is used to create network tunnels and cut Amazon Virtual Private Cloud (VPC) start-up times a thousand-fold.
Within VPC, eBPF is used to observe and tune TCP flow and parameters. EBPF is also used to implement network access control lists and security groups. And, perhaps most promising of all its uses, AWS uses eBPF for its new security program, Amazon GuardDuty.
Ready for GuardDuty
In GuardDuty, eBPF helps continuously monitor AWS instances, container workloads, and storage for potential threats.
Specifically, Muhammad Wasiq, AWS principal security engineer and a lead GuardDuty developer, told Silverlinings that, it monitors process creation, filesystem operations, network connections and DBS requests and responses.
Developers use eBPF rather than other low-level utilities, such as Linux kernel modules, because it can be implemented quickly, and it's safer and easier to install and customize than the alternatives. EBPF also provides runtime protection and detailed data on Linux processes and what their containers and Kubernetes pods are up to.
AWS isn't the only organization that's an eBPF fan. Other popular cloud-native security programs such as Isovalent Falco, and Kubernetes security applications using Cilium Hubble also use eBPF for similar purposes.
Amazon GuardDuty primarily uses eBPF for threat detection today, but in the future, AWS anticipates it preventing attacks.
“As the popularity of Kubernetes has grown, so has its appeal as a target. In an increasingly hostile environment, the ability to quickly flag suspicious behaviors and investigate and identify their source is becoming crucial,” said Cowan. So, as far as AWS is concerned, "AWS puts eBPF above other options,” he said.
It does this, for example, Wasiq explained by "using call tracing to capture the input arguments of the system calls so that you could figure out what an attacker is trying to do." Then, it "captures the details of the actor process. Finally, you can capture the return value of the system call so you could figure out if the system call was successful or it returned an error,” he said.
Cloud network architects and engineers can combine all the probes' data across systems for further processing and analysis. The result delivers a detailed, low-level view of what's going on with a potential attack from multiple vantage points.
To make GuardDuty more useful, Wasiq continued, GuardDuty also collects container-level details. "In other words, if a detection originated from inside a container, it will have that container's details. GuardDuty also collects existing flow logs and DNS logs to provide host-level details.”
Wasiq added, "Our strategy is to collect as much information or data from inside the kernel as much as possible because that's more efficient and it's also safer."
Needles in haystacks
GuardDuty is a very attractive option for threat detection because it can capture events from the kernel, said Cowan. When that data is enriched to provide additional context, it's really good for threat detection applications, such as GuardDuty. “It can be used to find the proverbial needles in the haystack that allow you to focus on the root cause of the securities,” he said.
All that said, GuardDuty isn't the easiest tool to work with. As Wasiq said, "Its tooling is immature, debugging eBPF programs is hard, and it does have some Linux kernel compatibility issues.
Ultimately, GuardDuty is far from being only an AWS approach. Many other open-source security programs based around, eBPF use a similar approach.
As Lucas Severo Alves, a container solutions' cloud-native engineer, put it, "While this may seem an exaggeration, eBPF can be the best solution to diagnose and solve multiple [low-level] problems. Or, as there are a lot of needles out there in many cloud haystacks, and eBPF can help, regardless of your preferred cloud platform."