What I Learned at the Container Vulnerability Hiding Workshop at SCaLE 23x

Most teams treat a clean vulnerability scan as a green light. That assumption got dismantled in about forty minutes at SCaLE 23x.

Kyle Quest presented a workshop called "These Are Not the Vulnerabilities You Are Looking For: Hiding Vulnerabilities in Containers." The title is a Star Wars reference, but the content is a genuine attack on how the industry thinks about container security. He opened with a line that stuck with me: "Most people are slaves of the vulnerability scanning tool. Not many look into how it works."

The demo was hard to argue with. He started with a 16GB Mint OS container image. Grype scanned it and found roughly 9,000 vulnerabilities. He then ran it through DockerSlim with the --obfuscate-metadata flag. The resulting image was 114MB. Grype found zero vulnerabilities. Trivy also found zero. Same container, same binaries, same code. Zero.

The trick is not about the vulnerabilities at all. It is about what scanners actually look at.

Diagram showing how removing OS metadata files (os-release, dpkg/status) causes vulnerability scanners to report zero CVEs, while the actual binaries and exploitable code remain unchanged. EDR tools doing behavioral detection at runtime are not affected.

Grype, Trivy, Syft, and similar tools do not inspect the binaries in your image for vulnerabilities. They inspect the metadata about packages. Specifically, they read /etc/os-release, /usr/lib/os-release, and /var/lib/dpkg/status. The pretty_name field in the OS release file tells the scanner what distribution it is looking at, and /var/lib/dpkg/status tells it what packages are installed. Remove those files, and the scanner has nothing to match against its CVE database. It reports zero findings because it cannot identify anything.

DockerSlim's obfuscate flag automates this process. It slims the image and strips the metadata in a single step. But you do not even need a tool. Four Dockerfile lines accomplish the same thing: remove os-release, usr/lib/os-release, dpkg/status, and the dpkg/info directory. Kyle demonstrated this against SNIG, a multimillion-dollar enterprise scanner. It failed on four RUN rm commands.

Syft was a little harder because it also reads package.json to detect Node packages. The fix there is renaming the name variable inside the file that system info tools reference. Syft gets confused and stops reporting packages. He called it making the scanner blind, which is accurate.

Kyle used a useful framing for all of this. Chainguard's minimal images do something similar by removing unnecessary OS packages. Fewer packages mean fewer CVEs in scan results. That is legitimate. What he demonstrated is the same mechanical outcome achieved without actually removing anything dangerous, just by hiding the metadata that scanners depend on.

The counterargument is binary fingerprinting. If you run strings against a binary, you can often extract version information directly. crane export can pull an image's contents for deeper analysis. A scanner built to do this kind of inspection would not be fooled by metadata removal. But no commercial scanner in the room was doing it. Kyle pointed out that nobody had built YARA rules for these patterns. When an audience member asked why, he said: "No one asked for it. It hasn't changed because users never demanded change."

That is the part that should make security teams uncomfortable.

The real answer Kyle kept coming back to is runtime security. EDR tools and platforms like CrowdStrike monitor behavior rather than static metadata. They watch for persistence events, privilege escalation, and anomalous process activity. A container that has had its metadata stripped but is actively exploiting something will still trigger those detections.

There is a container-specific wrinkle though. Containers are rebuilt and redeployed constantly. When a container spins down and back up, the digest changes. Digest changes are how traditional antivirus identifies modified software as potentially malicious. But in a container environment, digest changes happen all the time for completely legitimate reasons. An attacker who knows this can blend exploit behavior into the background noise of normal container churn.

The workshop's actual takeaway is not "here is how to hide vulnerabilities." It is "here is why your scanner output is not what you think it is." The false sense of security Kyle described is real. If you are running a tool through a scanner before deploying it, and that scan comes back clean, you have learned very little about whether the running instance of that container can be exploited. You have learned that the metadata was intact and matched known CVE signatures.

Scanners are useful. Vulnerability tracking matters. But runtime visibility is the part that catches the attacks scanners cannot see, and the threshold for what scanners cannot see is lower than most people assume.

Kyle's GitHub for the workshop: github.com/kcq/scale-23x-hide-vulnerabilities