--- id: mohitmishra786/low-level-dev-skills/containers-internals version: "abe27014" license: MIT install: manual updated: 2026-06-27 --- # containers-internals — Containers Internals covers the low-level mechanisms that power Docker and Podman: process namespaces for isolation, cgroups v2 for resource limits, overlayfs for layered storage, and seccomp-bpf for syscall filtering. Explore runc, the OCI spec, Linux capabilities, rootless user namespaces, and practical escape mitigations. Publisher: mohitmishra786 · Stars: 148 · Updated: 2026-06-27 Install (manual): `git clone https://github.com/mohitmishra786/low-level-dev-skills` ## SKILL.md # Containers Internals ## Purpose Guide agents through Linux container internals: namespaces (`clone`, `unshare`, `nsenter`), cgroups v2 resource limits, overlayfs storage, runc and the OCI runtime spec, seccomp-bpf filtering, Linux capabilities for privilege dropping, and container escape mitigations. ## When to Use - Understanding how Docker/Podman isolate processes under the hood - Debugging container resource limits (OOM, CPU throttling) - Writing custom seccomp profiles for sandboxed workloads - Building minimal containers without Docker - Investigating container escape vulnerabilities - Tuning cgroups v2 for Kubernetes pods ## Workflow ### 1. Namespaces ```bash # List namespaces for a process ls -la /proc/self/ns/ readlink /proc/self/ns/pid readlink /proc/1234/ns/net # Enter container namespaces nsenter -t -m -u -i -n -p bash # Unshare namespaces (manual container) unshare --fork --mount-proc --pid --net --uts --ipc bash ``` | Namespace | Isolates | |-----------|----------| | `CLONE_NEWNS` (mount) | Mount points, filesystem roots | | `CLONE_NEWPID` | Process IDs | | `CLONE_NEWNET` | Network stack | | `CLONE_NEWUTS` | Hostname | | `CLONE_NEWIPC` | SysV IPC, POSIX message queues | | `CLONE_NEWUSER` | UID/GID mappings | | `CLONE_NEWCGROUP` | cgroup root view | ```c // clone() with namespaces #define _GNU_SOURCE #include int container_init(void *arg) { sethostname("container", 9); mount("proc", "/proc", "proc", 0, NULL); execv("/bin/sh", (char *[]){"/bin/sh", NULL}); return 1; } int stack[1024 * 1024]; clone(container_init, stack + sizeof(stack)/sizeof(int), CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | SIGCHLD, NULL); ``` ### 2. cgroups v2 ```bash # Unified hierarchy (cgroup v2) mount -t cgroup2 none /sys/fs/cgroup # Create cgroup and set limits mkdir /sys/fs/cgroup/mycontainer echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs # Memory limit 256MB echo 256M > /sys/fs/cgroup/mycontainer/memory.max # CPU weight (relative to siblings, default 100) echo 50 > /sys/fs/cgroup/mycontainer/cpu.weight # IO weight echo default 100 > /sys/fs/cgroup/mycontainer/io.weight ``` ```bash # Check current cgroup cat /proc/self/cgroup # OOM events cat /sys/fs/cgroup/mycontainer/memory.events ``` ### 3. overlayfs ``` overlayfs layers ├── lowerdir (read-only image layers) ├── upperdir (container writes) ├── workdir (internal bookkeeping) └── merged (mount point seen by container) ``` ```bash mount -t overlay overlay \ -o lowerdir=lower1:lower2,upperdir=upper,workdir=work \ merged # Docker stores layers in /var/lib/docker/overlay2/ ``` Copy-on-write: reads from lower, writes go to upper. ### 4. runc and OCI spec ```bash # Generate default OCI config mkdir -p mycontainer/rootfs runc spec # Edit config.json — namespaces, mounts, process args # Run container sudo runc run mycontainer # List runc list ``` `config.json` key sections: ```json { "ociVersion": "1.0.2", "process": { "args": ["/bin/sh"], "capabilities": { "bounding": ["CAP_NET_BIND_SERVICE"] } }, "linux": { "namespaces": [ {"type": "pid"}, {"type": "network"}, {"type": "mount"} ], "seccomp": { ... }, "resources": { "memory": { "limit": 268435456 } } }, "root": { "path": "rootfs", "readonly": false } } ``` ### 5. seccomp-bpf ```c // libseccomp example — block mount #include scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW); seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(mount), 0); seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(pivot_root), 0); seccomp_load(ctx); ``` ```bash # Docker default seccomp profile (JSON) # https://github.com/moby/moby/blob/master/profiles/seccomp/default.json # Audit blocked syscalls # kernel: seccomp log via auditd ausearch -m SECCOMP ``` | Action | Effect | |--------|--------| | `SCMP_ACT_KILL` | Kill process | | `SCMP_ACT_ERRNO(n)` | Return error | | `SCMP_ACT_TRACE` | Notify tracer | | `SCMP_ACT_ALLOW` | Permit syscall | ### 6. Linux capabilities ```bash # Drop all caps except needed capsh --drop=all --add=net_bind_service -- -c '/app/server' # File capabilities setcap cap_net_bind_service+ep /usr/bin/myserver getcap /usr/bin/myserver ``` In containers: default Docker drops `CAP_SYS_ADMIN`, `CAP_NET_RAW`, etc. Run as non-root with minimal bounding set. ### 7. User namespace rootless ```bash # Rootless podman/docker maps root in container to unprivileged UID on host cat /proc/self/uid_map # 0 1000 1 # container UID 0 → host UID 1000 ``` Rootless limits: cannot mount most filesystems, no `CAP_SYS_ADMIN`. ### 8. Escape mitigations ``` Defense layers ├── User namespace (rootless) ├── seccomp (block dangerous syscalls) ├── AppArmor/SELinux (MAC) ├── Capabilities drop (--cap-drop=ALL) ├── Read-only rootfs ├── no-new-privileges └── Seccomp + Landlock for filesystem ``` ```bash docker run --read-only --cap-drop=ALL --security-opt=no-new-privileges \ --security-opt seccomp=default.json myimage ``` Known escape vectors: mounted docker.sock, privileged mode, kernel CVEs, `/proc` leaks. ## Common Problems | Symptom | Cause | Fix | |---------|-------|-----| | Container OOMKilled | memory.max exceeded | Raise limit or fix leak | | CPU throttled | cpu.max quota low | Adjust `cpu.max` or weight | | Permission denied in container | Capability dropped | Add specific cap or fix app | | seccomp kill on start | Missing syscall in profile | `strace` to find; allow syscall | | overlay mount fail | workdir not empty | Clean workdir; check permissions | | Rootless mount fail | User namespace limits | Use volume mounts from host | ## Related Skills - `skills/virtualization/qemu-kvm` — VM isolation vs containers - `skills/security/kernel-security` — SELinux, AppArmor, seccomp depth - `skills/observability/ebpf` — trace container syscalls - `skills/runtimes/binary-hardening` — seccomp and capabilities in production - `skills/kernel/kernel-internals` — cgroups and namespaces in kernel - `skills/profilers/strace-ltrace` — syscall tracing for seccomp tuning [View on SkillFed](https://skillfed.io/mohitmishra786/low-level-dev-skills/containers-internals) · [View on GitHub](https://github.com/mohitmishra786/low-level-dev-skills)