Containers have been around for decades, but their adoption has gained momentum quite recently. A container is a software solution that provides an additional layer of abstraction and automation of operating system-level virtualization on Windows and Linux.
Its benefits over hypervisor-based deployments range from eliminating performance penalties to increased visibility and decreased difficulty in debugging and management. The ability to easily run and accommodate legacy applications, reduced shipping pains and higher utilization of resources are among other factors driving its business needs.
Some commercial and open-source/commercial solutions of containers include: LxC, Docker, CoreOS, Windows Container, etc. As organizations gear up to run mission-critical services on containers, it begs the question: are containers secure?
In this article, we explore some of the security challenges faced by containers, measures taken to mitigate them, and security tools used by organizations to safeguard containers.
Container Security Challenges
A container at its core is an allocation, portioning, and assignment of host resources such as CPU Shares, Network I/O, Bandwidth, Block I/O, and Memory (RAM) so that kernel level constructs may jail-off, isolate or “contain” these protected resources so that specific running services (processes) and namespaces may solely utilize them without interfering with the rest of the system. To use containers safely, you need to be aware of the potential security threats and their implications. For most containers, some of the known security challenges include:
Large Attack Surface: A single host either on premise or in the cloud harbors multiple containers with varied services and processes. When the number of such hosts multiply, it becomes difficult to track or detect anomalies through all the noise generated by various containers.
Containers also connect to each other inside the same host or across clusters, thus making their communication invisible to traditional firewalls and networking tools, and limiting the ability to understand and control traffic at a granular level. In addition, Container engines like Docker provide a layer of abstraction that mask certain activities of a container, making it further difficult to detect anomalies.
Furthermore, lifespan of containers is far shorter than that of VMs, on average. Containers can be executed in an instant, run for a few minutes, then stopped and removed. This ephemerality makes it possible to launch attacks and disappear quickly, without any trace for detection.
Also, certain container processes or services require root/escalated privileges. Escalated privileges imply a greater risk from an attack. An attacker who gains access to such a container by compromising an application or kernel vulnerability, can manage to gain access to other containers on the host with elevated privileges as the kernel is shared among containers.
Kernel Insecurity: Unlike in a VM, the kernel and its resources are shared among all containers on the host. This presents a scenario of a homogeneous environment risking mass exploitation or vulnerability. It is susceptible to a single point of failure; if the host platform is disabled, crashes, needs to be rebooted or is compromised, all of the guest containers are affected. An instance of this is the Copy-on-Write vulnerability. Also, several areas of the kernel still lack support for namespaces (an isolation mechanism for OS virtualization). For example, in Linux, procfs and sysfs pseudo-filesystems are not namespace aware.
Container Insecurity: It is possible for one container to monopolize access to certain resources–including memory, user IDs (UIDs)—and can starve out other containers on the host, resulting in a denial-of-service (DoS). It is also possible for a compromised container on the host to alter the data or resources used by another container without having escalated privileges as the containers share the host and its resources. An instance of this is the vmsplice exploit, where the vulnerability is turned to a buffer overflow vulnerability letting a malicious person to jump into the code of his or her choice. A compromised container can also be leveraged to launch attacks across the network, provided that the raw sockets were not properly restricted.
Corrupt Images: Container images are usually open source and reside in public repositories, harboring an endless stream of uncontrolled code. There is a possibility that these images may be modified and contain malicious code or vulnerable software. Any organization that uses these tampered/malign images stands vulnerable to cyber-attacks.
Container security measures
The numerous compromises of containers have mainly been caused by improper implementation, rather than the security flaws or vulnerabilities present in the container itself. Traditional security controls, while robust, may not be appropriate or available to run in containers – Data at rest or file encryption – Endpoint protection – Backup & restore are some of the security measures containers find hard to implement and execute to fruition. However, the following tips on container security best practices will help align your efforts for a robust container deployment.
Host Hardening: As the host is shared among containers, hardening the host would be the first step towards securing containers.
- Configure namespaces for containers to maintain isolation of the container from the host and between containers. Lack of user namespaces can effectively undo the vast majority of container hardening. For instance, the guest may be able to remount specific system directories critical to security enforcement (cgroups, procfs, sysfs) or the host’s devpts can be exposed, allowing the guest to remount it and control it.
- Configure control groups (cgroups) for applying hardware resource limits and access controls to a process or collection of processes. This helps limit a given resource over a collection of containers to control performance and security. An alternative to cgroups is the use of ulimits/rlimts.
- Configure capabilities for privileged operations. They can be set and applied to either process threads or binary files using extended filesystem attributes. Avoid high-risk capabilities such as CAP_NET_RAW, CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH, CAP_NET_ADMIN, CAP_SYS_ADMIN, etc. Also, incorrect implementation of capabilities can lead to container escapes, and other capability vulnerabilities. Despite these risks, specifying capabilities can greatly reduce the potential privilege escalation, help restrict attack surfaces, and limit the impact of successful privileged process exploitation.
- Implement GRSEC and PAX. This provides features such as non-executable stack, address space layout randomization, setting function pointers to read-only, checking refcount overflows, etc. that aid in avoiding risks of buffer overflow attacks. GRSEC and PAX also provide safety checks at compile-time and run-time.
- Use security model templates. For instance, Ubuntu comes with AppArmor templates for LXC, and those templates provide an extra safety net (even though it overlaps greatly with capabilities). AppArmor also provides runtime protection, Mandatory Access Control and security profile for each program.
- Define policies. SELinux is an open source tool that provides runtime protection, Mandatory Access Control (MAC) for every user, application, process and file. It also offers an extra layer of access policies and isolation between the host and the containerized apps. Seccomp is another tool that has isolation capabilities.
- Whitelist files and executables that the container is allowed to run. This also provides a baseline for anomalies and prevents the use case of “noisy neighbors” and container breakout scenarios.
- Keep the host kernel updated, apply the necessary patches as and when they are made available.
- Least privilege at runtime is crucial to secure against privilege escalation attacks.
- Containers are useful for both stateless and stateful applications. Protecting attached storage is a key element of securing stateful services. Block storages can be secured using SELinux capabilities. Shared storage can be limited to certain containers using group ids.
Container image and registry hardening: It has become common for admins to pull images from public repositories maintained by people they don’t know and images that are not verified. Image forgery and tampering is rampant and a common threat in deploying containers. Hence, as a consequence, numerous vulnerabilities and configuration issues tend to trickle down into the production environment. Therefore, it is recommended:
- Verify the provenance of the Image. The standard for image provenance is Docker Content trust. A digital signature is added to images before they are pushed to the registry and when they are pulled, Docker Content trust verifies the signature, thereby ensuring the image comes from the correct organization and the contents of the image match the image pushed by the organization.
- Generally, it is best to consider using your own private registry. Image signatures and fingerprints can provide chain of custody and integrity checks. Also, a successful run of a deployment pipeline gives an immutable image for deploying to production. Employing full security scans (Blackbox and Whitebox) means you can catch security issues before releasing anything into production. Special care needs to be taken to ensure the secrets (keys, passwords, etc.) are not contained in the container images as well.
- Secrets should be available to relevant users only and should not be stored on disks or be exposed at a host level. Ensure the secret disappears once the container is stopped as well.
- Tools such as Tenable, Sysdig Falco, Twistlock Trust could be used for vulnerability scanning, troubleshooting and debugging, malware scanning, etc. of the image.
Container platform hardening: When managing container deployment at scale, you have to consider control access to-and management of-shared resources, how to scale on demand and how to keep track of deployed containers. In order to achieve this, container platform with built-in management and security features called “container orchestration” can be used. Some orchestration platforms include; Rancher, MESOS/Aurora, Docker Swarm, LXD, Openstack Container, Kubernetes, etc.
Given the wealth of capabilities orchestration platforms provide, strong role-based access control is a critical element of a container platform. Access to the platform needs to be limited to trusted developers and operators. API access control (authentication and authorization) is critical for securing the container platform. Validating all incoming requests to the platform and invoking triggers on faults is a level of security scrutiny to be achieved with container platforms. All communications to the platform must be over secure encrypted channels.
Network Hardening: With network defense in mind, network segmentation to segregate clusters or zones of containers by application or workload is crucial. To achieve this, each container must get its own network stack and mustn’t interfere with sockets or interfaces of other containers on the host. Network namespaces can be used to provide each collection of containers, its own IP and port range to bind to. Care needs to be taken to ensure that the collection of containers from different namespaces cannot send or receive packets to each other (exceptions apply). Egress and Ingress traffic needs to be monitored at the router or firewall. Network policies can be used to further harden the network.
Monitoring and reporting: a comprehensive strategy for monitoring container infrastructure is crucial for maintaining container security. Monitoring involves gleaning information (events, stats, configurations, user access and logs) and performing analytics on the gathered information to validate container resources and help make better and informed decisions regarding container deployment and security. Such monitoring systems would also aid in troubleshooting and performing root cause analysis of issues surfacing from distributed container-based applications and containers themselves.
Tools such as Sysdig Falco, AquaSec, StackRox serve as a comprehensive intrusion and detection platform. These platforms also help verify lack of user namespaces, insecure defaults or weak configuration, weak network defaults, unsafe exposures like procfs, lack of protection for sysfs, general information disclosure, networking exposure due to shared host network namespaces, etc. They also have auditing and compliance capabilities.
Auditing is also a crucial technique to ensure security. Auditing identifies which images are running in production and which version of the code they are running. Tools, such as Docker diff, can be used to verify that container filesystems have not diverged from the underlying image.
Conclusion
Managing security throughout the container lifecycle with proactive integrity checks of images in the registry and enforcing controls as changes are deployed into production would be the way to go. Integrating containers into a continuous security loop, including image provenance, patching, and security scanning and policy-based monitoring would be ideal for container security. Tools such as the Docker Bench for Security needs to be used religiously to check for dozens of common best-practices around deploying containers in production.
References
[1] https://aucouranton.com/2014/06/13/linux-containers-parallels-lxc-openvz-docker-and-more/
[2] A study of Security Vulnerabilities on Docker Hub: http://dance.csc.ncsu.edu/papers/codaspy17.pdf
[3] Ten Layers of Container Security: https://www.redhat.com/cms/managed-files/cl-container-security-openshift-cloud-devops-tech-detail-f7530kc-201705-en.pdf
[4] Understanding and Hardening Linux Containers: https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-1-1.pdf
[5] Docker Security: https://developers.redhat.com/blog/2016/02/24/10-things-to-avoid-in-docker-containers/
[6] Kernel space: the vmsplice() exploit: https://github.com/coreos/clair
[8] https://neuvector.com/blog/15-tips-run-time-container-security-strategy/
[9] http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf
Kudelski Security Research 