What is Multi Tenancy in Kubernetes

Kubernetes is a single-tenant orchestrator by design. Kubernetes does not offer multi-tenancy by default. Enabling a Kubernetes multi-tenant architecture comes with significant challenges, especially in regard to achieving true cluster isolation and fair resource allocation

Kubernetes multi-tenancy is an architectural model where multiple tenants share a cluster’s resources, including storage, networking, CPU, compute, and control plane. This model also includes resource quotas and limits that act as boundaries for resource usage.

Sharing Kubernetes cluster resources helps to reduce costs and enhance productivity through faster and more scalable container-based deployments.

While it’s possible to use single-tenant clusters rather than multi-tenant ones, this solution is ineffective and problematic, particularly at scale. For instance, imagine having to spin up and manage hundreds or thousands of clusters.

By moving to a multi-tenancy model, a tenant’s resources can be organized into separate groups in the control plane, enabling restricted access or visibility to resources not within the control plane domain. This helps you reduce operating costs, increase accessibility and productivity, and boost your return on investment.

In a multi-tenant architecture, multiple instances of an application operate in a shared environment. This architecture is able to work because each tenant is integrated physically but is logically separated. This means that a single instance of the software will run on one server and then serve multiple tenants. In this way, a software application in a multi-tenant architecture can share a dedicated instance of configurations, data, user management and other properties

Key Takeaways

Maximize Resource Usage: To efficiently use resources and to avoid the 'noisy neighbor' problem, set up resource usage limits for each tenant in a multi-tenant Kubernetes cluster

Implement Cost Tracking + Audit Logging: Monitor your infrastructure costs effectively by implementing cost-tracking tools such as Prometheus or Kubecost. Implementing audit logging is equally critical, which can help quickly identify issues like unusual activity or slow API requests.

Prioritize Security with RBAC & Encryption at Rest: Utilize the Role-Based Access Control (RBAC) system to manage who has access to what components within an organization on Kubernetes clusters and simultaneously maintain data security through encrypting sensitive information stored in secrets using options like HashiCorp Vault.

Multi-tenant Kubernetes cluster architecture allows multiple teams or customers to share a single Kubernetes cluster while maintaining isolation and security. Here are some key concepts and best practices:

Key Considerations

Isolation: Use namespaces, network policies, and resource quotas to ensure workloads are isolated and resources are fairly distributed

Security: Implement Role-Based Access Control (RBAC) to manage permissions and ensure only authorized users can access specific resources

Cost Optimization: Efficiently manage resources to reduce costs, especially important in multi-customer scenarios

Use cases
The first step to determining how to share your cluster is understanding your use case, so you can evaluate the patterns and tools available. In general, multi-tenancy in Kubernetes clusters falls into two broad categories, though many variations and hybrids are also possible.

Multiple teams
A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. These workloads frequently need to communicate with each other, and with other workloads located on the same or different clusters.

In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as kubectl, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.

Multiple customers
The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it “SaaS tenancy.” However, a better term might be “multi-customer tenancy,” since SaaS vendors may also use other deployment models, and this deployment model can also be used outside of SaaS.

In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other.

Based on the two usecases defined above, we can see that the Multiple Customer usecase very well fits our requirement and that is eventually what we would be discussing in this article on: Implementing a Multi Tenanacy solution in a kubernetes cluster.

Multi-tenancy isolation category into two levels

Soft multi-tenancy
More applicable to enterprise multi-tenancy, where there are no malicious tenants by default. In this context, isolation is intended to safeguard inter-team business operations and guard against potential security threats

Non-adversarial tenants

Different department/teams in the same company

Not trying to harm other tenants

Hard multi-tenancy — Tailored for service providers offering external services. Given the nature of their business, the security profiles of different tenants’ business users can’t be assured adversarial tenants

Different kinds of users who has no relation to each other

Trying to exploit the system

Focus on securing and isolating each tenant

Kubernetes multi-tenancy benefits

Optimizing resource utilization: make the most out of their cluster resources by sharing them among multiple users or teams. This can reduce the waste of idle or underutilized resources and lower the operational costs

Reducing infrastructure complexity: simplify the infrastructure by consolidating multiple clusters into one. This can reduce the maintenance burden and the risk of configuration drift or inconsistency

Increasing agility and collaboration: speed up the development and delivery cycles by enabling faster provisioning and deployment of workloads. It can also foster collaboration and innovation by allowing different users or teams to work on different projects or experiments within the same cluster

Kubernetes multi-tenancy challenges

Ensuring Isolation and Security Among Different Tenants

One of the main challenges of multi-tenancy in Kubernetes is ensuring the isolation and security of each tenant’s data and workloads.
Kubernetes doesn’t provide a native way to enforce strict boundaries between tenants, so administrators need to use various tools and techniques to achieve this by:

• Using namespaces to group and isolate resources within a cluster. Namespaces can help limit the scope of operations and access controls for each tenant, but they don’t prevent cross-namespace interactions or resource conflicts
• Using network policies to control the traffic flow between pods and services within and across namespaces. Network policies can help prevent unauthorized or malicious communication between tenants, but they do not protect against attacks from within the same namespace or from the cluster network
• Using resource quotas and limits to restrict the amount of CPU, memory, storage, and other resources that each tenant can consume. Resource quotas and limits can help prevent overcommitment and starvation of cluster resources, but they do not guarantee performance or availability for each tenant
• Using role-based access control (RBAC) to define the permissions and roles for each user and group within a cluster. RBAC can help enforce the principle of least privilege and prevent unauthorized actions or access to cluster resources, but it does not prevent human errors or misconfigurations
• Using service accounts and secrets to manage the credentials and tokens for each tenant’s workloads. Service accounts and secrets can help secure the authentication and authorization of pods and services within a cluster, but they do not protect against leaks or compromises of sensitive data

These methods can provide some level of isolation and security for multi-tenant clusters, but they are not foolproof or comprehensive.
Administrators need to carefully configure and monitor them to ensure that they are effective and consistent.
Moreover, these methods can also introduce some trade-offs and challenges in terms of complexity, performance, compatibility, and usability:
• using namespaces can increase the management overhead and the risk of name collisions or conflicts
• using network policies can affect the latency and throughput of network traffic and require additional tools or plugins to implement
• using resource quotas and limits can impact the scalability and reliability of workloads and require careful tuning and balancing
• using RBAC can complicate the user experience and the integration with external identity providers or systems
• using service accounts and secrets can increase the storage requirements and the exposure of sensitive data

Some of the potential issues that can arise from insufficient isolation and security are:
• Cross-tenant interference: One tenant’s workload can affect the performance, availability, or functionality of another tenant’s workload due to resource contention, noisy neighbors, or configuration errors
• Cross-tenant attacks: One tenant can compromise the security or integrity of another tenant’s workload by exploiting vulnerabilities, misconfigurations, or malicious code
• Cross-tenant data leakage: One tenant can access or expose sensitive data belonging to another tenant due to insufficient encryption, authentication, or authorization

Managing the Complexity and Diversity of Tenant Requirements

Different tenants may have different needs and expectations regarding the quality of service (QoS), availability, scalability, reliability, and functionality of their workloads. They may also have different preferences and constraints regarding the configuration, customization, and governance of their workloads.
Moreover, tenants may have different levels of expertise and experience with Kubernetes, which can affect their ability to use the platform effectively and efficiently.

Some of the potential issues that can arise from managing the complexity and diversity of tenant requirements are:
• Inconsistent service levels: One tenant’s workload may receive a higher or lower level of service than another tenant’s workload due to inconsistent resource allocation, scheduling, or monitoring
• Incompatible configurations: One tenant’s workload may conflict with another tenant’s workload due to incompatible settings, dependencies, or versions
• Inadequate governance: One tenant’s workload may violate the policies or regulations of another tenant or the cluster owner due to inadequate enforcement or auditing

Multi-tenancy approaches

Approach 1: one dedicated cluster per tenant

The simplest multi-tenancy model is to take Kubernetes’s single-tenancy design and assign a tenant per cluster, whether that tenant is a customer’s deployment, a business unit, a development team or a solitary developer and his or her application.

The obvious upside here is that no further work is needed to segregate workloads within Kubernetes. All of the isolation guarantees come from the underlying infrastructure provider and the boundaries are drawn at the cluster level. This is a ‘hard’ multitenancy approach.

The downside to this model is poor resource utilization and the cost associated with it. The cost here refers to not just the direct costs of running more virtual machines, VPCs, load balancers, etc, but also the indirect costs of managing such large numbers of clusters. It’s important to consider how these indirect costs can become a significant burden, especially considering the efforts to maintain a single, production-ready cluster.

While running a cluster per tenant sounds extreme, it is actually very popular depending on how you define ‘tenant’. Some large organizations do actually provision a cluster per use case or engineer, but the more common practice is to define tenancy around teams, environments, or product lines.

The simplest model of this for small companies might be a prod and non-prod cluster. For mid-sized companies, clusters may be divided by environment plus teams. Finally, some consulting or PaaS companies allow end-users to provision an isolated environment via separate clusters.

Since operating multiple clusters at scale can be a logistical challenge, there are some tools that help with management such as:
• Cluster API
• HyperShift
• Kamaji
• Gardener
…and this one called Palette that you might have heard of!

Approach 2: multiple tenants within a single cluster via namespaces

On the other end of the spectrum, we have the soft multi-tenancy model where multiple tenants are housed within a single cluster. Isolation is then enforced by various Kubernetes constructs.

The key concept here is Kubernetes namespaces, which allows logical separation of resources within the same cluster. But by default, namespaces alone do not enforce isolation. In fact, workloads in different namespaces can freely discover and communicate to each other and have no limits enforced on usage of shared resources like CPU, memory, or Kubernetes operations.

To successfully isolate each tenant via namespaces, the following is required:

• API isolation via roles and role-bindings
• Network isolation via network policies
• Guardrails against cluster resources via resource quotas and limit ranges
Let’s walk through each one in more detail.

API isolation

Kubernetes uses the role-based access control (RBAC) model to implement authorization policies. This means that the cluster admin is responsible for creating the right roles and policies to limit what different Kubernetes users or service accounts can get access to. Each role is scoped to a single namespace so the admin can carefully control what resources the role can have access to (e.g., secrets, volumes, etc).

However, Kubernetes also provides cluster roles that work across namespaces. While this is convenient for things that are meant to be shared across namespaces like ingress controllers or webhooks, a badly designed cluster role or over-scoped one can easily break the security and isolation model.

Network isolation

Since one of the goals of Kubernetes was to make it easier for pods to discover one another and communicate, by default, there is no network isolation. Any pod can talk to another pod in the same cluster with no restrictions. In a multi-tenancy model, we have use cases where we need to limit this either to enforce security or to simply reduce noise or spammy behavior.

There are several approaches to network isolation.

One can happen at the Container Network Interface (CNI) layer where network policies can be enforced. Popular open-source CNIs include Calico, Cilium, Flannel, and Weave. Different cloud providers may also have their own flavor of CNIs where network policies can be enforced to disallow communication.

The other approach is to address this on the network mesh layer. If you are running Istio or Linkerd, you can set policies to limit communication via proxy sidecars.

Guardrails for cluster resources

Since multiple tenants all share the same cluster-level resources, to achieve isolation, we also need some guardrails against either intentional or unintentional attacks on shared resources.

First, we need to remember that nodes are not namespace-scoped resources. That is, different workloads that fall under different namespaces may be physically running on the same node. This can present security risks when a malicious attack gains access to the underlying node either via privilege escalation or other means, and can affect more workloads than the compromised namespace.

To remediate, we can use node taints to segregate critical workloads onto different nodes and also utilize security contexts and other policy engines to prevent privileged access.

Next, we also need to make sure one tenant does not hog all the underlying resources available. We can accomplish this by setting resource quotas and limit ranges to different namespaces so that CPU and memory usage is within some bound.

This is not only a good cost-saving measure, but a practical way to prevent some misconfigured job left overnight from affecting more tenants.

Best practices for using namespaces

While this ‘soft’ tenancy model significantly increases resource utilization compared to the single tenancy model, it leaves a ton of responsibility for the cluster admin to enforce isolation correctly to limit blast radius.

Some controls are simple to implement, such as setting resource quotas and limit ranges. But as the user base grows, managing different roles and network policies becomes a challenge.

For this reason, this model works best when all of the tenants are trusted entities (e.g., different teams from the same organization) or if you control the entire cluster, like in a hosted SaaS model where the end users do not have direct access to the cluster. This way the various controls mentioned above can be frequently tweaked and refined behind the scenes.

In practice, some mix of single tenancy and multi tenancy is used. For example, a SaaS company may have multi-tenant clusters divided by environment (i.e., prod vs. non-prod) or even product lines. Others might opt to host a developer sandbox on a multi-tenant cluster and use cluster-segregation for higher environments.

Limitations of using namespaces

It is important to note that even with API isolation and network isolation in place, using namespaces does not provide the same level of security and isolation as using separate clusters. This is because some core Kubernetes resources are still shared across namespaces such as CoreDNS and Kubernetes API. This means that a single malicious actor abusing these components could impact the entire cluster.

The other limitation involves tenants who may want to span multiple namespaces. For example, we might want to have a hierarchical structure of an organization that spans across multiple teams. Kubernetes has a special interest group working on this problem via Hierarchical Namespace Controller (HNC) but the features are still relatively new.

Approach 3: using virtual control planes

Given the limitations of using namespaces as means to implement multi-tenancy and the operational burden associated with it, the open-source community has designed a new model using virtual control planes.

The key idea here is to extend Kubernetes APIs to form custom resource definitions (CRDs) and create virtual control planes that are assigned to each tenant. Each virtual control plane runs its own API server, controller manager, and etcd to decouple it from the host Kubernetes cluster. This solves the issue of shared Kubernetes API access that is not possible via namespace isolation.

Some popular implementations of virtual cluster mechanisms include:

• VirtualCluster
• vCluster
• Capsule
• Kiosk

With the rise of platform engineering in recent years, implementing multi-tenancy via virtual control planes has been a popular option. It is an elegant solution that balances the needs of tenant isolation with high resource utilization and cost savings.

Virtual clusters also abstract away a lot of the complexities of implementing controls at the namespace level, which also reduces the operational complexity.