Multi-Tenant Prometheus: Adobe’s Approach to Secure Kubernetes Metrics Isolation
To grant teams self-service access to their specific Prometheus metrics within shared Kubernetes setups without revealing other groups’ shared data, Adobe staff outlined an open-source methodology. The design is particularly aimed at GPU-heavy environments, where teams may need visibility into utilisation and power consumption to understand whether expensive accelerator capacity is actually being used.
An open-source strategy for letting teams query their personal Prometheus metrics inside multi-tenant Kubernetes setups without revealing other groups’ metrics has been detailed by Adobe staff. Specifically focusing on GPU-heavy clusters, the layout helps groups monitor power usage and utilization to check if costly accelerator resources are genuinely utilized. The problem is straightforward but difficult to solve safely. Because a central Prometheus deployment can hold metrics spanning thousands of namespaces, it remains unsafe for open tenant access. Granting all teams query privileges risks revealing rival data, and allowing heavy user volumes to query the shared database can trigger performance bottlenecks and noisy-neighbour issues. Adobe’s solution places a tenant-aware proxy between users and the central Prometheus and optionally gives each tenant its own smaller Prometheus instance. The architecture uses existing Kubernetes and CNCF technologies rather than introducing another metrics platform. Authentication and authorization take place via NGINX and kube-rbac-proxy before incoming requests hit a multi-tenant Prometheus proxy. The proxy identifies the tenant, discovers available Prometheus backends, and restricts queries to that tenant’s namespace. A key component is prom-label-proxy, which modifies incoming PromQL queries to enforce a namespace constraint. Relying simply on developers to specify the correct namespace in requests fails to establish an adequate security perimeter. The proxy enforces restrictions prior to queries hitting Prometheus, stopping tenants from intentionally writing queries that reach outside their namespace. The platform additionally brings in a Kubernetes custom resource known as MetricAccess, which lets groups specify their required metrics. Metric definitions can use exact metric names, regular expressions, or PromQL selectors. This enables self-service operation while keeping foundational data collection and permission rules in the hands of the platform group. For teams requiring their own dashboards and alerts, the design can periodically remote-write a curated set of metrics into a tenant-specific Prometheus instance. When metricIsolation is turned on, only the given tenant’s metrics are gathered, which cuts down storage use in their Prometheus deployment and restricts query scopes. Adobe notes that a sample setup dropped a tenant’s recorded metric series count from upwards of 10,000 down to roughly 300. Apart from reducing storage and query requirements, this creates another isolation boundary: data that is never collected into the tenant’s store cannot subsequently be exposed through that store. Infrastructure-wide metrics remain handled by the main Prometheus hub, whereas individual teams rely on isolated Prometheus instances for specific queries and dashboards. The shared system therefore does not have to serve every developer’s routine observability queries. The immediate use case is GPU visibility. Even though hardware allocation carries heavy infrastructure expenses, mere assignment does not guarantee that those GPUs are performing productive tasks. The authors note discovering a GPU that stayed at zero utilization across 11 straight days while remaining powered and assigned. By leveraging metrics covering framebuffer memory, power draws, request frequencies, and GPU usage rates, groups can craft queries to spot idle hardware, powered accelerators lacking active workloads, or underutilized GPUs processing traffic. Although GPUs provide the most obvious example, the architecture is not GPU-specific. At its core, the pattern relies on a tenant-aware observability framework combining authentication, query boundaries, curated metric access, and optional isolated storage per tenant. This gains importance as Kubernetes environments increasingly host mixed workloads spanning standard applications, data pipelines, and AI tasks. Platform engineering groups must deliver adequate monitoring tools for developers without converting core telemetry infrastructure into a security vulnerability or performance bottleneck. There are several established approaches to the same underlying multi-tenancy problem. Grafana Mimir builds multi-tenant isolation directly into its system framework, applying tenant tags to restrict metric searches and leveraging an auth layer to set proper tenant boundaries. Mimir also provides query-front-end capabilities for managing and scaling the read path. Cortex—the foundation for multiple managed services compatible with Prometheus—employs a comparable X-Scope-OrgID mechanism to keep tenant metric data separate.