In November 2025, Kubernetes SIG Network announced that the community-maintained ingress-nginx project would be retired in March 2026. No further releases, no bugfixes, no security patches. This affects roughly half of all cloud-native environments.

If you're running ingress-nginx (the community project, not F5's commercial NGINX-INGRESS), you now have an infrastructure dependency that is actively accumulating unpatched vulnerabilities. The NGINX "snippets" annotations that once provided flexibility became serious security vulnerabilities. The Ingress API itself is frozen and will not be extended. The entire Kubernetes networking ecosystem is moving to the Gateway API.

This post covers how we migrated Kloudfuse from NGINX Ingress Controller to Envoy Gateway with zero downtime. The key innovation was a service selector repointing technique that preserves the existing load balancer throughout the migration, avoiding DNS changes, IP address shifts, and the traffic interruptions that typically accompany ingress controller swaps. We'll walk through the three-step migration, the technical decisions behind it, and the gotchas we encountered.

Why Envoy Gateway Instead of Another Ingress Controller?

The question isn't just "what replaces NGINX" but "what does the Kubernetes networking ecosystem look like in two years?" The answer is the Kubernetes Gateway API, and Envoy Gateway is the CNCF's own choice for implementing it. When CNCF migrated their internal services cluster, they chose Envoy Gateway. That's a strong signal.

The specific advantages over NGINX Ingress Controller:

Dynamic configuration via xDS. NGINX relied on config files requiring process reloads. Envoy receives configuration updates at runtime through the xDS protocol (Listener Discovery, Route Discovery, Cluster Discovery, Endpoint Discovery, Secret Discovery). Routes can be swapped without affecting in-flight requests. Delta xDS transmits only changed resources, reducing overhead in large deployments.

Native rate limiting. Envoy supports both global rate limiting (consistent across the entire fleet via an external rate limiting service) and local rate limiting (per-instance, no external dependency) simultaneously on the same route. NGINX's rate limiting was configured via non-standard annotations.

Circuit breaking at the network level. Envoy enforces circuit breaking distributely, with configurable thresholds for maximum connections, pending requests, outstanding requests, and active retries. NGINX required per-application configuration.

Role-based configuration model. The Gateway API separates concerns into distinct resources (GatewayClass, Gateway, HTTPRoute) with separate roles for infrastructure providers, cluster operators, and application developers. NGINX's Ingress API had a single user role and relied heavily on provider-specific annotations for anything beyond basic routing.

Protocol support. First-class HTTP/2 with transparent proxying in both directions, native gRPC support, and protocol-agnostic routing (HTTP, TCP, gRPC). NGINX Ingress only supported HTTP with TLS termination through the Ingress API.

How Did We Achieve Zero-Downtime Migration?

Most migration guides recommend one of two approaches: weighted DNS shifting (gradually move traffic percentages) or parallel load balancers (stand up the new ingress alongside the old, then switch DNS). Both have problems.

Weighted DNS shifting depends on DNS TTL propagation, which is unpredictable. Clients cache DNS records, CDNs cache DNS records, and the actual traffic split never matches the configured weights during the transition window.

Parallel load balancers mean new IP addresses. Every external system pointing at the old IP (monitoring, webhooks, partner integrations, firewall rules) needs to be updated. For a platform like Kloudfuse deployed in customer environments, forcing an IP change on every customer is a non-starter.

The Service Selector Repointing Technique

Our approach preserves the existing NGINX Load Balancer Service throughout the migration. The key: repoint the service's pod selector from NGINX pods to Envoy pods. The same NLB/ALB keeps the same IP addresses and DNS records. No data loss, no traffic interruption.

Here's the three-step migration:

The helm.sh/resource-policy: keep annotation on the NGINX LB service is critical. It prevents Helm from deleting the service during the transition, even as NGINX controller resources are being removed. Without this annotation, Helm would garbage-collect the service and create a new one with a new IP address, which is exactly what we're trying to avoid.

Step-by-Step Configuration

Step 1: Prepare. Add to custom_values.yaml:

envoy-gateway:

  enabled: true

  installGatewayRoutes: true

  envoyMigration:

    enabled: true

ingress-nginx:

  enabled: true

  installIngressRules: true

Run helm upgrade. Verify Envoy pods are running alongside NGINX. Traffic still flows through NGINX.

Step 2: Switch. Update custom_values.yaml:

envoy-gateway:

  envoyMigration:

    external: true    # switch external LB to Envoy

    internal: true    # switch internal LB to Envoy

Run helm upgrade. The NGINX LB service selector now points to Envoy pods. Verify by checking that HTTPS returns a 401 (Envoy's auth challenge) instead of a 302 (NGINX's redirect behavior). AWS NLB health checks may take 15-30 seconds to converge after the targetPort change.

Step 3: Cleanup. Update custom_values.yaml:

ingress-nginx:

  enabled: false

  installIngressRules: false

Run helm upgrade. NGINX controller pods are removed. The LB service persists. Verify NGINX controller pods are gone and HTTPRoutes show Accepted/Resolved status.

Rollback

If issues arise after Step 2, the rollback is straightforward: remove the resource-policy annotations from the NGINX LB, disable Envoy, re-enable NGINX in custom_values.yaml, and run helm upgrade. Then clean up orphaned Envoy resources (Deployments, Services, GatewayClass finalizers).

What Gotchas Did We Encounter?

AWS NLB hairpin problem. On AWS, cert-manager's HTTP-01 self-check can fail during initial certificate issuance because pods cannot reach the external NLB from inside the cluster. The workaround: add a hostAliases entry mapping the domain to Envoy's ClusterIP, so cert-manager's self-check bypasses the external load balancer.

CRD scope. Gateway API CRDs are cluster-scoped. Deleting them removes Gateway, HTTPRoute, and SecurityPolicy resources across all namespaces, not just yours. In shared clusters, only delete the GatewayClass for your own namespace. Kloudfuse handles this with namespace-scoped controller configuration (provider.kubernetes.watch.type: Namespaces).

Orphaned resources. When running helm delete, the Envoy Gateway controller's runtime-created resources (Deployments, Services, ConfigMaps) are not deleted by Helm because Helm didn't create them. These require manual cleanup.

Gateway "Programmed: False." During migration, the Gateway resource shows Programmed: False because the existing NGINX LB is being reused rather than creating a new one. This is expected behavior, not an error. The Gateway becomes Programmed: True after migration completes and the LB is fully transitioned.

TLS configuration format change. NGINX's flat TLS configuration (tls.ingestInternalHosts) migrates to a nested format (tls.internalIngest.hosts). Both formats are maintained during the migration period; the old format is removed after Step 3.

What Are the Prerequisites?

The migration requires Kloudfuse 4.0.0 or later, Kubernetes 1.27+, Helm 3.x, and cert-manager v1.14+ with Gateway API support enabled (config.enableGatewayAPI=true). Envoy CRDs must be installed matching the Kloudfuse chart version (v1.3.0 for KF 4.0.0, v1.7.1 for KF 4.0.1+).

For multi-tenant clusters, each namespace scopes its own Envoy Gateway controller via namespace-specific controllerName, preventing cross-namespace interference.

How Does the New Architecture Support Advanced Traffic Management?

With Envoy Gateway in place, Kloudfuse deployments gain access to capabilities that were impractical or impossible with NGINX Ingress Controller:

• Separate internal and external traffic paths. Ingest traffic routes through an internal load balancer. Query traffic routes through an external load balancer. Each path has independent TLS configuration and access controls, configurable through the tls.internalIngest and tls.externalQuery fields in custom_values.yaml.

• Envoy's native traffic management. Envoy Gateway supports weighted cluster routing for progressive traffic splitting, enabling canary and blue-green deployment patterns for platform updates. These are Envoy Gateway capabilities available to any deployment using the Gateway API, and they become accessible to Kloudfuse environments after migration.

• Protocol and observability improvements. Envoy provides first-class HTTP/2 with transparent proxying, native gRPC support, and detailed metrics, traces, and access logs as built-in capabilities. For Kloudfuse deployments, this means OpenTelemetry's gRPC transport works without additional proxy configuration, and Prometheus remote write benefits from HTTP/2 connection multiplexing. Envoy's own telemetry can be ingested into the Kloudfuse platform alongside application data.

The Design Decision: Selector Repointing Over DNS Migration

The standard industry approach to ingress controller migration involves DNS changes: stand up the new controller with a new load balancer, then shift DNS to the new IP. This is the approach recommended by most migration guides, including CNCF's own documentation for other scenarios.

We chose selector repointing because Kloudfuse deploys in customer environments where we don't control DNS. Customers may have firewall rules, partner integrations, and monitoring systems pointed at specific IP addresses. Requiring an IP change would turn a platform upgrade into a multi-team coordination exercise involving network engineering, security, and every external integration.

The tradeoff is that selector repointing is more tightly coupled to Kubernetes service internals. It depends on the load balancer provider supporting selector changes without recreating the service, and on the health check convergence behavior of the specific cloud provider's load balancer. AWS NLB converges in 15-30 seconds. Other providers may differ.

We accepted this coupling because the alternative (DNS migration) would have required scheduling maintenance windows, which contradicts our goal of making infrastructure transitions routine rather than exceptional. As the 4.0 blog describes it: "This is the kind of infrastructure transition that typically requires a quarter of planning. With Kloudfuse 4.0, it is a documented, tested migration path."

Next Steps

The Envoy Gateway setup guide covers both fresh installations and migrations from NGINX. The Kubernetes Gateway API documentation provides background on the networking model that Envoy Gateway implements.

If you're still running ingress-nginx, the clock is ticking. The SIG Network retirement announcement is final, and no security patches will be backported. The migration is less disruptive than most teams expect, and the Gateway API's role-based configuration model is a genuine improvement over annotation-driven ingress rules.

Has your team started planning the ingress-nginx migration? We'd be interested to hear what approach you're considering and what's blocking you from starting.