This guide covers mTLS from first principles through production deployment: how the handshake works, enabling it in Istio, automating certificate lifecycle with cert-manager, implementing SPIFFE/SPIRE workload identity, and debugging the errors you’ll inevitably encounter.

Why mTLS Matters for Zero-Trust Kubernetes

Traditional network security assumed that traffic inside the cluster perimeter was safe. Zero-trust inverts this: trust nothing, verify everything. mTLS is the cryptographic mechanism that enforces this at the transport layer.

Without mTLS, a compromised frontend pod can call billing-service APIs directly. With mTLS, the billing-service Envoy proxy rejects any connection whose client certificate was not issued by the cluster’s trusted CA — even if the request comes from inside the cluster.

The practical benefits:

• Workload identity: Certificates encode the service account identity (via SPIFFE ID), enabling policy decisions based on who is calling, not just what IP address is calling
• Encryption in transit: All inter-service traffic is encrypted end-to-end, including east-west traffic that never leaves the cluster
• Compliance: PCI-DSS 4.0 (Requirement 4), SOC 2 Type II, and HIPAA all require encryption of data in transit — mTLS satisfies this for internal APIs
• Audit trail: Certificate subject/issuer fields appear in access logs, providing cryptographic proof of which workload made each call

How the mTLS Handshake Works

A regular TLS handshake has 3 steps: ClientHello → ServerHello+Certificate → Finished. mTLS adds one more:

1. Client sends ClientHello
2. Server responds with its certificate + a CertificateRequest
3. Client sends its own certificate along with its CertificateVerify (a signature proving it holds the private key)
4. Both sides derive the session key and begin encrypted communication

Both certificates must chain to a CA that the other side trusts. In Istio, this CA is Istiod, which acts as an internal PKI and issues certificates automatically to every Envoy sidecar.

The certificate encodes the workload’s SPIFFE ID in the Subject Alternative Name (SAN) field:

spiffe://cluster.local/ns/payments/sa/billing-service

This URI uniquely identifies the Kubernetes service account running the workload, enabling identity-based authorization policies.

Enabling mTLS with Istio

Istio’s service mesh uses Envoy sidecar proxies injected into every pod. These proxies handle mTLS transparently — your application code never manages certificates directly.

Step 1: Verify Istio is Installed

istioctl version
kubectl get pods -n istio-system

Istiod must be running. It serves as the Certificate Authority (CA) that issues certificates to all sidecars.

Step 2: Enable Sidecar Injection

Label your namespace to automatically inject Envoy sidecars:

kubectl label namespace payments istio-injection=enabled

Restart existing deployments to inject sidecars:

kubectl rollout restart deployment -n payments

Step 3: Apply PeerAuthentication Policy

The PeerAuthentication resource controls whether mTLS is required. Start with PERMISSIVE (allows both mTLS and plain HTTP) during migration, then switch to STRICT:

# peer-auth-permissive.yaml — migration phase
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: PERMISSIVE

# peer-auth-strict.yaml — final enforcement
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT

Apply with:

kubectl apply -f peer-auth-strict.yaml

Step 4: Configure DestinationRule for Outbound Traffic

The DestinationRule tells Envoy to use mTLS when calling services. Without this, even if the server enforces mTLS, outbound connections may use plain HTTP:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-mtls
  namespace: payments
spec:
  host: "*.payments.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

ISTIO_MUTUAL instructs Envoy to use certificates issued by Istiod — no manual certificate management needed.

Step 5: Verify mTLS is Active

# Check mTLS status for a specific pod
istioctl x describe pod billing-service-7d9f6-xk2p3.payments

# Inspect the certificate Envoy is using
kubectl exec -it billing-service-7d9f6-xk2p3 -n payments -c istio-proxy -- \
  pilot-agent request GET certs/

# Confirm traffic is encrypted (look for TLS handshake in Kiali or Grafana Istio dashboard)
istioctl dashboard kiali

Automating Certificate Rotation with cert-manager

Istio’s built-in CA (Istiod) handles certificate rotation for sidecar-to-sidecar mTLS automatically. But for services that need certificates outside the mesh — external load balancers, ingress TLS, job runners without sidecars — cert-manager is the standard solution.

Install cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=60s

Create a ClusterIssuer (using internal CA)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: ca-key-pair  # Secret containing ca.crt and tls.key

Issue a Certificate for a Service

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: billing-service-cert
  namespace: payments
spec:
  secretName: billing-service-tls
  duration: 24h
  renewBefore: 8h        # Renew 8 hours before expiry
  subject:
    organizations:
      - corp.example.com
  dnsNames:
    - billing-service.payments.svc.cluster.local
    - billing-service.payments.svc
  uris:
    - spiffe://cluster.local/ns/payments/sa/billing-service
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer

The spiffe:// URI in uris makes this certificate SPIFFE-compliant — it can participate in SPIFFE-aware identity verification alongside Istio-managed certificates.

Check certificate status:

kubectl get certificate -n payments
# NAME                    READY   SECRET                  AGE
# billing-service-cert    True    billing-service-tls     2m

kubectl describe certificate billing-service-cert -n payments
# Events: Successfully issued certificate from ClusterIssuer "internal-ca"

SPIFFE/SPIRE: Federation-Ready Workload Identity

SPIFFE (Secure Production Identity Framework For Everyone) solves a harder problem: how do services in different clusters, clouds, or data centers authenticate each other without sharing a common CA?

SPIRE (the SPIFFE Runtime Environment) is the reference implementation. It:

1. Attests each workload’s identity using platform evidence (Kubernetes node/pod metadata, AWS instance metadata, TPM attestation)
2. Issues short-lived X.509 SVIDs (SPIFFE Verifiable Identity Documents) to each workload
3. Federates trust across domains — a service in AWS us-east-1 can verify a certificate issued by a SPIRE server in GCP us-central1

Deploy SPIRE in Kubernetes

# Clone SPIRE quickstart
git clone https://github.com/spiffe/spire-tutorials.git
cd spire-tutorials/k8s/quickstart

# Deploy SPIRE server and agent
kubectl apply -f spire-namespace.yaml
kubectl apply -f server-account.yaml server-cluster-role.yaml server-configmap.yaml server-statefulset.yaml server-service.yaml
kubectl apply -f agent-account.yaml agent-cluster-role.yaml agent-configmap.yaml agent-daemonset.yaml

# Verify SPIRE server is running
kubectl get pods -n spire

Register a Workload Entry

# Register billing-service with its Kubernetes service account
kubectl exec -n spire spire-server-0 -- \
  /opt/spire/bin/spire-server entry create \
  -spiffeID spiffe://cluster.local/ns/payments/sa/billing-service \
  -parentID spiffe://cluster.local/spire/agent/k8s_sat/payments/$(kubectl get node -o jsonpath='{.items[0].metadata.name}') \
  -selector k8s:ns:payments \
  -selector k8s:sa:billing-service

SPIRE agents on each node deliver SVIDs to workloads via the SPIFFE Workload API (a Unix domain socket). Applications retrieve certificates programmatically without any manual secret management.

Implementing AuthorizationPolicy with mTLS Identity

Once mTLS is active and certificates carry SPIFFE IDs, you can write fine-grained authorization policies based on workload identity — not IP addresses, which are ephemeral in Kubernetes:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: billing-service-policy
  namespace: payments
spec:
  selector:
    matchLabels:
      app: billing-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals:
          # Only allow calls from checkout-service in the same namespace
          - "cluster.local/ns/payments/sa/checkout-service"
    to:
    - operation:
        methods: ["POST"]
        paths: ["/api/v1/charge"]

This policy allows only the checkout-service service account to call POST /api/v1/charge. Any other workload — even inside the cluster — gets a 403. The decision is based on the cryptographic identity in the mTLS certificate, not on IP allowlists or network ACLs.

Debugging mTLS Certificate Errors

Error: CERTIFICATE_VERIFY_FAILED

SSL routines:ssl3_read_bytes:certificate verify failed

Cause: The CA that signed the client certificate is not in the server’s trust bundle.

Fix: Verify both sides use the same CA root:

# Get the CA cert Istio is using
kubectl get configmap istio-ca-root-cert -n istio-system -o jsonpath='{.data.root-cert\.pem}' | openssl x509 -text -noout | grep Issuer

# Verify the client certificate was signed by the same CA
openssl verify -CAfile ca.crt client.crt

Error: upstream connect error, reset reason: connection termination

Cause: STRICT mTLS policy is blocking a client that doesn’t have a sidecar (e.g., a curl from a debug pod).

Fix: Either inject a sidecar into the debug pod, or add a port-level exception:

spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    9090:                # Health check port
      mode: DISABLE

Error: SSL_ERROR_RX_RECORD_TOO_LONG

Cause: The server is expecting TLS but the client sent plain HTTP (or vice versa).

Fix: Check if DestinationRule has the correct TLS mode:

kubectl get destinationrule -A -o yaml | grep -A 10 tls

General Debug Workflow

# 1. Check Envoy proxy logs for TLS errors
kubectl logs <pod-name> -c istio-proxy | grep -i "tls\|cert\|handshake"

# 2. Get full mTLS status for a pod
istioctl x describe pod <pod-name>.<namespace>

# 3. Inspect what certificates Envoy holds
kubectl exec -it <pod-name> -n <namespace> -c istio-proxy -- \
  curl -s localhost:15000/certs | python3 -m json.tool

# 4. Check cluster-level TLS configuration
istioctl proxy-config cluster <pod-name>.<namespace> --fqdn billing-service.payments.svc.cluster.local

# 5. Test TLS handshake from a debug pod
kubectl run debug --image=nicolaka/netshoot -it --rm -- \
  openssl s_client -connect billing-service.payments.svc.cluster.local:8080 \
  -cert /tmp/client.crt -key /tmp/client.key -CAfile /tmp/ca.crt

Certificate Lifecycle Best Practices

Production Rollout Checklist

Before switching to STRICT mTLS, validate each step in a staging environment:

# 1. Confirm all pods have sidecars injected
kubectl get pods -n payments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'

# 2. Check no services are using host networking (bypasses Envoy)
kubectl get pods -n payments -o jsonpath='{range .items[?(@.spec.hostNetwork==true)]}{.metadata.name}{"\n"}{end}'

# 3. Verify no hardcoded IP connections (these bypass service discovery and mTLS)
# Review application configs for direct IP references

# 4. Test with PERMISSIVE first, monitor for errors, then switch to STRICT
kubectl apply -f peer-auth-strict.yaml

# 5. Monitor Istio metrics for TLS handshake failures
kubectl exec -it <pod> -c istio-proxy -- curl -s localhost:15090/stats | grep ssl.handshake

Internal Linking

For token-based authentication in your APIs alongside mTLS, see the Client Credentials Flow in OAuth 2.0 guide — mTLS client authentication (token_endpoint_auth_method: tls_client_auth) is a supported OAuth 2.0 client authentication method (RFC 8705) and eliminates the need for client secrets entirely.

For workload identity that spans beyond Kubernetes into AWS, GCP, and Azure, see our guide on Workload Identity Federation — mTLS certificates can serve as the attestation mechanism in cross-cloud identity federation.

For the Kubernetes security layer above mTLS, the Kubernetes Service Account Security article covers projected tokens and IRSA patterns that complement mTLS-based service authentication.