On this page

API Credential Rotation

API credential rotation replaces active authentication secrets with new values before the originals expire, become compromised, or exceed their maximum permitted age. You perform this task on a regular schedule determined by credential type and risk classification, and on an emergency basis when compromise is suspected or confirmed.

The rotation procedure requires coordination between the system that issues credentials, the applications that consume them, and the secure storage that holds them. A successful rotation updates all three components in sequence, verifies continued operation, and documents the change for audit purposes.

Prerequisites

Before beginning credential rotation, confirm the following requirements are satisfied.

Requirement	Detail
Credential inventory	Complete list of all API credentials including issuing system, consuming applications, storage location, and last rotation date
Access rights	Administrative access to credential issuing systems; write access to secrets management system; deployment permissions for consuming applications
Dependency map	Documentation of which applications and integrations depend on each credential
Rollback capability	Ability to restore previous credential values within 15 minutes if rotation fails
Maintenance window	Scheduled time for rotation if the credential requires service restart (not needed for credentials supporting zero-downtime rotation)
Verification procedures	Documented tests to confirm each integration functions after rotation

Verify your credential inventory is current by comparing against your secrets management system:

# List all secrets with metadata
vault kv list -format=json secret/api-credentials/ | jq -r '.[]' | while read key; do
  vault kv metadata get -format=json "secret/api-credentials/$key" | \
    jq -r --arg key "$key" '[$key, .data.created_time, .data.current_version] | @tsv'
done

Expected output shows each credential with creation date and current version number:

payment-gateway-api    2024-08-15T10:30:00Z    4
crm-integration        2024-09-22T14:15:00Z    2
monitoring-service     2024-07-01T09:00:00Z    7

Credentials with creation dates older than your rotation policy maximum require immediate scheduling.

Credential Types and Rotation Schedules

Different credential types carry different risk profiles and support different rotation mechanisms. Your rotation schedule reflects these differences.

API keys are static strings that authenticate requests to external services. Most API keys support instant rotation where you generate a new key, update consumers, then revoke the old key. Rotate API keys every 90 days for standard integrations and every 30 days for credentials accessing financial or personal data.

OAuth client secrets authenticate your application to OAuth providers during token exchange. Rotating these secrets requires updating your application configuration and restarting services that hold the secret in memory. Rotate OAuth client secrets every 180 days.

Service account tokens authenticate automated processes to internal and cloud services. Many platforms support automatic token rotation through short-lived tokens issued from longer-lived refresh credentials. Where automatic rotation is not available, rotate service account credentials every 90 days.

Database connection credentials authenticate applications to database servers. These typically require application restart to pick up new values. Rotate database credentials every 90 days, scheduling rotation during maintenance windows.

Webhook signing secrets verify the authenticity of incoming webhooks. Both your system and the sending system must update simultaneously, making these the most coordination-intensive credentials to rotate. Rotate webhook secrets every 180 days with explicit coordination.

+--------------------------------------------------------------------+
|                    CREDENTIAL ROTATION SCHEDULE                    |
+--------------------------------------------------------------------+
|                                                                    |
|  CREDENTIAL TYPE              STANDARD      HIGH-RISK   EMERGENCY  |
|  ----------------------------------------------------------------  |
|  API keys                     90 days       30 days     Immediate  |
|  OAuth client secrets         180 days      90 days     Immediate  |
|  Service account tokens       90 days       30 days     Immediate  |
|  Database credentials         90 days       30 days     4 hours    |
|  Webhook signing secrets      180 days      90 days     24 hours   |
|                                                                    |
+--------------------------------------------------------------------+

High-risk credentials are those accessing financial systems, personal data, or systems with regulatory requirements. Emergency rotation timelines indicate maximum acceptable delay when compromise is confirmed.

Procedure

Standard Scheduled Rotation

Perform scheduled rotation during low-traffic periods. For credentials requiring service restart, coordinate with affected teams 48 hours in advance.

Identify the credential to rotate and retrieve its dependency map:

   # Query credential metadata
   vault kv get -format=json secret/api-credentials/payment-gateway-api | \
     jq '.data.data.metadata'

Expected output shows consuming applications:

   {
     "issuer": "payment-provider.example.com",
     "consumers": ["web-app", "mobile-backend", "batch-processor"],
     "rotation_contact": "payments-team@example.org",
     "last_rotated": "2024-08-15T10:30:00Z"
   }

Confirm all listed consumers are still active. Remove any decommissioned applications from the consumer list before proceeding.

Generate the new credential value from the issuing system. The exact procedure varies by provider.
For services with API-based key management:

   # Generate new API key (example: generic REST API)
   curl -X POST https://api.provider.example.com/v1/api-keys \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"name": "production-integration", "permissions": ["read", "write"]}' \
     | jq -r '.api_key'

For cloud provider service accounts:

   # AWS: Create new access key for service account
   aws iam create-access-key --user-name integration-service-account \
     --query 'AccessKey.[AccessKeyId,SecretAccessKey]' --output text

   # Azure: Reset service principal credentials
   az ad sp credential reset --id $SERVICE_PRINCIPAL_ID --query password --output tsv

   # GCP: Create new service account key
   gcloud iam service-accounts keys create new-key.json \
     --iam-account=integration@project-id.iam.gserviceaccount.com

Store the new credential value temporarily in a secure location. Do not commit to version control or transmit over unencrypted channels.

Update your secrets management system with the new credential while preserving the old value:

   # Store new credential as a new version (Vault example)
   vault kv put secret/api-credentials/payment-gateway-api \
     api_key="$NEW_API_KEY" \
     previous_key="$OLD_API_KEY" \
     rotated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     rotated_by="$USER"

For systems using AWS Secrets Manager:

   # Update secret with staging label for new value
   aws secretsmanager put-secret-value \
     --secret-id payment-gateway-api \
     --secret-string "$NEW_API_KEY" \
     --version-stages AWSPENDING

Preserving the old value enables rapid rollback if the new credential fails verification.

Deploy the new credential to consuming applications. For applications that read credentials at startup, this requires a restart. For applications that fetch credentials dynamically, this requires cache invalidation or wait for cache expiry.
For Kubernetes deployments using external secrets:

   # Trigger secret refresh
   kubectl annotate externalsecret payment-gateway-secret \
     force-sync=$(date +%s) --overwrite

   # Verify secret updated
   kubectl get secret payment-gateway-secret -o jsonpath='{.metadata.annotations.version}'

   # Rolling restart of consuming deployments
   kubectl rollout restart deployment/web-app
   kubectl rollout restart deployment/mobile-backend
   kubectl rollout restart deployment/batch-processor

   # Wait for rollout completion
   kubectl rollout status deployment/web-app --timeout=300s

For applications using environment variables:

   # Update environment and restart (systemd example)
   sudo sed -i "s/^PAYMENT_API_KEY=.*/PAYMENT_API_KEY=$NEW_API_KEY/" \
     /etc/application/environment
   sudo systemctl restart application.service

Verify all consuming applications function correctly with the new credential. Run integration tests or execute verification requests for each consumer:

   # Test web application integration
   curl -f https://app.example.org/api/health/payment-integration
   # Expected: HTTP 200 with {"status": "healthy", "payment_gateway": "connected"}

   # Test batch processor
   kubectl logs deployment/batch-processor --tail=50 | grep -i "payment"
   # Expected: No authentication errors in recent logs

   # Execute test transaction (if supported)
   curl -X POST https://app.example.org/api/payments/test \
     -H "Authorization: Bearer $TEST_TOKEN" \
     -d '{"amount": 0, "test": true}'
   # Expected: {"status": "success", "test_mode": true}

Verification must succeed for all consumers before proceeding to revocation.

Revoke the old credential after verification confirms the new credential works:

   # Revoke old API key at provider
   curl -X DELETE https://api.provider.example.com/v1/api-keys/$OLD_KEY_ID \
     -H "Authorization: Bearer $ADMIN_TOKEN"

   # For AWS access keys
   aws iam delete-access-key --user-name integration-service-account \
     --access-key-id $OLD_ACCESS_KEY_ID

   # For GCP service account keys
   gcloud iam service-accounts keys delete $OLD_KEY_ID \
     --iam-account=integration@project-id.iam.gserviceaccount.com

Some providers automatically revoke old keys after a grace period when new keys are activated. Verify your provider’s behaviour and revoke explicitly if automatic revocation is not enabled.

Update credential metadata and audit records:

   # Update metadata to remove previous key reference
   vault kv patch secret/api-credentials/payment-gateway-api \
     previous_key="" \
     rotation_verified_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

   # Log rotation event to audit system
   logger -p auth.info "API credential rotated: payment-gateway-api by $USER"

Notify stakeholders that rotation completed successfully:

   # Send notification (adapt to your notification system)
   curl -X POST https://chat.example.org/webhook/security-notifications \
     -H "Content-Type: application/json" \
     -d '{
       "text": "✓ Credential rotation complete: payment-gateway-api",
       "details": "Rotated by '"$USER"' at '"$(date)"'. All integrations verified."
     }'

Emergency Rotation

Emergency rotation bypasses normal scheduling when credential compromise is confirmed or suspected. Speed takes priority, but verification remains mandatory to avoid replacing a security incident with a service outage.

+------------------------------------------------------------------+
|                   EMERGENCY ROTATION FLOW                        |
+------------------------------------------------------------------+
|                                                                  |
|   COMPROMISE       GENERATE        UPDATE         VERIFY         |
|   DETECTED         NEW CRED        CONSUMERS      FUNCTION       |
|       |               |               |              |           |
|       v               v               v              v           |
|   +-------+       +-------+       +-------+      +-------+       |
|   |Assess |       |Create |       |Deploy |      |Test   |       |
|   |scope  +------>|new    +------>|to all +----->|all    |       |
|   |       |       |secret |       |apps   |      |paths  |       |
|   +---+---+       +-------+       +-------+      +---+---+       |
|       |                                              |           |
|       |           REVOKE OLD          DOCUMENT       |           |
|       |               |                   |          |           |
|       |               v                   v          |           |
|       |           +-------+           +-------+      |           |
|       |           |Delete |           |Audit  |<-----+           |
|       +---------->|old    +---------->|trail  |                  |
|        parallel   |cred   |           |       |                  |
|                   +-------+           +-------+                  |
|                                                                  |
+------------------------------------------------------------------+

Assess the compromise scope immediately upon detection:

   # Check when credential was last used legitimately
   vault audit log -format=json | jq -r \
     'select(.request.path == "secret/api-credentials/payment-gateway-api") |
      [.time, .request.operation, .request.remote_address] | @tsv' | tail -20

   # Check provider logs for unauthorized access
   curl -s https://api.provider.example.com/v1/audit-log \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     --data-urlencode "api_key_id=$COMPROMISED_KEY_ID" \
     --data-urlencode "since=2024-01-01T00:00:00Z" | jq '.entries[-20:]'

If unauthorized access is confirmed, record the earliest unauthorized timestamp for incident documentation.

Generate a new credential immediately following the standard generation procedure from step 2 above. Do not delay for coordination.
Revoke the compromised credential before deploying the replacement. This differs from scheduled rotation where revocation happens last. Immediate revocation stops ongoing unauthorized access even if it temporarily breaks integrations:

   # Revoke compromised credential immediately
   curl -X DELETE https://api.provider.example.com/v1/api-keys/$COMPROMISED_KEY_ID \
     -H "Authorization: Bearer $ADMIN_TOKEN"

   # Record revocation time
   echo "Compromised credential revoked at $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> incident-log.txt

Deploy the new credential to all consumers with maximum parallelism:

   # Update secrets store
   vault kv put secret/api-credentials/payment-gateway-api \
     api_key="$NEW_API_KEY" \
     emergency_rotation="true" \
     rotated_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     incident_reference="INC-2024-0042"

   # Parallel restart of all consumers
   kubectl rollout restart deployment/web-app &
   kubectl rollout restart deployment/mobile-backend &
   kubectl rollout restart deployment/batch-processor &
   wait

Verify all integrations as in scheduled rotation step 5. Emergency conditions do not excuse skipping verification.
Document the emergency rotation as part of the security incident record:

   # Append to incident documentation
   cat >> incident-log.txt << EOF

   Emergency Credential Rotation
   Credential: payment-gateway-api
   Revoked: $(date -u +%Y-%m-%dT%H:%M:%SZ)
   New credential deployed: $(date -u +%Y-%m-%dT%H:%M:%SZ)
   Verification complete: [pending/complete]
   Rotated by: $USER
   EOF

Webhook Secret Rotation

Webhook secrets require bilateral coordination because both the sending and receiving systems must use the same secret. Rotation follows a handshake pattern.

Configure your receiving system to accept signatures from both old and new secrets temporarily:

   # Example: webhook verification with dual-secret support
   import hmac
   import hashlib

   def verify_webhook(payload, signature, secrets):
       """Verify webhook against multiple valid secrets during rotation."""
       for secret in secrets:
           expected = hmac.new(
               secret.encode(),
               payload.encode(),
               hashlib.sha256
           ).hexdigest()
           if hmac.compare_digest(expected, signature):
               return True
       return False

   # During rotation, accept both secrets
   WEBHOOK_SECRETS = [
       os.environ['WEBHOOK_SECRET_NEW'],
       os.environ['WEBHOOK_SECRET_OLD']
   ]

Deploy this dual-secret configuration before notifying the sender.

Generate the new webhook secret and update your receiving application:

   # Generate cryptographically secure secret
   NEW_WEBHOOK_SECRET=$(openssl rand -hex 32)

   # Store in secrets management
   vault kv put secret/webhooks/provider-events \
     current_secret="$NEW_WEBHOOK_SECRET" \
     previous_secret="$OLD_WEBHOOK_SECRET" \
     rotation_phase="dual-accept"

   # Deploy to receiver
   kubectl set env deployment/webhook-receiver \
     WEBHOOK_SECRET_NEW="$NEW_WEBHOOK_SECRET" \
     WEBHOOK_SECRET_OLD="$OLD_WEBHOOK_SECRET"

Update the sending system with the new secret. This procedure varies by provider:

   # API-based webhook secret update
   curl -X PATCH https://api.provider.example.com/v1/webhooks/$WEBHOOK_ID \
     -H "Authorization: Bearer $ADMIN_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"signing_secret": "'"$NEW_WEBHOOK_SECRET"'"}'

For providers without API access, update through their administrative console.

Verify webhooks are received and validated with the new secret:

   # Trigger a test webhook if provider supports it
   curl -X POST https://api.provider.example.com/v1/webhooks/$WEBHOOK_ID/test \
     -H "Authorization: Bearer $ADMIN_TOKEN"

   # Check receiver logs for successful verification
   kubectl logs deployment/webhook-receiver --tail=20 | grep "webhook verified"

Remove the old secret from your receiving application after confirming the sender uses the new secret:

   # Update to single-secret mode
   kubectl set env deployment/webhook-receiver \
     WEBHOOK_SECRET_NEW="$NEW_WEBHOOK_SECRET" \
     WEBHOOK_SECRET_OLD-

   vault kv patch secret/webhooks/provider-events \
     previous_secret="" \
     rotation_phase="complete"

Integration Dependency Mapping

Maintaining accurate dependency maps prevents rotation failures from surprising dependent applications. Your credential inventory must record every application that consumes each credential.

+------------------------------------------------------------------+
|                 CREDENTIAL DEPENDENCY MAP                        |
+------------------------------------------------------------------+
|                                                                  |
|  payment-gateway-api                                             |
|  +------------------+                                            |
|  | Issuer:          |                                            |
|  | payment.example  +---+                                        |
|  +------------------+   |                                        |
|                         |                                        |
|    +--------------------+--------------------+                   |
|    |                    |                    |                   |
|    v                    v                    v                   |
|  +---------+      +-----------+      +-------------+             |
|  | web-app |      | mobile-   |      | batch-      |             |
|  |         |      | backend   |      | processor   |             |
|  +---------+      +-----------+      +-------------+             |
|    |                    |                    |                   |
|    | reads at           | reads at           | reads at          |
|    | startup            | startup            | scheduled         |
|    |                    |                    | job start         |
|    v                    v                    v                   |
|  +---------+      +-----------+      +-------------+             |
|  | restart |      | restart   |      | next job    |             |
|  | required|      | required  |      | uses new    |             |
|  +---------+      +-----------+      +-------------+             |
|                                                                  |
+------------------------------------------------------------------+

Query your deployment configurations to build and verify dependency maps:

# Find all Kubernetes deployments referencing a secret
SECRET_NAME="payment-gateway-api"
kubectl get deployments --all-namespaces -o json | jq -r \
  '.items[] |
   select(.spec.template.spec.containers[].env[]?.valueFrom.secretKeyRef.name == "'"$SECRET_NAME"'") |
   [.metadata.namespace, .metadata.name] | @tsv'

# Find all environment variable references
kubectl get deployments --all-namespaces -o json | jq -r \
  '.items[] |
   select(.spec.template.spec.containers[].envFrom[]?.secretRef.name == "'"$SECRET_NAME"'") |
   [.metadata.namespace, .metadata.name] | @tsv'

Update your credential metadata when deploying new consumers or decommissioning old ones. Stale dependency maps cause rotation failures when forgotten applications lose authentication.

Verification

After completing rotation, verify complete success before closing the rotation task.

Confirm the new credential is active at the issuing system:

# Verify new key exists and is active
curl -s https://api.provider.example.com/v1/api-keys \
  -H "Authorization: Bearer $ADMIN_TOKEN" | \
  jq '.keys[] | select(.id == "'"$NEW_KEY_ID"'") | {id, status, created}'

# Expected output
{
  "id": "key_abc123...",
  "status": "active",
  "created": "2024-11-15T14:30:00Z"
}

Confirm the old credential is revoked:

# Attempt to use old credential (should fail)
curl -s -o /dev/null -w "%{http_code}" \
  https://api.provider.example.com/v1/test \
  -H "Authorization: Bearer $OLD_API_KEY"

# Expected: 401 or 403

Confirm all consuming applications authenticate successfully:

# Health check endpoints for each consumer
for app in web-app mobile-backend batch-processor; do
  echo -n "$app: "
  kubectl exec deployment/$app -- curl -sf localhost:8080/health/external-services | \
    jq -r '.payment_gateway'
done

# Expected output for each:
# web-app: connected
# mobile-backend: connected
# batch-processor: connected

Confirm secrets management reflects the rotation:

vault kv get -format=json secret/api-credentials/payment-gateway-api | \
  jq '{version: .metadata.version, rotated_at: .data.data.rotated_at, has_previous: (.data.data.previous_key != "")}'

# Expected after old key revoked:
{
  "version": 5,
  "rotated_at": "2024-11-15T14:30:00Z",
  "has_previous": false
}

Confirm audit trail records the rotation:

vault audit log -format=json | jq -r \
  'select(.request.path == "secret/api-credentials/payment-gateway-api" and
          .request.operation == "update") |
   [.time, .auth.display_name] | @tsv' | tail -1

# Expected: timestamp and username of person who performed rotation

Troubleshooting

Symptom	Cause	Resolution
Application returns 401 after rotation	New credential not deployed to application	Verify secret sync completed; check external-secrets operator logs; trigger manual sync with `kubectl annotate externalsecret <name> force-sync=$(date +%s)`
Application returns 401 but credential is correct	Credential cached in memory	Restart the application pod or process; for connection pools, may require pool drain
Some requests succeed, others fail	Rolling restart incomplete	Check rollout status with `kubectl rollout status`; wait for all pods to restart
Provider rejects new credential immediately	Credential format or permissions incorrect	Verify credential was generated with correct permissions; check provider documentation for required scopes
Cannot revoke old credential	Credential still in use by unknown consumer	Query provider access logs to identify consumer; update dependency map; deploy new credential to missing consumer
Vault returns permission denied	Token lacks update capability	Request elevated token or have secrets administrator perform rotation
New credential works in test but fails in production	Environment-specific configuration	Verify credential is deployed to production namespace; check environment variable names match
Webhook verification fails after rotation	Sender still using old secret	Confirm sender received update; check sender’s webhook configuration; extend dual-secret acceptance period
Integration test times out	Service not yet restarted or healthy	Wait for rollout completion; check pod logs for startup errors
Rotation succeeds but monitoring alerts	Monitoring using old credential	Update monitoring configuration; restart monitoring agents
Cannot generate new credential at provider	Rate limit exceeded	Wait for rate limit reset (check provider headers for reset time); request limit increase for rotation operations
Batch jobs fail after rotation	Job started before credential deployed	Jobs in progress may hold old credential in memory; wait for job completion or terminate and retry
Old credential appears active after revocation	Provider propagation delay	Some providers take 5-15 minutes to propagate revocation; verify after propagation delay
Application crashes after receiving new credential	Credential format incompatible	Check credential encoding (base64 vs plaintext); verify application parses credential type correctly

Automation

Automate scheduled rotation to reduce manual effort and ensure consistency. The following patterns support different infrastructure configurations.

For Kubernetes environments, External Secrets Operator combined with provider-specific credential rotation automates the full cycle:

# ExternalSecret with rotation annotation
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-gateway-api
  annotations:
    # Refresh every 24 hours
    external-secrets.io/refresh-interval: "24h"
spec:
  refreshInterval: 24h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payment-gateway-api
    creationPolicy: Owner
  data:
    - secretKey: api_key
      remoteRef:
        key: secret/api-credentials/payment-gateway-api
        property: api_key

For HashiCorp Vault, database secrets engines handle rotation automatically for supported databases:

# Configure database secrets engine with rotation
vault write database/config/production-postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.example.org:5432/production" \
  allowed_roles="app-readonly,app-readwrite" \
  username="vault-admin" \
  password="$VAULT_DB_PASSWORD"

# Create role with automatic rotation
vault write database/roles/app-readonly \
  db_name=production-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                       GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Applications request short-lived credentials that Vault rotates automatically:

# Application requests database credentials
vault read database/creds/app-readonly

# Returns time-limited credentials
Key                Value
---                -----
lease_id           database/creds/app-readonly/abc123
lease_duration     1h
username           v-app-readonly-xyz789
password           A1b2C3d4E5f6...

For cloud provider service accounts, configure workload identity where available to eliminate static credentials entirely:

# GKE Workload Identity configuration
apiVersion: v1
kind: ServiceAccount
metadata:
  name: integration-service
  annotations:
    iam.gke.io/gcp-service-account: integration@project-id.iam.gserviceaccount.com
---
# Pods using this ServiceAccount authenticate as the GCP service account
# without static credentials

Schedule rotation reminders for credentials that cannot be automated:

# Cron job to check for credentials approaching rotation deadline
0 9 * * 1 /usr/local/bin/check-credential-expiry.sh

#!/bin/bash
DAYS_WARNING=14

vault kv list -format=json secret/api-credentials/ | jq -r '.[]' | while read key; do
  rotated_at=$(vault kv get -format=json "secret/api-credentials/$key" | \
    jq -r '.data.data.rotated_at // "1970-01-01T00:00:00Z"')

  days_since=$(( ($(date +%s) - $(date -d "$rotated_at" +%s)) / 86400 ))

  # Assume 90-day rotation policy
  if [ $days_since -gt 76 ]; then
    echo "WARNING: $key last rotated $days_since days ago"
  fi
done

Automation limitations

Automated rotation requires robust monitoring. A failed automated rotation can cause outages if not detected immediately. Configure alerts for rotation failures and verify integrations after each automated rotation.