Certificate Management

Certificate management encompasses the ongoing maintenance of TLS/SSL certificates throughout their lifecycle, from issuance through renewal to revocation. These procedures ensure encrypted communications remain functional by preventing certificate expiry, which causes immediate service disruption when browsers and clients reject connections to endpoints presenting expired credentials.

Prerequisites

Before performing certificate management tasks, verify the following requirements are met:

Confirm certbot installation meets version requirements:

certbot --version
# Required: certbot 2.0.0 or higher
# If older or missing:
sudo apt update && sudo apt install certbot python3-certbot-nginx

Verify OpenSSL version supports modern cipher requirements:

openssl version
# Required: OpenSSL 1.1.1 or higher (for TLS 1.3 support)
# LibreSSL 3.3.0+ is acceptable on macOS

Certificate Inventory Maintenance

The certificate inventory forms the foundation of certificate management by documenting every certificate the organisation relies upon. Without accurate inventory, certificates expire without warning because monitoring systems cannot track unknown certificates.

Building the Initial Inventory

1. Scan external-facing domains to discover certificates in use:

   # Scan a domain and extract certificate details
   echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
     openssl x509 -noout -subject -issuer -dates -fingerprint

Expected output shows certificate metadata:

   subject=CN = example.org
   issuer=C = US, O = Let's Encrypt, CN = R3
   notBefore=Nov 15 00:00:00 2024 GMT
   notAfter=Feb 13 23:59:59 2025 GMT
   SHA1 Fingerprint=A1:B2:C3:D4:E5:F6:...

2. Query internal certificate stores for certificates not exposed externally:

   # List certificates in system trust store (Debian/Ubuntu)
   ls -la /etc/ssl/certs/ | head -20

   # List certificates managed by certbot
   sudo certbot certificates

3. Document each certificate with required metadata:

   Domain: example.org, www.example.org
   Issuer: Let's Encrypt
   Type: DV (Domain Validated)
   Expiry: 2025-02-13
   Renewal: Automated (certbot)
   Server: web-prod-01.internal
   Purpose: Public website
   Owner: Web Team

4. Record certificates in the configuration management database or dedicated inventory system. Each entry requires: common name, subject alternative names, issuer, expiry date, renewal method, server locations, and responsible team.

Ongoing Inventory Updates

Review the certificate inventory monthly. Add new certificates when services launch, update entries when certificates renew, and remove entries when services decommission. Integrate inventory updates into change management by requiring certificate documentation for any new service deployment.

+---------------------------------------------------------------------+
|                       CERTIFICATE INVENTORY FLOW                    |
+---------------------------------------------------------------------+
|                                                                     |
|  +------------------+     +------------------+     +--------------+ |
|  | New Service      |     | Certificate      |     | Inventory    | |
|  | Deployment       +---->| Request/Issue    +---->| Update       | |
|  |                  |     |                  |     |              | |
|  +------------------+     +------------------+     +------+-------+ |
|                                                           |         |
|  +------------------+     +------------------+            |         |
|  | Service          |     | Inventory        |<-----------+         |
|  | Decommission     +---->| Removal          |                      |
|  |                  |     |                  |                      |
|  +------------------+     +------------------+                      |
|                                                                     |
|  +------------------+     +------------------+     +--------------+ |
|  | Automated        |     | Renewal          |     | Inventory    | |
|  | Renewal Runs     +---->| Verification     +---->| Expiry       | |
|  |                  |     |                  |     | Update       | |
|  +------------------+     +------------------+     +--------------+ |
|                                                                     |
+---------------------------------------------------------------------+

Figure 1: Certificate inventory maintenance integrated with service lifecycle

Expiry Monitoring Configuration

Certificate monitoring detects approaching expiry dates and alerts administrators before services fail. Configure monitoring to alert at 30 days (informational), 14 days (warning), and 7 days (critical) before expiry.

Configuring Monitoring Checks

1. Create a monitoring script that checks certificate expiry:

   #!/bin/bash
   # check_cert_expiry.sh - Check certificate expiry for a domain

   DOMAIN=$1
   WARN_DAYS=${2:-14}
   CRIT_DAYS=${3:-7}

   EXPIRY=$(echo | openssl s_client -servername "$DOMAIN" -connect "$DOMAIN":443 2>/dev/null | \
     openssl x509 -noout -enddate | cut -d= -f2)

   EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
   NOW_EPOCH=$(date +%s)
   DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))

   if [ $DAYS_LEFT -lt $CRIT_DAYS ]; then
     echo "CRITICAL: $DOMAIN certificate expires in $DAYS_LEFT days"
     exit 2
   elif [ $DAYS_LEFT -lt $WARN_DAYS ]; then
     echo "WARNING: $DOMAIN certificate expires in $DAYS_LEFT days"
     exit 1
   else
     echo "OK: $DOMAIN certificate expires in $DAYS_LEFT days"
     exit 0
   fi

2. Deploy the script to your monitoring system. For Nagios/Icinga:

   command[check_cert_example]=/usr/local/bin/check_cert_expiry.sh example.org 14 7
   command[check_cert_api]=/usr/local/bin/check_cert_expiry.sh api.example.org 14 7

3. Configure alerting thresholds in your monitoring platform:

   # Prometheus alerting rule example
   groups:
     - name: certificates
       rules:
         - alert: CertificateExpirySoon
           expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 14
           for: 1h
           labels:
             severity: warning
           annotations:
             summary: "Certificate expires within 14 days"

         - alert: CertificateExpiryCritical
           expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
           for: 1h
           labels:
             severity: critical
           annotations:
             summary: "Certificate expires within 7 days"

4. Test monitoring by verifying alerts trigger correctly. Create a test certificate with short validity to confirm the alerting pipeline functions.

Automated Renewal with ACME

The Automatic Certificate Management Environment (ACME) protocol enables automated certificate issuance and renewal without manual intervention. Let’s Encrypt certificates issued via ACME have 90-day validity and renew automatically when configured correctly.

Initial ACME Configuration

1. Register with the ACME provider and accept terms of service:

   sudo certbot register --email security@example.org --agree-tos --no-eff-email

This creates an account at /etc/letsencrypt/accounts/ used for all future certificate operations.

2. Request an initial certificate using the appropriate challenge method. For HTTP-01 challenge with Nginx:

   sudo certbot certonly --nginx -d example.org -d www.example.org

For DNS-01 challenge (required for wildcards):

   sudo certbot certonly --manual --preferred-challenges dns \
     -d example.org -d "*.example.org"

When prompted, create the DNS TXT record:

   _acme-challenge.example.org.  300  IN  TXT  "gfj9Xq...Rg85nM"

3. Verify the certificate installed correctly:

   sudo certbot certificates

Expected output:

   Certificate Name: example.org
     Serial Number: 4a8b...
     Key Type: ECDSA
     Domains: example.org www.example.org
     Expiry Date: 2025-02-13 (VALID: 89 days)
     Certificate Path: /etc/letsencrypt/live/example.org/fullchain.pem
     Private Key Path: /etc/letsencrypt/live/example.org/privkey.pem

4. Configure the web server to use the certificate. For Nginx:

   server {
       listen 443 ssl http2;
       server_name example.org www.example.org;

       ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;

       # Modern TLS configuration
       ssl_protocols TLSv1.2 TLSv1.3;
       ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
       ssl_prefer_server_ciphers off;
   }

5. Test the configuration and reload:

   sudo nginx -t && sudo systemctl reload nginx

Automated Renewal Setup

Certbot installs a systemd timer or cron job during installation that attempts renewal twice daily. Verify this automation is active and functioning.

1. Check the renewal timer status:

   sudo systemctl status certbot.timer

Expected output shows active timer:

   ● certbot.timer - Run certbot twice daily
        Loaded: loaded (/lib/systemd/system/certbot.timer; enabled)
        Active: active (waiting) since Mon 2024-11-18 00:00:00 UTC
       Trigger: Mon 2024-11-18 12:00:00 UTC; 5h left

If inactive, enable it:

   sudo systemctl enable --now certbot.timer

2. Configure renewal hooks to reload services after certificate updates. Create a deploy hook:

   sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy
   sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh << 'EOF'
   #!/bin/bash
   systemctl reload nginx
   EOF
   sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

3. Test renewal with a dry run:

   sudo certbot renew --dry-run

Expected output ends with:

   Congratulations, all simulated renewals succeeded:
     /etc/letsencrypt/live/example.org/fullchain.pem (success)

4. Force a renewal to verify the complete process (only if certificate is within renewal window or use --force-renewal):

   sudo certbot renew --force-renewal --cert-name example.org

5. Verify the renewed certificate is served:

   echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
     openssl x509 -noout -dates

The notBefore date should reflect the renewal time.

+------------------------------------------------------------------------------+
|                          AUTOMATED RENEWAL FLOW                              |
+------------------------------------------------------------------------------+
|                                                                              |
|  +------------------+                                                        |
|  | Systemd Timer    |                                                        |
|  | (twice daily)    |                                                        |
|  +--------+---------+                                                        |
|           |                                                                  |
|           v                                                                  |
|  +--------+---------+     +------------------+                               |
|  | certbot renew    +---->| Check each cert  |                               |
|  |                  |     | expiry < 30 days?|                               |
|  +------------------+     +--------+---------+                               |
|                                    |                                         |
|                     +--------------+--------------+                          |
|                     |                             |                          |
|                     v                             v                          |
|            +--------+--------+           +--------+--------+                 |
|            | No: Skip        |           | Yes: Renew      |                 |
|            | (log only)      |           |                 |                 |
|            +-----------------+           +--------+--------+                 |
|                                                   |                          |
|                                                   v                          |
|                                          +--------+--------+                 |
|                                          | ACME challenge  |                 |
|                                          | (HTTP-01/DNS-01)|                 |
|                                          +--------+--------+                 |
|                                                   |                          |
|                                    +--------------+--------------+           |
|                                    |                             |           |
|                                    v                             v           |
|                           +--------+--------+           +--------+--------+  |
|                           | Success:        |           | Failure:        |  |
|                           | Deploy hook     |           | Alert           |  |
|                           | (reload nginx)  |           | admin           |  |
|                           +-----------------+           +-----------------+  |
|                                                                              |
+------------------------------------------------------------------------------+

Figure 2: Certbot automated renewal process with deploy hooks

Manual Renewal Procedures

Manual renewal is required when automated renewal fails, when using certificate authorities that do not support ACME, or when organisational policy requires manual control over certificate issuance.

Certificate Signing Request Generation

1. Generate a new private key and certificate signing request (CSR):

   openssl req -new -newkey rsa:2048 -nodes \
     -keyout example.org.key \
     -out example.org.csr \
     -subj "/CN=example.org/O=Example Organisation/C=GB"

For ECDSA keys (smaller, faster, recommended):

   openssl ecparam -genkey -name prime256v1 -out example.org.key
   openssl req -new -key example.org.key \
     -out example.org.csr \
     -subj "/CN=example.org/O=Example Organisation/C=GB"

2. Add Subject Alternative Names (SANs) for multiple domains. Create a configuration file:

   cat > san.cnf << EOF
   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   prompt = no

   [req_distinguished_name]
   CN = example.org
   O = Example Organisation
   C = GB

   [v3_req]
   subjectAltName = @alt_names

   [alt_names]
   DNS.1 = example.org
   DNS.2 = www.example.org
   DNS.3 = api.example.org
   EOF

Generate the CSR with SANs:

   openssl req -new -key example.org.key -out example.org.csr -config san.cnf

3. Verify the CSR contains correct information:

   openssl req -in example.org.csr -noout -text | grep -A1 "Subject:"
   openssl req -in example.org.csr -noout -text | grep -A4 "Subject Alternative Name"

4. Submit the CSR to your certificate authority through their portal or API. Retain the CSR file as some CAs require it for revocation.

5. Complete domain validation as required by the CA (DNS record, email verification, or HTTP file placement).

Certificate Installation

1. Download the issued certificate and intermediate certificates from the CA. You need:

   • The server certificate (your domain)
   • The intermediate certificate(s) (CA’s chain)
   • Optionally, the root certificate (usually in trust stores)

2. Create the certificate chain file by concatenating in order:

   cat example.org.crt intermediate.crt > example.org.chain.pem

The order matters: server certificate first, then intermediates, with root last (if included).

3. Verify the chain is complete:

   openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt example.org.chain.pem

Expected output:

   example.org.chain.pem: OK

4. Install the certificate and key with appropriate permissions:

   sudo cp example.org.chain.pem /etc/ssl/certs/
   sudo cp example.org.key /etc/ssl/private/
   sudo chmod 644 /etc/ssl/certs/example.org.chain.pem
   sudo chmod 600 /etc/ssl/private/example.org.key
   sudo chown root:root /etc/ssl/private/example.org.key

5. Update the web server configuration to reference the new certificate:

   ssl_certificate /etc/ssl/certs/example.org.chain.pem;
   ssl_certificate_key /etc/ssl/private/example.org.key;

6. Test and reload:

   sudo nginx -t && sudo systemctl reload nginx

Wildcard Certificate Considerations

Wildcard certificates secure a domain and all single-level subdomains using a certificate with *.example.org as the common name or SAN. A wildcard for *.example.org covers www.example.org, api.example.org, and mail.example.org but does not cover example.org itself (the apex domain) or multi-level subdomains like dev.api.example.org.

When to Use Wildcards

Wildcard certificates reduce management overhead when many subdomains exist, but they increase risk because a single compromised private key affects all covered subdomains. Use wildcards when:

• More than 5 subdomains share the same server infrastructure
• Subdomains are created dynamically and unpredictably
• Certificate management capacity is limited

Avoid wildcards when:

• Subdomains have different security requirements or owners
• Regulatory requirements mandate separate certificates per service
• The organisation has capacity for individual certificate management

Wildcard Issuance via ACME

Let’s Encrypt requires DNS-01 challenge for wildcard certificates because HTTP-01 cannot prove control over arbitrary subdomains.

1. Request the wildcard certificate with both the wildcard and apex domain:

   sudo certbot certonly --manual --preferred-challenges dns \
     -d example.org -d "*.example.org"

2. When prompted, create the DNS TXT record. The same record name is used twice for both domains:

   _acme-challenge.example.org.  300  IN  TXT  "abc123..."
   _acme-challenge.example.org.  300  IN  TXT  "def456..."

3. Verify DNS propagation before continuing:

   dig +short TXT _acme-challenge.example.org

Both values should appear.

4. For automated wildcard renewal, configure a DNS plugin. Example with Cloudflare:

   sudo apt install python3-certbot-dns-cloudflare

   # Create credentials file
   sudo mkdir -p /etc/letsencrypt/secrets
   sudo tee /etc/letsencrypt/secrets/cloudflare.ini << EOF
   dns_cloudflare_api_token = your-api-token-here
   EOF
   sudo chmod 600 /etc/letsencrypt/secrets/cloudflare.ini

   # Request with automated DNS
   sudo certbot certonly --dns-cloudflare \
     --dns-cloudflare-credentials /etc/letsencrypt/secrets/cloudflare.ini \
     -d example.org -d "*.example.org"

Private CA Management

Organisations operating internal services not exposed to the internet can issue certificates from a private certificate authority rather than public CAs. Private CAs eliminate external dependencies and enable certificate issuance for internal hostnames that public CAs cannot validate.

Creating a Private CA

1. Generate the CA private key with strong protection:

   openssl genrsa -aes256 -out ca.key 4096

Enter a strong passphrase when prompted. Store this passphrase in a secrets manager.

2. Create the CA certificate:

   openssl req -new -x509 -days 3650 -key ca.key \
     -out ca.crt \
     -subj "/CN=Example Organisation Internal CA/O=Example Organisation/C=GB"

The 10-year validity (3650 days) is typical for root CAs. Intermediate CAs use shorter periods.

3. Create a serial number file and database for tracking issued certificates:

   echo 1000 > ca.srl
   touch ca-index.txt

4. Store the CA key securely. The CA private key must be protected with hardware security modules (HSMs) in production environments or, at minimum, encrypted storage with restricted access. Compromise of the CA key compromises all certificates issued by that CA.

Issuing Certificates from Private CA

1. Generate a key and CSR for the internal service:

   openssl req -new -newkey rsa:2048 -nodes \
     -keyout internal-app.key \
     -out internal-app.csr \
     -subj "/CN=app.internal.example.org/O=Example Organisation/C=GB"

2. Sign the CSR with the CA:

   openssl x509 -req -days 365 \
     -in internal-app.csr \
     -CA ca.crt -CAkey ca.key \
     -CAserial ca.srl \
     -out internal-app.crt

Enter the CA key passphrase when prompted.

3. Verify the issued certificate:

   openssl x509 -in internal-app.crt -noout -text | head -20
   openssl verify -CAfile ca.crt internal-app.crt

4. Distribute the CA certificate to clients that need to trust certificates issued by this CA:

   # Debian/Ubuntu - add to system trust store
   sudo cp ca.crt /usr/local/share/ca-certificates/example-internal-ca.crt
   sudo update-ca-certificates

   # Verify addition
   ls /etc/ssl/certs/ | grep -i example

Certificate Revocation

Certificate revocation invalidates a certificate before its expiry date. Revoke certificates when private keys are compromised, when personnel with access leave the organisation, or when certificates are issued incorrectly.

Revoking Let’s Encrypt Certificates

1. Revoke using certbot with the certificate file:

   sudo certbot revoke --cert-path /etc/letsencrypt/live/example.org/cert.pem

Or using the private key:

   sudo certbot revoke --cert-path /etc/letsencrypt/live/example.org/cert.pem \
     --key-path /etc/letsencrypt/live/example.org/privkey.pem

2. Optionally delete the certificate files after revocation:

   sudo certbot delete --cert-name example.org

3. Issue a new certificate to replace the revoked one if the service should continue operating:

   sudo certbot certonly --nginx -d example.org -d www.example.org

Revocation for Private CA Certificates

1. Add the certificate to the CA’s revocation list:

   openssl ca -config ca.cnf -revoke internal-app.crt -keyfile ca.key -cert ca.crt

2. Generate an updated Certificate Revocation List (CRL):

   openssl ca -config ca.cnf -gencrl -out ca.crl -keyfile ca.key -cert ca.crt

3. Distribute the updated CRL to all systems that verify certificates against this CA. Systems must be configured to check CRLs:

   ssl_crl /etc/ssl/crl/ca.crl;

Emergency Certificate Issuance

Emergency issuance is required when certificates expire unexpectedly, when compromise necessitates immediate replacement, or when new services require certificates outside normal change windows.

1. Assess the situation:

   • Is the certificate expired or compromised?
   • What services are affected?
   • Is there an existing backup certificate?

2. For Let’s Encrypt, force immediate renewal:

   sudo certbot certonly --nginx --force-renewal -d example.org -d www.example.org

Rate limits apply: 5 duplicate certificates per week, 50 certificates per registered domain per week.

3. For commercial CAs, use expedited issuance if available. Most CAs offer emergency issuance with additional verification calls.

4. If DNS validation is not possible quickly, use HTTP-01 challenge:

   sudo certbot certonly --webroot -w /var/www/html \
     -d example.org -d www.example.org

5. Deploy the certificate and reload services:

   sudo nginx -t && sudo systemctl reload nginx

6. Verify the new certificate is serving:

   echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
     openssl x509 -noout -dates -serial

7. Update monitoring and inventory to reflect the new certificate.

Verification

After completing certificate operations, verify correct deployment:

Confirm the certificate chain is valid and complete:

# Check certificate validity
echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
  openssl x509 -noout -dates

# Expected output shows valid date range:
# notBefore=Nov 18 00:00:00 2024 GMT
# notAfter=Feb 16 23:59:59 2025 GMT

# Verify chain completeness
echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
  grep -E "^(depth|verify)"

# Expected: verify return:1 (chain validates correctly)

Confirm the correct certificate is serving:

# Compare certificate fingerprint with expected
echo | openssl s_client -servername example.org -connect example.org:443 2>/dev/null | \
  openssl x509 -noout -fingerprint -sha256

Test from an external perspective using online tools or a separate network:

# Using curl with verbose output
curl -vI https://example.org 2>&1 | grep -E "(SSL|subject|expire)"

Verify monitoring detects the updated expiry date:

# Run the monitoring check manually
/usr/local/bin/check_cert_expiry.sh example.org 14 7
# Expected: OK: example.org certificate expires in 89 days

Troubleshooting

For persistent issues, examine certbot logs:

sudo tail -100 /var/log/letsencrypt/letsencrypt.log

And web server error logs:

sudo tail -100 /var/log/nginx/error.log

See also

• Cryptography Standard for cipher suite and key length requirements
• Encryption Standards for encryption implementation guidance
• API Credential Rotation for related credential management
• Vulnerability Remediation for addressing certificate-related vulnerabilities