Skip to main content
On this page

Vulnerability Remediation

Vulnerability remediation converts scan findings into security improvements through prioritised fixing, mitigation, or documented acceptance. This task covers the complete remediation cycle from receiving vulnerability data through verification that fixes are effective, including handling systems that cannot be immediately patched due to operational constraints or field deployment.

Prerequisites

RequirementDetail
Vulnerability dataScan results from vulnerability management platform with CVSS scores and affected assets
Remediation authorityApproval to implement changes on affected systems, or escalation path for restricted systems
Asset inventory accessRead access to CMDB or asset management system showing system owners, criticality, and maintenance windows
Change managementAuthority to raise change requests, or pre-approved change window for security remediations
Testing capabilityAccess to non-production environment for patch testing, or documented test procedure
Rollback capabilityBackup or snapshot access for affected systems, tested restore procedure
Time allocation2-4 hours per critical vulnerability; 30-60 minutes per high vulnerability; batch processing for medium and low

Verify access to the vulnerability management platform and confirm you can view current scan results:

Terminal window
# Example: Query vulnerability scanner API for current findings
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/findings?status=open" | jq '.count'
# Expected: Returns count of open findings (e.g., 247)

Confirm you have the remediation tracking spreadsheet or ticketing system access:

Terminal window
# If using ticketing system, verify API access
curl -s -H "Authorization: Bearer $TICKET_API_TOKEN" \
  "https://tickets.example.org/api/projects/security-remediation/issues?status=open" | jq '.total'
# Expected: Returns count of open remediation tickets

Procedure

Prioritising vulnerabilities

  1. Export current vulnerability findings from the scanning platform, filtering for open status and grouping by severity:
Terminal window
   # Export findings to CSV for analysis
   curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
     "https://scanner.example.org/api/v2/findings?status=open&format=csv" \
     -o findings_$(date +%Y%m%d).csv


   # Count by severity
   awk -F',' 'NR>1 {severity[$5]++} END {for (s in severity) print s, severity[s]}' \
     findings_$(date +%Y%m%d).csv
   # Expected output:
   # Critical 3
   # High 24
   # Medium 156
   # Low 64

  1. Calculate the remediation priority score for each vulnerability by combining CVSS base score with asset criticality and exploitability factors. The remediation priority score determines the order in which vulnerabilities are addressed, accounting for factors beyond raw severity.

    The score uses this formula:

   Priority Score = CVSS Base × Asset Multiplier × Exploit Multiplier


   Where:
   - CVSS Base: 0.0 to 10.0 from scan results
   - Asset Multiplier: 1.5 for critical assets, 1.2 for important, 1.0 for standard
   - Exploit Multiplier: 2.0 if exploit available, 1.5 if PoC exists, 1.0 otherwise

For a vulnerability with CVSS 7.5 on a critical asset with known exploit:

   Priority Score = 7.5 × 1.5 × 2.0 = 22.5

For a vulnerability with CVSS 9.0 on a standard asset with no known exploit:

   Priority Score = 9.0 × 1.0 × 1.0 = 9.0

Despite the higher CVSS, the first vulnerability receives priority due to exploitability and asset criticality.

  1. Cross-reference vulnerabilities against threat intelligence to identify active exploitation. Check CISA Known Exploited Vulnerabilities (KEV) catalogue:
Terminal window
   # Download current KEV catalogue
   curl -s "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json" \
     -o kev_catalogue.json


   # Check if specific CVE is in catalogue
   jq -r '.vulnerabilities[] | select(.cveID=="CVE-2024-1234") | .cveID + " - " + .dueDate' \
     kev_catalogue.json
   # If present, output shows CVE and required remediation date
   # If absent, no output

Any vulnerability appearing in the KEV catalogue receives automatic critical priority regardless of calculated score. CISA mandates federal agencies remediate KEV entries within specified timeframes, and this timeline provides a reasonable baseline for all organisations.

  1. Generate the prioritised remediation queue by sorting all vulnerabilities by priority score descending:
Terminal window
   # Example prioritisation script output
   cat << 'EOF'
   PRIORITY QUEUE - Generated 2024-11-16


   IMMEDIATE (Priority Score > 20 or KEV listed):
   1. CVE-2024-1234  Score: 22.5  Asset: finance-db-01     Action: Patch
   2. CVE-2024-5678  Score: KEV   Asset: web-proxy-01      Action: Patch (KEV due: 2024-11-20)


   HIGH (Priority Score 15-20):
   3. CVE-2024-2345  Score: 18.0  Asset: auth-server-02    Action: Patch
   4. CVE-2024-3456  Score: 16.5  Asset: mail-gateway-01   Action: Mitigate


   STANDARD (Priority Score 10-15):
   5-12. [8 vulnerabilities - batch remediation scheduled]


   QUEUE (Priority Score < 10):
   13-247. [235 vulnerabilities - standard maintenance cycle]
   EOF

The prioritisation matrix below shows how CVSS severity, asset criticality, and exploitability combine to determine remediation urgency:

+--------------------------------------------------------------------+
|                      PRIORITISATION MATRIX                         |
+--------------------------------------------------------------------+
|                                                                    |
|  CVSS Score      Asset Criticality    Exploit Status    Timeline   |
|  ----------------------------------------------------------------  |
|                                                                    |
|  9.0-10.0        Critical             Known exploit     24 hours   |
|  9.0-10.0        Critical             PoC available     48 hours   |
|  9.0-10.0        Critical             None known        7 days     |
|  9.0-10.0        Standard             Known exploit     48 hours   |
|  9.0-10.0        Standard             None known        14 days    |
|                                                                    |
|  7.0-8.9         Critical             Known exploit     48 hours   |
|  7.0-8.9         Critical             PoC available     7 days     |
|  7.0-8.9         Critical             None known        14 days    |
|  7.0-8.9         Standard             Any               30 days    |
|                                                                    |
|  4.0-6.9         Critical             Known exploit     7 days     |
|  4.0-6.9         Any                  Any               90 days    |
|                                                                    |
|  0.1-3.9         Any                  Any               180 days   |
|                                                                    |
+--------------------------------------------------------------------+

Assessing remediation options

  1. Determine the appropriate remediation approach for each vulnerability. Three options exist: patching applies vendor-supplied fixes; mitigation reduces exploitability without patching; acceptance documents the risk when neither patch nor mitigation is feasible.

    Patching applies when:

    • Vendor has released a fix
    • Testing confirms compatibility with your environment
    • Maintenance window is available within the required timeline
    • Rollback procedure exists

    Mitigation applies when:

    • No patch is available
    • Patch cannot be applied within required timeline
    • System cannot tolerate restart
    • Patch breaks critical functionality

    Acceptance applies when:

    • Compensating controls reduce risk to acceptable level
    • System is scheduled for decommissioning within 90 days
    • Cost of remediation exceeds value of protected asset
    • Technical constraints prevent any remediation

  2. For patch remediation, identify the specific fix and test it:

Terminal window
   # Example: Linux kernel vulnerability
   # Identify current kernel version
   uname -r
   # Output: 5.15.0-84-generic


   # Check available patched kernel
   apt-cache policy linux-image-generic
   # Look for version addressing the CVE


   # In test environment, apply patch
   sudo apt update && sudo apt install linux-image-5.15.0-89-generic


   # Verify installation
   dpkg -l | grep linux-image-5.15.0-89
   # Expected: ii  linux-image-5.15.0-89-generic  5.15.0-89.99  amd64
  1. For mitigation remediation, implement compensating controls. Document each control and its risk reduction effect:
Terminal window
   # Example: Mitigating web server vulnerability with WAF rule
   # Add WAF rule to block exploit pattern
   cat >> /etc/nginx/modsecurity/custom-rules.conf << 'EOF'
   # CVE-2024-1234 mitigation - block path traversal variant
   SecRule REQUEST_URI "@rx \.\.\/.*\.\./" \
     "id:100001,\
     phase:1,\
     deny,\
     status:403,\
     msg:'CVE-2024-1234 exploit attempt blocked',\
     severity:CRITICAL,\
     tag:'CVE-2024-1234'"
   EOF


   # Test rule syntax
   nginx -t
   # Expected: nginx: configuration file /etc/nginx/nginx.conf syntax is ok


   # Reload configuration
   sudo systemctl reload nginx

The mitigation documentation must include:

  • CVE reference
  • Control implemented
  • Residual risk assessment
  • Review date (maximum 90 days)
  • Condition for removing mitigation (patch availability, system decommission)
  1. For risk acceptance, prepare the formal acceptance documentation:
risk-acceptance-template.yaml
   vulnerability_id: CVE-2024-1234
   cvss_score: 7.5
   affected_asset: legacy-app-server-01
   asset_criticality: standard
   asset_owner: finance-team


   acceptance_rationale: |
     System scheduled for decommissioning on 2025-01-31.
     Migration to replacement system is 80% complete.
     Cost of patching ($15,000 vendor support renewal) exceeds
     residual value of 75-day operational period.


   compensating_controls:
     - Network segmentation isolates system from internet
     - Access restricted to 3 named users via jump host
     - Enhanced logging enabled, reviewed daily
     - Data backup increased to 4-hourly


   residual_risk_level: medium
   acceptance_period_days: 90
   review_date: 2025-01-15


   approved_by: security-manager
   approval_date: 2024-11-16

Acceptance requires authority

Risk acceptance decisions require approval from personnel with authority to accept organisational risk. For most organisations, this means IT manager level or above for medium risks, and senior leadership for high or critical risks. Document the approval chain in your security risk management policy.

Planning remediation

  1. Create remediation tickets for each vulnerability requiring action. The ticket must contain sufficient detail for execution without additional research:
   ## Remediation Ticket: CVE-2024-1234


   Priority: Critical (KEV listed, due 2024-11-20)
   Affected System: web-proxy-01 (192.168.10.50)
   System Owner: infrastructure-team
   Maintenance Window: Sunday 02:00-06:00 UTC


   ### Vulnerability Details
   - CVE: CVE-2024-1234
   - CVSS: 9.8 (Critical)
   - Description: Remote code execution in Squid proxy < 6.4
   - Current Version: 6.2
   - Fixed Version: 6.4


   ### Remediation Steps
   1. Create VM snapshot before changes
   2. Stop squid service
   3. Apply package update: apt install squid=6.4-1
   4. Review configuration for deprecated options
   5. Start squid service
   6. Verify service operation
   7. Run verification scan


   ### Rollback Procedure
   1. Stop squid service
   2. Restore VM snapshot
   3. Verify service operation
   4. Notify security team of rollback


   ### Success Criteria
   - Squid version reports 6.4
   - Service passes health checks
   - Vulnerability scan shows CVE-2024-1234 as resolved

  1. Schedule remediation within the SLA timeline. The remediation SLA defines maximum time from vulnerability identification to verified fix:

    PrioritySLAExample
    KEV listedPer CISA deadline14-21 days from listing
    Critical (Score > 20)72 hoursExploited critical on production
    High (Score 15-20)7 daysCritical on non-production
    Standard (Score 10-15)30 daysHigh severity, standard asset
    Queue (Score < 10)90 daysMedium and low severity

    Track SLA compliance using this calculation:

    SLA Status = (Current Date - Discovery Date) / SLA Days × 100%


    Example:
    Discovery: 2024-11-10
    Current: 2024-11-16
    SLA: 7 days


    Status = (6 days / 7 days) × 100% = 86% consumed
    Remaining: 1 day
  1. Coordinate remediation with system owners and change management. For critical vulnerabilities, invoke emergency change procedures if standard windows are insufficient:
Terminal window
    # Check next available maintenance window
    curl -s -H "Authorization: Bearer $CHANGE_API_TOKEN" \
      "https://change.example.org/api/windows?asset=web-proxy-01&next=true" | jq '.window'
    # Output: {"start": "2024-11-17T02:00:00Z", "end": "2024-11-17T06:00:00Z"}


    # If SLA requires faster action, request emergency change
    curl -X POST -H "Authorization: Bearer $CHANGE_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://change.example.org/api/emergency" \
      -d '{
        "asset": "web-proxy-01",
        "justification": "KEV CVE-2024-1234 due 2024-11-20, next standard window 2024-11-24",
        "risk_assessment": "Exploitation in the wild, internet-facing system",
        "requested_window": "2024-11-17T22:00:00Z"
      }'

Executing remediation

  1. Create a pre-remediation backup or snapshot. Verify the backup completes successfully before proceeding:
Terminal window
    # For VM-based systems
    virsh snapshot-create-as web-proxy-01 \
      "pre-CVE-2024-1234-$(date +%Y%m%d%H%M)" \
      "Snapshot before CVE-2024-1234 remediation"


    # Verify snapshot exists
    virsh snapshot-list web-proxy-01 --tree
    # Expected: Shows new snapshot in list


    # For cloud instances (AWS example)
    aws ec2 create-snapshot \
      --volume-id vol-0123456789abcdef0 \
      --description "Pre CVE-2024-1234 remediation" \
      --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Purpose,Value=vulnerability-remediation}]'


    # Wait for completion
    aws ec2 wait snapshot-completed --snapshot-ids snap-0123456789abcdef0
  1. Execute the remediation according to the ticket. Record exact commands and outputs:
Terminal window
    # Connect to target system
    ssh admin@web-proxy-01


    # Record current state
    squid -v | head -1
    # Output: Squid Cache: Version 6.2


    # Stop service
    sudo systemctl stop squid


    # Apply update
    sudo apt update
    sudo apt install squid=6.4-1ubuntu1
    # Record output - confirm no errors


    # Verify new version
    squid -v | head -1
    # Expected: Squid Cache: Version 6.4


    # Check configuration compatibility
    squid -k parse
    # Expected: No errors or warnings about deprecated options


    # Start service
    sudo systemctl start squid


    # Verify service running
    sudo systemctl status squid
    # Expected: Active: active (running)


    # Test functionality
    curl -x http://localhost:3128 -I https://example.org
    # Expected: HTTP/1.1 200 OK (or connection established for HTTPS)
  1. Execute rollback if remediation causes service disruption:
Terminal window
    # If service fails to start or functionality broken


    # Stop failed service
    sudo systemctl stop squid


    # Restore from snapshot (VM example)
    virsh snapshot-revert web-proxy-01 "pre-CVE-2024-1234-202411170200"


    # Or restore package to previous version
    sudo apt install squid=6.2-1ubuntu1


    # Verify service restored
    sudo systemctl start squid
    sudo systemctl status squid


    # Document rollback reason in ticket
    # Escalate to vendor support if patch incompatible

The remediation execution flow follows this sequence:

+------------------------------------------------------------------+
|                  REMEDIATION EXECUTION FLOW                      |
+------------------------------------------------------------------+
|                                                                  |
|  +------------------+                                            |
|  | Create backup/   |                                            |
|  | snapshot         |                                            |
|  +--------+---------+                                            |
|           |                                                      |
|           v                                                      |
|  +--------+---------+     +------------------+                   |
|  | Verify backup    +---->| Backup failed    |                   |
|  | successful       | No  | Resolve before   |                   |
|  +--------+---------+     | proceeding       |                   |
|           | Yes           +------------------+                   |
|           v                                                      |
|  +--------+---------+                                            |
|  | Stop service     |                                            |
|  | (if required)    |                                            |
|  +--------+---------+                                            |
|           |                                                      |
|           v                                                      |
|  +--------+---------+     +------------------+                   |
|  | Apply patch/     +---->| Installation     |                   |
|  | configuration    | Err | failed - check   |                   |
|  +--------+---------+     | dependencies     |                   |
|           | OK            +--------+---------+                   |
|           v                        |                             |
|  +--------+---------+              |                             |
|  | Start service    |              |                             |
|  +--------+---------+              |                             |
|           |                        |                             |
|           v                        |                             |
|  +--------+---------+     +--------v---------+                   |
|  | Verify service   +---->| Service failed   |                   |
|  | operational      | No  | Execute rollback |                   |
|  +--------+---------+     +--------+---------+                   |
|           | Yes                    |                             |
|           v                        |                             |
|  +--------+---------+              |                             |
|  | Run verification |              |                             |
|  | scan             |              |                             |
|  +--------+---------+              |                             |
|           |                        |                             |
|           v                        v                             |
|  +--------+---------+     +--------+---------+                   |
|  | Document         |     | Document failure |                   |
|  | completion       |     | Escalate         |                   |
|  +------------------+     +------------------+                   |
|                                                                  |
+------------------------------------------------------------------+

Verifying remediation

  1. Run a targeted vulnerability scan against the remediated system to confirm the fix is effective:
Terminal window
    # Trigger targeted scan (scanner-specific)
    curl -X POST -H "Authorization: Bearer $VULN_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://scanner.example.org/api/v2/scans" \
      -d '{
        "targets": ["192.168.10.50"],
        "policy": "targeted-verification",
        "cve_filter": ["CVE-2024-1234"]
      }'
    # Returns: {"scan_id": "abc123", "status": "running"}


    # Wait for completion and check results
    curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
      "https://scanner.example.org/api/v2/scans/abc123/results" | jq '.findings'
    # Expected: [] (empty - vulnerability no longer detected)
    # Or: [{"cve": "CVE-2024-1234", "status": "not_vulnerable"}]
  1. Update the vulnerability tracking system to mark the finding as remediated:
Terminal window
    # Update finding status
    curl -X PATCH -H "Authorization: Bearer $VULN_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://scanner.example.org/api/v2/findings/finding-12345" \
      -d '{
        "status": "remediated",
        "remediation_date": "2024-11-17",
        "remediation_method": "patch",
        "remediated_by": "admin@example.org",
        "verification_scan_id": "abc123"
      }'
  1. Close the remediation ticket with evidence of completion:
    ## Resolution Notes


    Remediation completed: 2024-11-17 03:45 UTC
    Method: Patch - upgraded Squid 6.2 → 6.4


    Evidence:
    - Pre-remediation snapshot: pre-CVE-2024-1234-202411170200
    - Package installation log: attached
    - Version verification: Squid Cache: Version 6.4
    - Service status: active (running) since 03:42 UTC
    - Verification scan ID: abc123 - no findings


    Post-implementation test: curl proxy test successful
    Rollback required: No


    Closed by: admin@example.org

Handling field and offline systems

Field-deployed systems present unique remediation challenges. Systems operating on satellite links, intermittent cellular, or fully offline cannot receive patches through standard deployment mechanisms. These systems require staged remediation approaches.

  1. Identify field systems requiring special handling:
Terminal window
    # Query asset inventory for field-deployed systems
    curl -s -H "Authorization: Bearer $CMDB_API_TOKEN" \
      "https://cmdb.example.org/api/assets?location_type=field&connectivity=limited" | \
      jq '.assets[] | {hostname, location, last_seen, connectivity_type}'


    # Example output:
    # {"hostname": "field-laptop-ke-001", "location": "Nairobi", "last_seen": "2024-11-15", "connectivity_type": "cellular"}
    # {"hostname": "clinic-tablet-ug-003", "location": "Gulu", "last_seen": "2024-11-10", "connectivity_type": "satellite"}
    # {"hostname": "offline-server-ss-001", "location": "Juba", "last_seen": "2024-10-01", "connectivity_type": "offline"}
  1. For systems with limited connectivity, prepare offline patch packages:
Terminal window
    # Create offline patch bundle
    mkdir -p /tmp/patch-bundle-CVE-2024-1234
    cd /tmp/patch-bundle-CVE-2024-1234


    # Download package and dependencies for offline installation
    apt-get download squid=6.4-1ubuntu1
    apt-cache depends squid=6.4-1ubuntu1 | grep Depends | \
      awk '{print $2}' | xargs apt-get download


    # Create installation script
    cat > install.sh << 'EOF'
    #!/bin/bash
    # Offline installation for CVE-2024-1234
    # Run as root on target system


    set -e
    echo "Creating pre-patch snapshot..."
    # Add snapshot command appropriate for system


    echo "Installing packages..."
    dpkg -i *.deb


    echo "Verifying installation..."
    squid -v | head -1


    echo "Restarting service..."
    systemctl restart squid


    echo "Installation complete. Verify service status."
    systemctl status squid
    EOF


    chmod +x install.sh


    # Create bundle archive
    cd /tmp
    tar -czvf patch-bundle-CVE-2024-1234.tar.gz patch-bundle-CVE-2024-1234/


    # Generate checksum
    sha256sum patch-bundle-CVE-2024-1234.tar.gz > patch-bundle-CVE-2024-1234.tar.gz.sha256

  1. Distribute patch bundles through available channels:

    For systems with intermittent connectivity, stage the bundle for download during the next connection window:

Terminal window
    # Stage on sync server
    cp patch-bundle-CVE-2024-1234.tar.gz* /srv/field-sync/patches/


    # Add to deployment manifest
    cat >> /srv/field-sync/patches/manifest.json << EOF
    {
      "cve": "CVE-2024-1234",
      "file": "patch-bundle-CVE-2024-1234.tar.gz",
      "sha256": "$(cat patch-bundle-CVE-2024-1234.tar.gz.sha256 | awk '{print $1}')",
      "priority": "critical",
      "sla_date": "2024-11-20",
      "instructions": "Run install.sh as root. Requires 15 minutes and service restart."
    }
    EOF

For fully offline systems, coordinate physical delivery through logistics or staff travel:

Terminal window
    # Create USB deployment package
    # Format USB with ext4 (avoid exFAT for Linux compatibility)
    sudo mkfs.ext4 -L PATCH-CVE2024 /dev/sdb1


    # Mount and copy
    sudo mount /dev/sdb1 /mnt/usb
    sudo cp -r /tmp/patch-bundle-CVE-2024-1234 /mnt/usb/
    sudo cp /srv/field-sync/patches/manifest.json /mnt/usb/


    # Add readme for field staff
    cat > /mnt/usb/README.txt << EOF
    SECURITY PATCH DELIVERY
    CVE: CVE-2024-1234 (Critical)
    Deadline: 2024-11-20


    Installation:
    1. Connect USB to target system
    2. Open terminal as administrator
    3. Navigate to USB: cd /media/*/PATCH-CVE2024/patch-bundle-CVE-2024-1234
    4. Run: sudo ./install.sh
    5. Confirm "Installation complete" message
    6. Report completion to IT: security@example.org


    If errors occur, do not retry. Contact IT for support.
    EOF


    sudo umount /mnt/usb
  1. Track field system remediation status separately, acknowledging extended timelines:
Terminal window
    # Field system remediation tracking
    curl -X POST -H "Authorization: Bearer $VULN_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://scanner.example.org/api/v2/findings/finding-67890/field-remediation" \
      -d '{
        "status": "patch_staged",
        "delivery_method": "sync_server",
        "expected_completion": "2024-11-25",
        "justification": "Limited connectivity - satellite window weekly",
        "compensating_controls": [
          "System isolated from internet",
          "Local firewall blocks vulnerable port",
          "Enhanced monitoring via local SIEM"
        ]
      }'

Adjusted SLAs for field systems

Field system SLAs extend by a factor based on connectivity type: satellite systems receive 2× standard SLA; offline systems receive 4× standard SLA. Document the extended timeline and compensating controls in the remediation record.

Managing exceptions

  1. When remediation cannot occur within SLA, document the exception formally. The exception record must include business justification, technical constraints, compensating controls, and approval:
exception-record.yaml
    exception_id: EXC-2024-0147
    vulnerability_id: CVE-2024-5678
    affected_asset: legacy-finance-01


    standard_sla_days: 7
    requested_extension_days: 60
    new_deadline: 2025-01-15


    justification: |
      System runs legacy financial application with no vendor support.
      Application fails when underlying OS is patched due to
      kernel API dependency. Vendor ceased operations 2022.


      Replacement project approved, deployment scheduled Q1 2025.
      Migration testing begins 2024-12-01.


    technical_constraints:
      - Application binary incompatible with patched kernel
      - No source code available for recompilation
      - Application handles year-end financial close (Nov-Jan)


    compensating_controls:
      - control: Network isolation
        implementation: VLAN 99, no internet access, restricted to finance subnet
        effectiveness: Reduces attack surface to internal only


      - control: Enhanced monitoring
        implementation: Dedicated SIEM rules for vulnerability signature
        effectiveness: Detection within 5 minutes of exploitation attempt


      - control: Access restriction
        implementation: PAM controls limit access to 4 named users
        effectiveness: Reduces exposure to privileged accounts only


      - control: Increased backup frequency
        implementation: Hourly snapshots retained 72 hours
        effectiveness: Maximum 1 hour data loss if compromise occurs


    residual_risk_assessment: |
      With compensating controls, residual risk reduced from Critical to Medium.
      Exploitation requires internal network access and privileged credentials.
      Impact limited to single system with 72-hour recovery capability.


    approvals:
      - role: IT Manager
        name: J. Smith
        date: 2024-11-16
        decision: approved


      - role: CISO
        name: A. Johnson
        date: 2024-11-16
        decision: approved
        condition: Monthly review until resolved


    review_schedule:
      - date: 2024-12-15
        reviewer: security-team
      - date: 2025-01-15
        reviewer: security-team
        action: Confirm migration complete or extend exception
  1. Enter the exception in the vulnerability management system:
Terminal window
    curl -X POST -H "Authorization: Bearer $VULN_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://scanner.example.org/api/v2/findings/finding-23456/exception" \
      -d @exception-record.yaml


    # Verify exception recorded
    curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
      "https://scanner.example.org/api/v2/findings/finding-23456" | jq '.exception'
    # Expected: Shows exception details with status "approved"
  1. Schedule exception reviews and ensure they execute:
Terminal window
    # Create calendar entry for review
    curl -X POST -H "Authorization: Bearer $CALENDAR_API_TOKEN" \
      -H "Content-Type: application/json" \
      "https://calendar.example.org/api/events" \
      -d '{
        "title": "Exception Review: CVE-2024-5678 legacy-finance-01",
        "date": "2024-12-15",
        "attendees": ["security-team@example.org", "it-manager@example.org"],
        "description": "Monthly review of vulnerability exception EXC-2024-0147",
        "recurrence": "monthly until 2025-01-15"
      }'

Reporting and metrics

SLA compliance tracking provides visibility into remediation effectiveness. Calculate metrics weekly for operational reporting and monthly for leadership:

+------------------------------------------------------------------+
|                     SLA TRACKING DASHBOARD                       |
+------------------------------------------------------------------+
|                                                                  |
|  Period: 2024-11-01 to 2024-11-15                                |
|                                                                  |
|  REMEDIATION VOLUME                                              |
|  +------------------+------------------+------------------+      |
|  | Opened           | Closed           | Net Change       |      |
|  | 47               | 52               | -5               |      |
|  +------------------+------------------+------------------+      |
|                                                                  |
|  SLA COMPLIANCE BY PRIORITY                                      |
|  +------------------+--------+--------+--------+---------+       |
|  | Priority         | Total  | On-time| Late   | % Comp  |       |
|  +------------------+--------+--------+--------+---------+       |
|  | Critical         | 4      | 4      | 0      | 100%    |       |
|  | High             | 18     | 15     | 3      | 83%     |       |
|  | Standard         | 22     | 19     | 3      | 86%     |       |
|  | Queue            | 8      | 8      | 0      | 100%    |       |
|  +------------------+--------+--------+--------+---------+       |
|  | TOTAL            | 52     | 46     | 6      | 88%     |       |
|  +------------------+--------+--------+--------+---------+       |
|                                                                  |
|  MEAN TIME TO REMEDIATE (MTTR)                                   |
|  +------------------+------------------+------------------+      |
|  | Priority         | Target           | Actual           |      |
|  +------------------+------------------+------------------+      |
|  | Critical         | 72 hours         | 48 hours         |      |
|  | High             | 7 days           | 5.2 days         |      |
|  | Standard         | 30 days          | 22 days          |      |
|  +------------------+------------------+------------------+      |
|                                                                  |
|  OPEN EXCEPTIONS                                                 |
|  +------------------+------------------+------------------+      |
|  | Exception ID     | Vulnerability    | Review Date      |      |
|  +------------------+------------------+------------------+      |
|  | EXC-2024-0147    | CVE-2024-5678    | 2024-12-15       |      |
|  | EXC-2024-0143    | CVE-2024-4321    | 2024-11-30       |      |
|  +------------------+------------------+------------------+      |
|                                                                  |
+------------------------------------------------------------------+

Generate the compliance report:

Terminal window
# Calculate SLA compliance
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/reports/sla-compliance?\
start_date=2024-11-01&end_date=2024-11-15" | jq '.'


# Calculate MTTR
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/reports/mttr?\
start_date=2024-11-01&end_date=2024-11-15&group_by=priority" | jq '.'


# Export for leadership report
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/reports/executive-summary?\
start_date=2024-11-01&end_date=2024-11-15&format=pdf" \
  -o vulnerability-report-2024-11.pdf

Verification

After completing remediation activities, confirm the following:

Run a verification scan against all systems remediated during the period:

Terminal window
# Bulk verification scan
curl -X POST -H "Authorization: Bearer $VULN_API_TOKEN" \
  -H "Content-Type: application/json" \
  "https://scanner.example.org/api/v2/scans" \
  -d '{
    "targets": ["192.168.10.0/24"],
    "policy": "verification-full",
    "compare_to": "scan-previous-week"
  }'


# After scan completes, verify reduction
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/scans/latest/comparison" | \
  jq '.summary'
# Expected: {"new": 5, "resolved": 52, "unchanged": 195}
# resolved >= remediated count

Verify no regression occurred from remediation activities:

Terminal window
# Check for new vulnerabilities introduced by patches
curl -s -H "Authorization: Bearer $VULN_API_TOKEN" \
  "https://scanner.example.org/api/v2/findings?\
status=open&first_seen_after=2024-11-15" | jq '.findings | length'
# Expected: Low number, reviewed for patch-related issues

Confirm all tickets are closed with appropriate documentation:

Terminal window
# Query ticketing system for open remediation tickets past SLA
curl -s -H "Authorization: Bearer $TICKET_API_TOKEN" \
  "https://tickets.example.org/api/projects/security-remediation/issues?\
status=open&sla_status=breached" | jq '.issues[] | {id, title, sla_breach_days}'
# Expected: [] (empty) or documented exceptions only

Troubleshooting

SymptomCauseResolution
Patch installation fails with dependency errorMissing prerequisite packagesRun apt-get -f install to resolve dependencies; verify package repository access
Service fails to start after patchingConfiguration incompatibility with new versionCheck service logs for specific error; review changelog for breaking changes; restore from snapshot if needed
Vulnerability reappears in next scanPatch not fully applied or wrong systemVerify patch applied to correct system; check if vulnerability has multiple affected components
Scanner reports vulnerability fixed but version unchangedFalse positive from scanner; or scanner signature out of dateManually verify installed version; update scanner signatures; report false positive to vendor
Cannot create snapshot on target systemInsufficient storage or hypervisor issueFree disk space on datastore; check hypervisor health; proceed without snapshot only with documented approval
Patch breaks dependent applicationApplication incompatibility not identified in testingRollback patch; test in isolated environment; coordinate with application owner for compatibility fix
Exception approval delayed past SLAApproval chain unavailable or disagreementEscalate to next level; document compensating controls implemented pending approval
Field system unreachable for verificationConnectivity unavailableAccept extended verification timeline; request local verification from field staff with photo evidence
CVSS score disputed by asset ownerDisagreement on environmental factorsUse scanner-provided base score for prioritisation; document owner assessment in exception if requested
Bulk patch deployment causes widespread outageInsufficient testing or stagingExecute rollback; implement staged deployment; review change management process
Mitigation rule blocks legitimate trafficRule too broad or incorrect patternRefine rule to target exploit specifically; test with legitimate traffic patterns before deployment
Remediation ticket assigned to wrong teamAsset ownership unclear in CMDBUpdate CMDB with correct owner; establish process for ownership verification
Patch available but not in repositoryRepository synchronisation delay or restricted packageCheck upstream repository; download package directly; request repository update
Exception review missedCalendar entry ignored or deletedRe-establish review with escalation to approver; consider automated reminders
Metrics show declining SLA complianceInsufficient resources or process issuesReview remediation capacity; analyse bottlenecks; request additional resources if workload exceeds capacity

See also