Patch Management

Patch management maintains system security and stability through controlled application of vendor-released updates. This task covers the complete patching cycle from identification through deployment, including emergency procedures for critical vulnerabilities and exception handling for systems that cannot be patched immediately.

Prerequisites

Before establishing or executing patch management procedures, ensure the following requirements are met.

Verify patch management platform connectivity before beginning any patch cycle:

# For WSUS - check client registration
wuauclt /detectnow /reportnow

# For Ansible - verify inventory reachability
ansible all -m ping -i inventory/production.yml

# For Linux systems - verify repository access
apt update && apt list --upgradable
# or
dnf check-update

Expected output shows systems reporting to the management platform and repositories accessible. Systems not reporting require investigation before proceeding with deployment.

Patch Identification and Assessment

Patch identification begins with monitoring vendor release channels. Microsoft releases patches on the second Tuesday of each month (Patch Tuesday), with out-of-band releases for critical vulnerabilities. Linux distributions publish updates continuously, with security advisories flagged separately from feature updates. Third-party applications follow vendor-specific schedules.

Configure your patch management platform to synchronise with vendor catalogues:

# WSUS synchronisation (PowerShell)
$wsus = Get-WsusServer
$subscription = $wsus.GetSubscription()
$subscription.StartSynchronization()

For Linux environments, repository synchronisation occurs during standard update checks. Create a local mirror for bandwidth-constrained field offices:

# Create local apt mirror (Ubuntu/Debian)
apt-mirror /etc/apt/mirror.list

# Sync RPM repositories (RHEL/CentOS)
reposync -p /var/www/html/repos/ --repo=updates

After synchronisation completes, assess each patch against four criteria: security severity, operational impact, system applicability, and deployment urgency.

Severity Classification

Patch severity determines response timelines. The Common Vulnerability Scoring System (CVSS) provides a standardised severity rating from 0.0 to 10.0.

These windows represent maximum acceptable time from patch release to production deployment. Actively exploited vulnerabilities (known exploitation in the wild) compress all windows to 24-72 hours regardless of CVSS score.

Operational Impact Assessment

Evaluate each patch for potential operational disruption. Patches fall into three impact categories based on deployment requirements.

Reboot-required patches include kernel updates, driver updates, and core system library changes. These patches interrupt service and require maintenance window scheduling. Windows cumulative updates almost always require reboot. Linux kernel updates require reboot to load the new kernel, though live patching solutions (kpatch, livepatch) can defer reboot for many kernel security fixes.

Service-restart patches affect specific applications without full system reboot. Database patches, web server updates, and application runtime patches typically restart only the affected service. Service restart duration varies from seconds to minutes depending on application initialisation time.

No-restart patches apply immediately without service interruption. Definition updates for antimalware, application plugins, and some security configuration changes fall into this category.

Document impact assessment for each patch in your tracking system:

Patch ID: KB5034441
Severity: High (CVSS 7.8)
Impact: Reboot required
Affected systems: 47 Windows Server 2022
Dependencies: None
Testing priority: High

Criticality and Urgency Determination

Combine severity, impact, and organisational context to determine deployment priority. A critical severity patch affecting internet-facing systems demands faster response than the same patch affecting isolated internal systems.

+------------------------------------------------------------------+
|                    PATCH PRIORITISATION MATRIX                   |
+------------------------------------------------------------------+
|                                                                  |
|              CVSS SEVERITY                                       |
|              Critical  High    Medium   Low                      |
|              (9.0+)    (7-8.9) (4-6.9)  (0-3.9)                  |
|  +---------+---------+--------+--------+--------+                |
|  |Internet |    P1   |   P1   |   P2   |   P3   |                |
|  |facing   | 24 hrs  | 72 hrs | 14 days| 30 days|                |
|  +---------+---------+--------+--------+--------+                |
|  |Internal |    P1   |   P2   |   P2   |   P3   |  ASSET         |
|  |critical | 72 hrs  | 14 days| 21 days| 60 days|  EXPOSURE      |
|  +---------+---------+--------+--------+--------+                |
|  |Internal |    P2   |   P2   |   P3   |   P4   |                |
|  |standard | 7 days  | 21 days| 30 days| 90 days|                |
|  +---------+---------+--------+--------+--------+                |
|  |Isolated |    P2   |   P3   |   P3   |   P4   |                |
|  |systems  | 14 days | 30 days| 45 days| 90 days|                |
|  +---------+---------+--------+--------+--------+                |
|                                                                  |
|  P1 = Emergency   P2 = Urgent   P3 = Standard   P4 = Scheduled   |
+------------------------------------------------------------------+

Priority determines both deployment speed and approval requirements. P1 patches follow emergency change procedures with post-implementation approval. P2-P4 patches follow standard change management.

Active Exploitation Modifier

Patches addressing vulnerabilities with confirmed active exploitation override the standard matrix. The Cybersecurity and Infrastructure Security Agency (CISA) maintains the Known Exploited Vulnerabilities (KEV) catalogue. Any vulnerability appearing in KEV automatically becomes P1 regardless of CVSS score or asset exposure.

Check KEV status during assessment:

# Query CISA KEV catalogue
curl -s https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json | \
  jq '.vulnerabilities[] | select(.cveID == "CVE-2024-XXXXX")'

Empty response indicates the vulnerability is not in KEV. Any returned record confirms active exploitation and triggers emergency procedures.

Testing Procedures

Testing validates patch compatibility before production deployment. Testing scope scales with patch impact and organisational risk tolerance.

Test Environment Configuration

Maintain test systems that mirror production configurations. For Windows environments, clone representative servers using Hyper-V checkpoints or VMware snapshots. For Linux systems, maintain configuration-managed test instances that match production state.

# Ansible playbook to create test environment
---
- name: Prepare patch testing environment
  hosts: test_servers
  become: yes
  tasks:
    - name: Sync configuration from production template
      ansible.builtin.copy:
        src: /configs/production/
        dest: /etc/
        backup: yes

    - name: Start critical services
      ansible.builtin.service:
        name: "{{ item }}"
        state: started
      loop:
        - nginx
        - postgresql
        - application-server

Testing Execution

Execute patch installation on test systems following the same procedures used for production:

1. Create snapshot or checkpoint of test system before patching:

   # VMware snapshot via govc
   govc snapshot.create -vm /DC/vm/test-server-01 "pre-patch-$(date +%Y%m%d)"

   # Hyper-V checkpoint
   Checkpoint-VM -Name "test-server-01" -SnapshotName "pre-patch-$(Get-Date -Format yyyyMMdd)"

2. Apply patches to test system using production deployment method:

   # Windows via WSUS
   wuauclt /detectnow
   Install-WindowsUpdate -AcceptAll -AutoReboot

   # Ubuntu/Debian
   apt update && apt upgrade -y

   # RHEL/CentOS
   dnf update -y

3. Verify system boot and service startup after reboot:

   # Check critical services
   systemctl is-active nginx postgresql application-server

   # Verify network connectivity
   ping -c 3 gateway.example.org
   curl -I https://api.example.org/health

4. Execute application functionality tests covering core workflows:

   # Run automated test suite
   ./run-tests.sh --suite=smoke --environment=test

   # Expected output: All tests passed (23/23)

5. Monitor system for 24-48 hours for stability issues:

   Check logs for errors introduced by patches:

   journalctl --since "24 hours ago" --priority=err
   grep -i error /var/log/application/*.log | grep "$(date +%Y-%m-%d)"

6. Document test results with specific pass/fail criteria:

   Record patch ID, test date, test systems, results, and any issues observed. Attach to change request as evidence.

Test Failure Handling

When testing reveals problems, categorise the failure before determining next steps.

Application incompatibility occurs when patched system components break application functionality. Document the incompatibility, notify the application vendor, and raise an exception request. The application team must provide a remediation timeline or accept the risk of running unpatched.

System instability manifests as crashes, high resource usage, or service failures after patching. Report to the OS vendor, revert to snapshot, and await a corrected patch release. Do not deploy unstable patches to production.

Partial functionality loss affects some but not all system functions. Assess whether affected functionality is critical. If non-critical, document as known issue and proceed with deployment. If critical, treat as application incompatibility.

Approval Workflow

All non-emergency patches require change management approval before production deployment. The approval level correlates with deployment scope and risk.

+------------------------------------------------------------------+
|                      APPROVAL REQUIREMENTS                       |
+------------------------------------------------------------------+
|                                                                  |
|  +--------------------+                                          |
|  | Patch identified   |                                          |
|  +---------+----------+                                          |
|            |                                                     |
|            v                                                     |
|  +---------+----------+                                          |
|  | Priority assigned  |                                          |
|  +---------+----------+                                          |
|            |                                                     |
|    +-------+-------+                                             |
|    |               |                                             |
|    v               v                                             |
|  +-+-----+    +----+-----+                                       |
|  | P1    |    | P2-P4    |                                       |
|  | Emerg |    | Standard |                                       |
|  +---+---+    +----+-----+                                       |
|      |             |                                             |
|      v             v                                             |
|  +---+--------+  +-+----------+                                  |
|  | Deploy     |  | Submit     |                                  |
|  | immediately|  | change     |                                  |
|  +---+--------+  | request    |                                  |
|      |           +-----+------+                                  |
|      |                 |                                         |
|      |                 v                                         |
|      |           +-----+------+    +-------------+               |
|      |           | CAB review |<---+ Test results|               |
|      |           +-----+------+    +-------------+               |
|      |                 |                                         |
|      |         +-------+-------+                                 |
|      |         |               |                                 |
|      |         v               v                                 |
|      |   +-----+-----+   +-----+-----+                           |
|      |   | Approved  |   | Rejected  |                           |
|      |   +-----+-----+   +-----+-----+                           |
|      |         |               |                                 |
|      |         v               v                                 |
|      |   +-----+-----+   +-----+-----+                           |
|      |   | Schedule  |   | Remediate |                           |
|      |   | deploy    |   | & resubmit|                           |
|      |   +-----------+   +-----------+                           |
|      |                                                           |
|      v                                                           |
|  +---+---------------+                                           |
|  | Post-incident     |                                           |
|  | change review     |                                           |
|  +-------------------+                                           |
+------------------------------------------------------------------+

Change Request Content

Submit change requests with sufficient detail for informed approval. Include:

Change Title: Monthly security patches - November 2024
Change Type: Standard (Patch deployment)
Priority: P2 - Urgent

Scope:
- 47 Windows Server 2022 systems
- 23 Ubuntu 22.04 LTS systems
- Excludes: Finance database server (exception #2024-089)

Patches:
- KB5032190 (Windows cumulative update, CVSS 8.1)
- KB5032189 (Windows .NET update, CVSS 6.5)
- USN-6478-1 (Ubuntu kernel update, CVSS 7.8)
- USN-6481-1 (OpenSSL update, CVSS 5.3)

Testing:
- Tested 2024-11-08 on TEST-WIN-01, TEST-UBU-01
- All smoke tests passed
- 48-hour monitoring showed no stability issues
- Test evidence attached

Deployment Schedule:
- Wave 1: Non-critical servers - 2024-11-12 22:00 UTC
- Wave 2: Application servers - 2024-11-13 22:00 UTC
- Wave 3: Database servers - 2024-11-14 22:00 UTC

Rollback Plan:
- Windows: System restore point created pre-deployment
- Linux: Snapshot via VMware, kernel rollback via GRUB

Risk Assessment: Medium
- Service restart required for all systems
- Maintenance window minimises user impact
- Tested successfully, low regression risk

Change Advisory Board (CAB) reviews change requests against testing evidence, deployment schedule, and rollback capability. Approval grants deployment authorisation within the specified maintenance window.

Deployment Scheduling

Schedule deployments to minimise operational impact while meeting security timelines. Organisations with global operations coordinate across time zones; those with field offices consider connectivity windows.

Maintenance Windows

Define regular maintenance windows by system tier. Systems supporting 24/7 operations require rolling deployments or high-availability configurations that permit patching without service interruption.

Field office scheduling accounts for connectivity constraints. Offices with satellite connectivity may require patch content pre-staged locally to avoid consuming bandwidth during business hours.

Deployment Waves

Deploy patches in waves to detect problems before they affect the entire environment. Each wave should be large enough to reveal issues but small enough to limit blast radius.

Wave 1: Canary (5% of systems)
        Deploy to non-critical systems
        Monitor for 4-24 hours
        Proceed if no issues
             |
             v
Wave 2: Early adopters (20% of systems)
        Deploy to representative production systems
        Monitor for 24-48 hours
        Proceed if no issues
             |
             v
Wave 3: Majority (50% of systems)
        Deploy to remaining non-critical systems
        Monitor for 24 hours
        Proceed if no issues
             |
             v
Wave 4: Final (25% of systems)
        Deploy to critical systems
        Extended monitoring
        Patch cycle complete

For small organisations with fewer than 50 systems, three waves suffice: test (1-2 systems), pilot (5-10 systems), and production (remainder).

Scheduling Commands

Schedule deployments using your patch management platform’s scheduling capability:

# SCCM deployment scheduling (PowerShell)
New-CMSoftwareUpdateDeployment `
  -SoftwareUpdateGroupName "November 2024 Security Updates" `
  -CollectionName "Wave 1 - Canary Servers" `
  -DeploymentType Required `
  -AvailableDateTime (Get-Date "2024-11-12 22:00") `
  -DeadlineDateTime (Get-Date "2024-11-13 02:00") `
  -UserNotification DisplaySoftwareCenterOnly `
  -AllowRestart $true

# Ansible scheduled deployment
---
- name: Deploy patches - Wave 1
  hosts: canary_servers
  become: yes
  vars:
    deployment_date: "2024-11-12"
  tasks:
    - name: Update all packages
      ansible.builtin.apt:
        upgrade: dist
        update_cache: yes
      when: ansible_date_time.date == deployment_date

    - name: Reboot if required
      ansible.builtin.reboot:
        msg: "Rebooting for patch deployment"
        reboot_timeout: 600
      when: reboot_required.stat.exists

Deployment Execution

Execute deployment during the scheduled maintenance window following standardised procedures.

1. Confirm maintenance window and send notification 30 minutes before start:

   Subject: [Maintenance] Patch deployment starting in 30 minutes

   Scheduled maintenance for patch deployment will begin at 22:00 UTC.

   Affected systems: [List from change request]
   Expected duration: 4 hours
   Expected impact: Brief service interruptions during reboots

   Contact IT Service Desk for issues: servicedesk@example.org

2. Create system restore points or snapshots immediately before deployment:

   # Windows restore point
   Checkpoint-Computer -Description "Pre-patch $(Get-Date -Format yyyyMMdd)" -RestorePointType MODIFY_SETTINGS

   # VMware snapshot via Ansible
   ansible-playbook create-snapshots.yml -e "snapshot_name=pre-patch-$(date +%Y%m%d)"

3. Initiate patch deployment through management platform:

   # Trigger WSUS update detection and installation
   Invoke-WUJob -ComputerName (Get-Content servers-wave1.txt) -Script {
     Install-WindowsUpdate -AcceptAll -IgnoreReboot
   } -Confirm:$false -RunNow

   # Ansible deployment
   ansible-playbook deploy-patches.yml -l wave1_servers --extra-vars "reboot=true"

4. Monitor deployment progress through console or command line:

   # Check Windows Update status
   Get-WUHistory -Last 5 | Select-Object Date, Title, Result

   # Monitor Ansible deployment
   tail -f /var/log/ansible/patch-deployment.log

5. Verify each system completes patching and reboots successfully:

   # Check system uptime (should show recent reboot)
   ansible wave1_servers -m command -a "uptime"

   # Verify services running
   ansible wave1_servers -m service -a "name=nginx state=started"

6. Execute post-deployment verification tests:

   # Application health checks
   for server in $(cat servers-wave1.txt); do
     curl -s -o /dev/null -w "%{http_code}" https://$server/health
   done

   # Expected: 200 for all servers

7. Document deployment completion with results:

   Record completion time, systems patched, any failures, and verification results. Update change request with implementation evidence.

Handling Deployment Failures

When systems fail to patch successfully, isolate the failure type and respond accordingly.

Installation failure leaves the system unpatched. Review Windows Update or package manager logs for specific error codes:

# Windows Update error log
Get-WindowsUpdateLog
# Outputs to Desktop\WindowsUpdate.log

# Search for specific update failures
Select-String -Path $env:USERPROFILE\Desktop\WindowsUpdate.log -Pattern "FAILED|ERROR" | Select-Object -Last 20

# apt failure investigation
cat /var/log/apt/term.log | grep -A5 "Error"

# dnf failure investigation
cat /var/log/dnf.log | grep -i error

Common installation errors and resolutions:

Reboot failure leaves the system unresponsive after restart. Use out-of-band management (iLO, iDRAC, IPMI) to access console:

# ipmitool console access
ipmitool -I lanplus -H server-ipmi.example.org -U admin chassis power status
ipmitool -I lanplus -H server-ipmi.example.org -U admin sol activate

If system is stuck at boot, restore from snapshot. Do not proceed with further waves until root cause is identified.

Service failure results in successful patching but broken applications. Follow application-specific recovery procedures. If resolution is not immediate, restore from snapshot and exclude the system from current patch cycle while investigating.

Verification and Reporting

Verification confirms patches are installed and systems are functional. Reporting provides evidence for compliance and audit purposes.

Patch Verification

Query patched systems to confirm update installation:

# Verify specific Windows update installed
Get-HotFix -Id KB5032190

# List all updates installed in date range
Get-HotFix | Where-Object {$_.InstalledOn -gt (Get-Date).AddDays(-7)}

# Verify Linux package versions
dpkg -l | grep linux-image
rpm -q kernel

# Check specific package version
apt show openssl | grep Version

Compare installed versions against expected versions from patch release documentation. Discrepancies indicate incomplete deployment.

Compliance Reporting

Generate patch compliance reports showing deployment status across the environment:

# WSUS compliance report
Get-WsusComputer -All | Select-Object FullDomainName, LastReportedStatusTime,
  @{N='Needed';E={($_ | Get-WsusComputer).GetUpdateInstallationInfoPerComputer() |
    Where-Object {$_.UpdateApprovalAction -eq 'Install' -and $_.UpdateInstallationState -eq 'NotInstalled'} |
    Measure-Object | Select-Object -ExpandProperty Count}}

Track compliance metrics:

Sample compliance report - November 2024

Total managed systems: 70
Systems patched: 66 (94.3%)
Systems pending: 2 (2.9%)
Systems excepted: 2 (2.9%)

Patch cycle started: 2024-11-12
Patch cycle completed: 2024-11-14
Duration: 48 hours

Exceptions:
- FINANCE-DB-01: Oracle compatibility (Exception #2024-089, expires 2024-12-15)
- LEGACY-APP-02: End of life scheduled (Exception #2024-091, expires 2024-11-30)

Pending:
- FIELD-KEN-03: Awaiting connectivity window
- FIELD-UGA-01: Awaiting connectivity window

Emergency Patching Procedures

Emergency patches address actively exploited vulnerabilities requiring immediate deployment outside normal maintenance windows. These procedures compress the standard patch cycle from days to hours.

Emergency Activation Criteria

Activate emergency procedures when any of the following conditions exist:

• Vulnerability has confirmed active exploitation (appears in CISA KEV)
• Vendor rates vulnerability as critical with wormable potential
• Security team identifies active targeting of the organisation
• Regulatory body mandates immediate patching

Emergency Procedure

1. Confirm emergency status with IT security lead and document authorisation:

   Emergency Patch Authorisation

   Vulnerability: CVE-2024-XXXXX
   Severity: Critical (CVSS 9.8)
   KEV Status: Listed 2024-11-10
   Authorised by: [Security Lead Name]
   Authorised at: 2024-11-10 14:30 UTC

2. Download and stage patches immediately without waiting for standard synchronisation:

   # Direct Microsoft Update Catalog download
   wget "https://catalog.update.microsoft.com/..." -O KB5032190.msu

   # Stage to distribution point
   scp KB5032190.msu fileserver:/patches/emergency/

3. Conduct abbreviated testing (2-4 hours maximum) on representative systems:

   Deploy to 2-3 test systems, verify boot, check critical applications. Accept reduced testing confidence due to emergency timeline.

4. Deploy to all affected systems without wave sequencing:

   # Parallel deployment to all systems
   ansible-playbook emergency-patch.yml -l affected_systems -f 20

5. Monitor deployment and address failures immediately:

   Assign dedicated personnel to monitor each deployment batch. Failures receive immediate attention rather than queuing for later resolution.

6. Submit emergency change request within 24 hours post-deployment:

   Document actions taken, systems affected, and results. Emergency changes receive retrospective CAB review.

Exceptions and Deferrals

Some systems cannot be patched within standard timelines due to compatibility issues, change freezes, or operational constraints. Exception management balances security risk against operational requirements.

Exception Request Process

Submit exception requests with:

Patch Exception Request

System(s): FINANCE-DB-01
Patches affected: KB5032190, KB5032189
Reason: Oracle Database 19c incompatibility confirmed by vendor
Alternative controls:
  - Enhanced monitoring via SIEM
  - Network segmentation from general population
  - Restricted administrative access
Risk acceptance: Finance Director (data owner)
Remediation plan: Upgrade to Oracle 21c (project scheduled Q1 2025)
Exception expiry: 2025-03-31

Exception Approval Authority

Compensating Controls

Systems with patch exceptions require compensating controls to reduce exposure:

• Network segmentation limiting connectivity to minimum required
• Enhanced logging with real-time alerting for suspicious activity
• Restricted access removing unnecessary user and administrative accounts
• Additional endpoint protection where compatible
• Increased monitoring frequency for vulnerability scanning

Document compensating controls in the exception request and verify implementation before granting approval.

Exception Review

Review all active exceptions monthly. Expired exceptions require either patch deployment or renewal with updated justification. Track exception aging:

# Query exceptions approaching expiry
SELECT system_name, exception_id, expiry_date,
       DATEDIFF(expiry_date, CURDATE()) as days_remaining
FROM patch_exceptions
WHERE expiry_date <= DATE_ADD(CURDATE(), INTERVAL 14 DAY)
ORDER BY expiry_date;

Field and Offline Systems

Systems in field offices with intermittent connectivity require modified procedures. Standard deployment methods may fail due to bandwidth constraints or connection unavailability.

Pre-staged Deployment

Stage patch content locally before the deployment window:

# Ansible playbook to pre-stage patches
---
- name: Pre-stage patches to field offices
  hosts: field_file_servers
  tasks:
    - name: Synchronise patch repository
      ansible.posix.synchronize:
        src: /var/patches/current/
        dest: /srv/patches/
        compress: yes
      when: "'field_office' in group_names"

Configure field systems to pull patches from local sources:

# Configure local apt repository
echo "deb [trusted=yes] http://fileserver.local/patches/ stable main" > /etc/apt/sources.list.d/local.list

# Prioritise local repository
cat > /etc/apt/preferences.d/local << EOF
Package: *
Pin: origin fileserver.local
Pin-Priority: 1001
EOF

Connectivity Window Deployment

Schedule deployments to coincide with reliable connectivity windows. Field offices with scheduled satellite connectivity may have specific hours with adequate bandwidth:

Field office patch schedule:

Site: Kenya Field Office
Connectivity: VSAT (0.5 Mbps)
Reliable window: 04:00-08:00 UTC daily
Patch window: First Sunday of month, 04:00-08:00 UTC
Pre-staging: Thursday before patch window

Disconnected System Patching

Systems without network connectivity require manual patch delivery:

1. Download patches to encrypted USB drive at connected location:

   # Create encrypted volume
   cryptsetup luksFormat /dev/sdb1
   cryptsetup open /dev/sdb1 patches
   mkfs.ext4 /dev/mapper/patches
   mount /dev/mapper/patches /mnt/patches

   # Copy patches
   cp -r /var/patches/current/* /mnt/patches/

2. Transport USB drive to disconnected location following data carriage procedures.

3. Apply patches from local media:

   # Mount encrypted drive
   cryptsetup open /dev/sdb1 patches
   mount /dev/mapper/patches /mnt/patches

   # Install packages from local source
   dpkg -i /mnt/patches/*.deb
   apt-get -f install

4. Document deployment and return USB drive for secure erasure.

Troubleshooting

Patch Installation Failures

Symptom: Update fails with error 0x80070005 (Access Denied)

Cause: Insufficient permissions for Windows Update service or user account

Resolution:

# Reset Windows Update permissions
Stop-Service wuauserv
Remove-Item $env:systemroot\SoftwareDistribution -Recurse -Force
Start-Service wuauserv

# Verify service account
sc.exe qc wuauserv | findstr SERVICE_START_NAME
# Should show: LocalSystem

Symptom: Patch downloads but installation hangs indefinitely

Cause: Corrupt Windows Update cache or conflicting update

Resolution:

# Clear update cache
Stop-Service wuauserv, bits
Rename-Item $env:systemroot\SoftwareDistribution SoftwareDistribution.old
Rename-Item $env:systemroot\System32\catroot2 catroot2.old
Start-Service wuauserv, bits

# Retry update
wuauclt /detectnow /updatenow

Symptom: Linux package installation fails with dependency errors

Cause: Package version conflicts or missing dependencies

Resolution:

# Identify broken packages
apt --fix-broken install

# Force dependency resolution
apt install -f

# If persistent, remove and reinstall problematic package
apt remove --purge package-name
apt install package-name

Symptom: WSUS clients not detecting available updates

Cause: Group Policy not applied or WSUS server communication failure

Resolution:

# Force Group Policy refresh
gpupdate /force

# Reset Windows Update client
wuauclt /resetauthorization /detectnow

# Verify WSUS server configured
reg query "HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate" /v WUServer

Symptom: Ansible patch deployment times out

Cause: SSH connection limit or target system overload

Resolution:

# Reduce parallel execution in ansible.cfg
[defaults]
forks = 10

# Increase timeout in playbook
- name: Apply updates with extended timeout
  apt:
    upgrade: dist
  async: 3600
  poll: 60

Application Compatibility Issues

Symptom: Application fails to start after patch deployment

Cause: Patch modified shared library or runtime dependency

Resolution:

# Identify missing dependencies
ldd /path/to/application | grep "not found"

# Check library versions
ldconfig -p | grep library-name

# Restore specific package version if needed
apt install package-name=specific-version

Symptom: .NET application throws runtime errors post-patch

Cause: .NET framework update changed behaviour

Resolution:

# Check installed .NET versions
Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP' -Recurse |
  Get-ItemProperty -Name Version -ErrorAction SilentlyContinue |
  Select-Object PSChildName, Version

# Repair .NET installation
DISM /Online /Cleanup-Image /RestoreHealth
sfc /scannow

Symptom: Database connectivity fails after OS patch

Cause: Network driver or TLS configuration changed

Resolution:

# Verify network interface status
ip addr show
systemctl status NetworkManager

# Check TLS configuration
openssl s_client -connect database.example.org:5432 -tls1_2

# Restore previous network driver if needed
apt install linux-modules-extra-$(uname -r)

Rollback Procedures

Symptom: System unstable after patching, requires rollback

Cause: Patch introduced incompatibility or bug

Resolution for Windows:

# Uninstall specific update
wusa /uninstall /kb:5032190 /quiet /norestart

# Restore from system restore point
rstrui.exe
# Select restore point created before patching

Resolution for Linux:

# Boot to previous kernel (GRUB menu)
# Select "Advanced options" then previous kernel version

# Remove problematic kernel after booting to previous
apt remove linux-image-version-to-remove

# Downgrade specific package
apt install package-name=previous-version
apt-mark hold package-name  # Prevent re-upgrade

Resolution using snapshots:

# VMware snapshot revert
govc snapshot.revert -vm /DC/vm/server-01 "pre-patch-20241112"

# Hyper-V checkpoint restore
Restore-VMCheckpoint -VMName "server-01" -Name "pre-patch-20241112" -Confirm:$false

Symptom: Patch creates boot loop

Cause: Critical system file corrupted or driver incompatibility

Resolution:

1. Access recovery console (Windows) or single-user mode (Linux)
2. For Windows: Boot to recovery, select "Uninstall Updates"
3. For Linux: Boot with init=/bin/bash, mount filesystem, remove package
4. If recovery fails, restore from snapshot or re-image system

Reporting Discrepancies

Symptom: Compliance reports show systems as unpatched despite deployment

Cause: Reporting agent not updated or reporting delay

Resolution:

# Force WSUS client reporting
wuauclt /reportnow

# Verify update actually installed
Get-HotFix -Id KB5032190
# If not found, update failed silently - check Windows Update log

# Force inventory update in management platform
ansible server-name -m setup
ansible server-name -m package_facts

See also

• Vulnerability Management -for vulnerability scanning and assessment
• Vulnerability Remediation -for fixing identified vulnerabilities
• Change Management -for change request procedures
• Backup Verification -for restore capability testing
• Operating System Upgrade -for major OS upgrades