Offline Data Architecture

Offline data architecture is a structural pattern that enables applications to function without continuous network connectivity by storing data locally and synchronising with central systems when connections become available. The pattern addresses the fundamental challenge of field operations where internet access is intermittent, unreliable, or entirely absent for extended periods. Organisations operating in remote locations, disaster-affected areas, or regions with limited infrastructure require systems that continue working regardless of connectivity state.

Problem context

Field operations generate data that must be captured at the point of activity: beneficiary registrations during distributions, health assessments at mobile clinics, protection case updates during home visits, and monitoring observations at programme sites. These activities occur in locations where connectivity is constrained by geography, infrastructure damage, cost, or intentional network restrictions.

The connectivity constraints fall into distinct categories. Intermittent connectivity describes environments where internet access exists but drops unpredictably due to network congestion, power outages, or infrastructure limitations. Rural mobile networks in many operating contexts provide connectivity for 4 to 8 hours daily with speeds between 50 and 200 kbps. Scheduled connectivity occurs when staff travel periodically to locations with internet access, creating sync windows of 30 minutes to 2 hours at intervals of 1 to 7 days. Extended offline describes scenarios where devices remain disconnected for weeks or months, such as operations in areas without any telecommunications infrastructure or during acute emergencies when networks fail entirely.

The pattern applies when these forces converge: data must be captured at the point of activity rather than deferred; multiple users work on related data simultaneously across locations; the organisation requires eventual consistency with central systems for reporting and coordination; and operations cannot depend on real-time connectivity for core functions.

The pattern does not apply when operations occur in consistently connected environments, when data entry can wait until connectivity is available, or when the complexity of offline capability exceeds the operational benefit. A headquarters office with reliable internet gains nothing from offline architecture and incurs significant implementation and maintenance costs. Similarly, data that requires immediate central validation before use, such as real-time financial authorisations, cannot function in an offline model without fundamental workflow redesign.

Solution

The solution implements a local-first architecture where each client device maintains a complete local data store capable of supporting all application functions without network access. The local store is not a cache of remote data but the primary data source for the application. Central systems aggregate data from distributed local stores rather than serving as the authoritative source during normal operations.

+--------------------------------------------------------------------+
|                    LOCAL-FIRST ARCHITECTURE                        |
+--------------------------------------------------------------------+
|                                                                    |
|  +----------------------------+    +----------------------------+  |
|  |      FIELD DEVICE A        |    |      FIELD DEVICE B        |  |
|  |                            |    |                            |  |
|  |  +----------------------+  |    |  +----------------------+  |  |
|  |  |    APPLICATION       |  |    |  |    APPLICATION       |  |  |
|  |  +----------+-----------+  |    |  +----------+-----------+  |  |
|  |             |              |    |             |              |  |
|  |  +----------v-----------+  |    |  +----------v-----------+  |  |
|  |  |    LOCAL DATABASE    |  |    |  |    LOCAL DATABASE    |  |  |
|  |  |                      |  |    |  |                      |  |  |
|  |  | - Full data store    |  |    |  | - Full data store    |  |  |
|  |  | - Change log         |  |    |  | - Change log         |  |  |
|  |  | - Sync metadata      |  |    |  | - Sync metadata      |  |  |
|  |  +----------+-----------+  |    |  +----------+-----------+  |  |
|  |             |              |    |             |              |  |
|  +-------------|--------------+    +-------------|--------------+  |
|                |                                 |                 |
|                |   SYNC                          |    SYNC         |
|                |   QUEUE                         |    QUEUE        |
|                |                                 |                 |
+----------------|---------------------------------|-----------------+
                 +-----------------+---------------+
                                   |
                     CONNECTIVITY  |  (when available)
                                   |
                                   |
                                   |
                        +----------v-----------+
                        |                      |
                        |    CENTRAL SERVER    |
                        |                      |
                        |  +----------------+  |
                        |  | Sync Engine    |  |
                        |  +----------------+  |
                        |  | Conflict       |  |
                        |  | Resolution     |  |
                        |  +----------------+  |
                        |  | Consolidated   |  |
                        |  | Data Store     |  |
                        |  +----------------+  |
                        |                      |
                        +----------------------+

Figure 1: Local-first architecture with distributed data stores and central aggregation

The architecture comprises four core components. The local data store on each device holds a complete working dataset sufficient for all application functions. This store uses an embedded database such as SQLite, PouchDB, or IndexedDB depending on the client platform. The local store maintains not just current data but a log of all changes made while offline, including the sequence, timestamp, and originating device for each modification.

The sync engine manages bidirectional data transfer between local and central stores. When connectivity becomes available, the sync engine transmits local changes to the central server and retrieves changes from other devices that have previously synchronised. The engine operates incrementally, transferring only changes since the last successful sync rather than complete datasets. A sync session for a moderately active field device handling 50 to 100 records daily completes in 30 seconds to 2 minutes on a 100 kbps connection.

The conflict resolution layer handles cases where multiple devices modify the same record while disconnected from each other. Every record carries version metadata enabling the detection of divergent modifications. The resolution mechanism applies deterministic rules to produce consistent outcomes regardless of which device synchronises first.

The central aggregation store consolidates data from all field devices into a unified dataset for reporting, analysis, and coordination. This store represents the eventually consistent view of all field operations but is not the authoritative source during data capture. Field devices continue operating with their local data regardless of central store availability.

Data partitioning

Effective offline operation requires partitioning data so that each device holds the subset relevant to its operations. A device deployed for beneficiary registration in one district does not require beneficiary records from other districts. Partitioning reduces local storage requirements, accelerates sync operations, and limits the scope of potential conflicts.

+-------------------------------------------------------------------+
|                    PARTITIONING STRATEGIES                        |
+-------------------------------------------------------------------+
|                                                                   |
|  GEOGRAPHIC PARTITIONING                                          |
|  +------------------+  +------------------+  +------------------+ |
|  | Device: Turkana  |  | Device: Marsabit |  | Device: Wajir    | |
|  |                  |  |                  |  |                  | |
|  | Data scope:      |  | Data scope:      |  | Data scope:      | |
|  | - Turkana HHs    |  | - Marsabit HHs   |  | - Wajir HHs      | |
|  | - Local sites    |  | - Local sites    |  | - Local sites    | |
|  | - Local staff    |  | - Local staff    |  | - Local staff    | |
|  +------------------+  +------------------+  +------------------+ |
|                                                                   |
|  FUNCTIONAL PARTITIONING                                          |
|  +------------------+  +------------------+  +------------------+ |
|  | Device: Reg Team |  | Device: Health   |  | Device: M&E      | |
|  |                  |  |                  |  |                  | |
|  | Data scope:      |  | Data scope:      |  | Data scope:      | |
|  | - Registration   |  | - Health records |  | - Indicator data | |
|  | - Demographics   |  | - Referrals      |  | - Surveys        | |
|  | - Verification   |  | - Follow-ups     |  | - Observations   | |
|  +------------------+  +------------------+  +------------------+ |
|                                                                   |
|  TEMPORAL PARTITIONING                                            |
|  +------------------+  +------------------+  +------------------+ |
|  | Active period    |  | Recent archive   |  | Deep archive     | |
|  | (0-90 days)      |  | (90-365 days)    |  | (>365 days)      | |
|  |                  |  |                  |  |                  | |
|  | Sync: Always     |  | Sync: On demand  |  | Sync: Never      | |
|  | Local: Required  |  | Local: Optional  |  | Local: Excluded  | |
|  +------------------+  +------------------+  +------------------+ |
|                                                                   |
+-------------------------------------------------------------------+

Figure 2: Data partitioning strategies for offline devices

Geographic partitioning assigns data based on operational location. A device deployed in Turkana County receives and synchronises only records associated with that county. This approach works well when operations are location-bound and staff rarely cross geographic boundaries. The partition boundaries align with administrative structures such as districts, sub-counties, or programme zones.

Functional partitioning assigns data based on operational role. A health team device receives health assessment records, referral data, and follow-up schedules but not distribution records or financial data. This approach works when different teams operate in the same geographic area with minimal data overlap.

Temporal partitioning separates data by age. Active records from the past 90 days synchronise to all relevant devices. Records between 90 and 365 days old remain available on request but do not synchronise automatically. Records older than 365 days exist only in the central store. This approach bounds local storage growth while maintaining access to recent operational data.

Most implementations combine these strategies. A device for a health team in Marsabit County uses geographic partitioning (Marsabit only), functional partitioning (health data only), and temporal partitioning (active cases from past 90 days). The resulting local dataset is small enough for mobile device storage while containing everything needed for field operations.

Conflict detection and resolution

When two devices modify the same record while disconnected, a conflict arises during synchronisation. Conflict handling determines the system’s behaviour when this occurs, and the chosen strategy has significant implications for data integrity and user experience.

The sync engine detects conflicts by comparing version vectors. Each record carries a vector containing the last known version from each device that has modified it. When a device attempts to sync a record, the engine compares the incoming vector with the stored vector. If neither vector dominates the other (meaning each contains at least one version the other lacks), the modifications are concurrent and conflict.

+------------------------------------------------------------------+
|                    CONFLICT DETECTION                            |
+------------------------------------------------------------------+
|                                                                  |
|  Device A modifies record at version [A:3, B:2]                  |
|  Device B modifies same record at version [A:2, B:3]             |
|                                                                  |
|  Neither dominates:                                              |
|    A:3 > A:2  (A ahead)                                          |
|    B:2 < B:3  (B ahead)                                          |
|                                                                  |
|  Result: CONFLICT DETECTED                                       |
|                                                                  |
+------------------------------------------------------------------+
|                                                                  |
|  RESOLUTION STRATEGIES                                           |
|                                                                  |
|  +--------------------+                                          |
|  | Conflict detected  |                                          |
|  +---------+----------+                                          |
|            |                                                     |
|            v                                                     |
|  +---------+----------+                                          |
|  | Automatic or       |                                          |
|  | manual resolution? |                                          |
|  +---------+----------+                                          |
|            |                                                     |
|     +------+------+                                              |
|     |             |                                              |
|     v             v                                              |
|  +--+--------+  +-+------------+                                 |
|  | AUTOMATIC |  | MANUAL       |                                 |
|  +--+--------+  +-+------------+                                 |
|     |             |                                              |
|     v             v                                              |
|  +--+--------+  +-+------------+                                 |
|  | Apply     |  | Queue for    |                                 |
|  | rule:     |  | human review |                                 |
|  | - LWW     |  |              |                                 |
|  | - Merge   |  | Preserve     |                                 |
|  | - Custom  |  | both versions|                                 |
|  +-----------+  +--------------+                                 |
|                                                                  |
+------------------------------------------------------------------+

Figure 3: Conflict detection mechanism and resolution strategy selection

Last-write-wins (LWW) resolution automatically selects the modification with the most recent timestamp. This strategy is simple to implement and produces deterministic outcomes, but it discards earlier modifications entirely. LWW works well for fields where only the most recent value matters, such as status updates, timestamps, and flags. It fails badly for cumulative data where both modifications contain important information.

Field-level merge resolution compares individual fields rather than complete records. When Device A modifies the beneficiary’s phone number while Device B modifies the address, the merged record contains both changes. This strategy preserves more data than LWW but requires schema-aware conflict handling and fails when both devices modify the same field.

Custom resolution rules encode domain-specific logic. In a beneficiary registration system, a conflict on vulnerability status resolves by selecting the higher vulnerability level on the principle that overcounting vulnerability is safer than undercounting. In an inventory system, quantity conflicts resolve by summing the changes (both devices recorded distributions that reduced stock). Custom rules require significant implementation effort but produce outcomes aligned with operational intent.

Manual resolution queues conflicting records for human review. A data manager examines both versions and determines the correct merged state. This strategy produces the highest quality outcomes but creates operational overhead. Records remain in a conflicted state until reviewed, which blocks downstream processes that depend on authoritative data.

The appropriate strategy varies by data type. Implementing a conflict resolution matrix that maps each entity type to a resolution strategy enables consistent handling:

Sync protocol

The sync protocol defines the sequence of operations when a device connects and synchronises with the central server. A well-designed protocol handles partial sync (connection lost mid-transfer), bandwidth constraints, and priority ordering for limited sync windows.

  Device                          Server
    |                               |
    |---(1) Auth + last sync seq--->|
    |                               |
    |<--(2) Changes since seq-------|
    |       [priority ordered]      |
    |                               |
    |---(3) ACK received changes--->|
    |                               |
    |---(4) Local changes---------->|
    |       [priority ordered]      |
    |                               |
    |<--(5) ACK + conflicts---------|
    |                               |
    |---(6) Conflict resolutions--->|
    |       [if manual]             |
    |                               |
    |<--(7) Final ACK + new seq-----|
    |                               |
    |---(8) Update local seq--------|
    |       [sync complete]         |
    |                               |

Figure 4: Sync protocol sequence with acknowledgement and conflict handling

The protocol begins with the device authenticating and transmitting its last successful sync sequence number. This number identifies the point in the change log from which the device requires updates. The server responds with all changes logged since that sequence, ordered by priority. High-priority entities such as active beneficiaries and open cases transmit before lower-priority entities such as historical records and reference data.

The device acknowledges receipt of server changes, then transmits its own local changes accumulated since the last sync. The server processes these changes, detecting conflicts where applicable. The response to the device includes acknowledgement of received changes and a list of any conflicts requiring attention. For automatic resolution strategies, the server includes the resolved values. For manual resolution, the server includes both conflicting versions.

The device processes conflict information, applying automatic resolutions to the local store and queuing manual conflicts for user attention. A final acknowledgement confirms the sync completed, and both device and server update their sequence numbers.

The protocol handles interruption gracefully. If connection drops during step 4, the device retains unacknowledged local changes and retransmits them on the next sync attempt. The server ignores duplicate changes based on their unique identifiers and version vectors. Resumption from any interruption point produces the same final state as an uninterrupted sync.

Priority ordering ensures the most operationally important data synchronises first when sync windows are limited. A 30-minute connectivity window at a weekly market town visit synchronises active case updates before attempting to sync historical records. Configuration specifies entity priorities and the minimum acceptable sync scope for a session to be marked successful.

Implementation

Implementation begins with selecting a data synchronisation stack appropriate to the client platform and organisational technical capacity. The primary decision is between using a purpose-built sync framework and implementing custom synchronisation logic.

CouchDB with PouchDB provides a mature sync framework with built-in conflict detection, revision tracking, and bidirectional replication. CouchDB runs as the central server while PouchDB operates as the local database in browser-based or mobile applications. The replication protocol handles all sync mechanics, and conflicts surface through a standard API for application-level resolution. This stack suits organisations with JavaScript development capacity building web or hybrid mobile applications. Storage overhead runs approximately 20% above raw data size due to revision history.

SQLite with custom sync provides maximum flexibility at the cost of implementation effort. SQLite operates on all mobile platforms with native performance. The organisation implements change tracking (typically through triggers populating a changes table), sync protocol (typically REST API calls), and conflict handling according to its specific requirements. This approach suits organisations with strong mobile development capacity needing tight integration with native platform features.

Realm with Atlas Device Sync provides a commercial solution with automatic conflict resolution and real-time sync when connected. Realm’s object database maps naturally to mobile application data structures. Atlas Device Sync handles the sync protocol with configurable conflict resolution. This stack suits organisations prioritising development speed over cost, with sync licensing fees of approximately $30 per active device per month.

The implementation sequence follows a consistent pattern regardless of stack selection:

Local schema design establishes the data structures stored on devices. The schema includes all entities required for offline operation plus metadata fields for sync tracking: record identifier, version vector, last modified timestamp, originating device, and sync status flag. Schema design must account for storage constraints; a 32GB mobile device with 4GB available for application data supports approximately 200,000 moderately complex records with revision history.

Change capture mechanisms record all local modifications. In SQLite implementations, database triggers fire on insert, update, and delete operations, writing to a changes log table with the affected record, operation type, timestamp, and before/after values for updates. In CouchDB/PouchDB, the database’s built-in revision system captures changes automatically.

Sync endpoint implementation creates the server-side API for receiving and distributing changes. The endpoint authenticates requests, determines the device’s partition scope, retrieves changes since the specified sequence, and returns them in priority order. For inbound changes, the endpoint applies modifications, detects conflicts, and returns outcomes.

Conflict resolution implementation encodes the resolution strategy matrix into executable logic. Custom resolution rules require careful testing with synthetic conflict scenarios covering all entity types and field combinations.

Offline mode detection determines application behaviour based on connectivity state. The application monitors network availability and adjusts its user interface accordingly: hiding sync-dependent features when offline, displaying last-sync timestamps, and triggering sync operations when connectivity returns.

Sync queue management handles background synchronisation without blocking user operations. The sync process runs in a background thread or worker, with progress and status information available to the user interface. Sync failures trigger retry with exponential backoff: 30 seconds, 1 minute, 2 minutes, 5 minutes, 10 minutes, then holding at 10-minute intervals until success.

Configuration parameters

Effective offline operation requires configuration aligned with operational reality. The following parameters require explicit values during implementation:

Sync window configuration adapts to known connectivity patterns. For operations where staff visit a location with internet weekly, configure the sync process to attempt full synchronisation including lower-priority historical data. For operations with only brief daily connectivity, configure sync to prioritise current operational data and skip historical synchronisation entirely.

Consequences

Adopting offline data architecture produces specific trade-offs that organisations must accept.

Eventual consistency replaces immediate consistency. When Device A records a new beneficiary, that record does not appear on Device B until both devices synchronise with the central server. The delay between creation and universal visibility ranges from minutes (when both devices have continuous connectivity) to weeks (when devices sync only during periodic headquarters visits). Operations requiring immediate cross-device visibility, such as preventing duplicate registrations during simultaneous distribution activities, require additional mechanisms like pre-allocated identifier ranges or connectivity-dependent workflows.

Conflict management introduces operational overhead. Even with automatic resolution strategies, some conflicts require human review. Organisations must allocate staff time for conflict review, establish escalation paths for complex cases, and train field teams on conflict-minimising data entry practices. In a typical implementation, 2% to 5% of synced records generate conflicts requiring attention, with 0.5% to 1% requiring manual resolution.

Storage constraints limit data scope. Mobile devices have finite storage, and offline applications compete with photos, other applications, and operating system requirements. A realistic allocation of 2GB to 4GB for application data bounds the local dataset size. Organisations with large historical datasets must implement aggressive temporal partitioning, which complicates historical data access from field locations.

Complexity cost affects development and maintenance. Offline-capable applications require 40% to 60% more development effort than equivalent online-only applications. The additional complexity appears in sync logic, conflict handling, offline mode detection, queue management, and testing across connectivity states. Maintenance burden increases proportionally, as sync bugs are difficult to reproduce and diagnose.

Testing requirements expand significantly. The application must be tested in online, offline, and transitional states. Conflict scenarios require testing across all entity types and resolution strategies. Sync interruption and resumption testing ensures data integrity across failure modes. A comprehensive offline application test suite runs 3 to 5 times longer than an equivalent online application suite.

The pattern produces significant benefits when applied appropriately. Field operations continue regardless of connectivity, eliminating the common frustration of lost work or blocked activities. Data capture at the point of activity improves accuracy compared to transcription from paper forms during connectivity windows. Staff productivity increases when time previously spent waiting for connectivity becomes productive working time.

Variants

Full offline

The full offline variant assumes devices never connect to networks and synchronise only through physical media transfer. A field device operates for the duration of a deployment (weeks to months), then returns to a location where data is extracted to USB storage and transferred to headquarters systems.

This variant eliminates network dependency entirely but requires robust export mechanisms, physical media handling procedures, and import processing at headquarters. Conflict detection operates at import time with all resolution happening centrally. The variant suits operations in areas with no telecommunications infrastructure or where network usage creates security risks.

Implementation requires an export function that produces a complete, self-describing data package including schema version, device identifier, export timestamp, and all local changes since the previous export. The import processor validates package integrity, checks schema compatibility, applies changes, and reports conflicts for resolution.

Selective sync

The selective sync variant synchronises subsets of local changes based on priority and connectivity quality. When the device detects a low-bandwidth connection (under 50 kbps), it synchronises only critical entities: new beneficiaries, status changes, and alerts. When bandwidth improves, it synchronises additional entities progressively.

This variant maximises the value of constrained connectivity by ensuring the most important data moves first. It requires bandwidth detection, priority classification of all entities, and a sync process that adjusts its scope dynamically.

Implementation adds connection quality assessment to the sync protocol. Before transmitting changes, the device tests current throughput with a small payload. Based on measured bandwidth and configured thresholds, it selects an appropriate sync scope: minimal (critical only), standard (current operational data), or full (including historical and reference data).

Store and forward

The store-and-forward variant uses intermediary devices to relay sync data between field devices and central servers. A supervisor’s tablet with periodic connectivity aggregates changes from multiple field worker phones via local wireless, then forwards the combined changes when reaching internet connectivity.

This variant extends the reach of limited connectivity by concentrating it at a single relay point. Field devices sync locally with the relay device using Bluetooth or local WiFi without internet. The relay device handles central server synchronisation.

Implementation requires a relay mode in the sync engine. The relay device receives changes from multiple source devices, stores them locally, and transmits them to the central server with source device attribution preserved. Return changes for each source device are received, stored, and distributed during the next local sync session. Conflict handling occurs at the central server; the relay device passes resolution information back to source devices.

Anti-patterns

Assuming connectivity will improve

Designing for offline as a temporary state until connectivity improves leads to architectural decisions that fail under sustained disconnection. Applications that disable features progressively as offline duration extends, or that require connectivity for critical operations scheduled “for later”, leave field staff unable to complete their work.

The correct approach treats offline as the normal state. All core functions operate fully offline. Connectivity, when available, provides an optimisation (faster sync, larger data scope) rather than a requirement.

Ignoring conflict potential

Assuming conflicts will be rare and handling them as exceptions leads to data integrity problems. When conflicts are unhandled, one device’s changes silently overwrite another’s. When conflicts are handled only through error messages, they accumulate in queues until the backlog overwhelms review capacity.

The correct approach designs conflict handling into every entity from the start. Each entity type has an explicit resolution strategy. Conflict queues have size limits and escalation triggers. Dashboards expose conflict rates for operational monitoring.

Synchronising everything

Attempting to synchronise complete datasets to every device creates storage problems, bandwidth problems, and conflict explosion. A device receiving all beneficiary records from all locations in a country-wide programme stores data it will never use while generating conflicts with devices in other locations.

The correct approach partitions data based on operational scope. Each device receives only the data relevant to its assigned function and location. Reference data that rarely changes synchronises separately from operational data that changes frequently.

Client-side timestamp trust

Using device-generated timestamps for conflict resolution without validation allows a misconfigured device clock to corrupt resolution logic. A device with its clock set one year in the future wins all last-write-wins resolutions regardless of actual modification order.

The correct approach validates timestamps against server time during sync, rejects or adjusts timestamps outside acceptable bounds (such as within 24 hours of server time, accounting for timezone differences), and uses logical clocks (version vectors) rather than physical timestamps for ordering when sequence matters.

Skipping offline testing

Testing only with simulated offline (network disabled) misses failure modes that occur during connectivity transitions. Applications that work perfectly offline and online break when the connection drops mid-operation or returns during sync.

The correct approach tests in actual field conditions or in lab conditions that replicate field connectivity patterns. Network simulation tools that introduce latency, packet loss, and intermittent disconnection expose failure modes invisible in binary online/offline testing.

See also

• Data Synchronisation Setup
• Sync Conflict Resolution
• Intermittent Connectivity Patterns
• Last-Mile Connectivity
• Data Pipeline Design
• Mobile Network Utilisation