Data Sync Patterns

Sync Modes

Backfill Pattern

1. Start with a minimal field set

   Start with the smallest set of fields that proves the pipeline works.

   You can prove the pipeline end-to-end before chasing edge cases.
2. Adopt cursor pagination

   Use cursor pagination wherever the endpoint supports it.

   The walk is deterministic and can resume after a failure.
3. Persist the checkpoint alongside the write

   Persist both the downstream write result and the last successful cursor checkpoint.

   The next run restarts exactly where the last left off.
4. Expand the field set

   Only after the happy path is stable, widen the field set.

   You can load new fields without risking the existing load.

let cursor: string | null = null

while (true) {
const url = new URL("https://api.lev.com/api/external/v2/deals")
url.searchParams.set("limit", "100")
if (cursor) url.searchParams.set("cursor", cursor)

const response = await fetch(url, {
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "X-Origin-App": "warehouse-sync",
  },
})

const payload = await response.json()
await writeBatch(payload.data)

if (!payload.pagination?.has_more) break
cursor = payload.pagination.next_cursor
}

Incremental Pattern

Use an updated_at or similar time-based filter together with a persisted checkpoint:

• Store the timestamp of the last fully successful run.
• Re-read a small overlap window to tolerate clock skew and delayed writes.
• Deduplicate in your destination using resource IDs.

Operational Checklist

• Monitor request_id values for failed batches.
• Alert on repeated 401, 403, and 429 responses.
• Keep writes idempotent in your destination so replaying a batch is safe.
• Version your destination schema deliberately as Lev fields expand.