Design Migrate 1PB of Data to Cloud - System Design

“Move 1PB to the cloud” sounds like a copy job, and people design it like one: point a script at the source, rsync to an S3 bucket, wait. Then the interviewer drops the two numbers that change everything. The link is 10 Gbps. The files are in active use during the migration. Now do the arithmetic: at 10 Gbps, wire-speed and perfect, 1PB takes about 9.5 days of continuous transfer. Real throughput after overhead and contention with production traffic is half that, so ~20 days. And for those 20 days the source data keeps changing underneath you, so whatever you copied on day one is already stale by day two.

The single decision that shapes everything: you cannot move a petabyte over that wire in a reasonable window, and you cannot freeze the source while you try. That forces two independent problems apart - bulk transfer of the huge static baseline (solved with physical shipping, not the network) and reconciliation of the changes that happen during and after the bulk move (solved with change capture and a tiny final cutover). Get that split right and the rest is verification and orchestration.

Let me do it properly.

Functional Requirements (FR)

In scope:

• Move 1PB of data from an on-prem datacenter to cloud object storage (S3 or GCS) in full.
• Keep the source live during migration. Applications continue reading and writing the on-prem data the whole time. No maintenance window measured in days.
• Minimal downtime at cutover. The final switch from on-prem to cloud as the system of record must fit in a short window (minutes to low single-digit hours), not a multi-day freeze.
• Data integrity. Every byte that lands in the cloud must be verifiably identical to the source. No silent corruption, no missed files.
• Handle files that change during migration. A file copied on day 3 that is edited on day 10 must end up with its day-10 contents in the cloud, not its day-3 contents.
• Resumability. A transfer that dies at 60% must resume, not restart. At this scale a full restart is catastrophic.

Explicitly out of scope (say this out loud so you control scope):

• Application re-architecture. We move the data; rewriting the apps to be cloud-native is a separate project. We do assume apps can be repointed at a new storage endpoint at cutover.
• Schema/format transformation. This is a lift of bytes, not an ETL. If the source is files, they land as objects; if it is a database, we treat DB replication separately (noted, not designed in depth here).
• Ongoing hybrid operation as a permanent state. The goal is a one-time migration ending in cloud as system of record, not a forever-federated setup.
• Cost optimization of the final cloud layout (storage classes, lifecycle) beyond a mention. Get it there correctly first.

The one decision that drives the design: bandwidth-bound bulk + live-source delta. The petabyte baseline goes physical; the deltas go over the wire. Every choice serves keeping downtime near zero while guaranteeing byte-for-byte correctness.

Non-Functional Requirements (NFR)

• Scale: 1PB total (call it ~1,000 TB). Assume ~200 million files averaging ~5MB, with a long tail from tiny config files to multi-GB media/backup blobs. File count matters as much as byte count - 200M tiny objects is a metadata problem, not just a bandwidth one.
• Network: a single 10 Gbps link to the cloud, shared with production egress. Usable migration throughput budgeted at ~5 Gbps so we do not starve live traffic.
• Cutover downtime target: under 1 hour of write-freeze, ideally minutes. Read downtime effectively zero.
• Durability: target 11 nines in the cloud (native to S3/GCS). Zero data loss during transfer - every object verified by checksum end to end.
• Consistency at cutover: the cloud copy must be a point-in-time-consistent, complete image of the source as of the freeze instant. No file left behind, no file half-written.
• Availability of the source: 100% during migration. The migration is a background citizen; production always wins for bandwidth and IO.
• Resumability / fault tolerance: any component (an appliance, a worker, the delta pipeline) can die and resume from its last committed checkpoint without recopying completed work.

Back-of-the-Envelope Estimation (BoE)

The numbers are the whole argument here, so do them first and loud.

Can we do it over the network at all?

Link: 10 Gbps = 10 / 8 = 1.25 GB/sec at wire speed (theoretical max)
1 PB = 1,000,000 GB (using 1000-based TB/PB for transfer math)

Time at full 10 Gbps, 100% utilization:
  1,000,000 GB / 1.25 GB/sec = 800,000 sec = 9.26 days

Realistic: we can only take ~5 Gbps (half the link) to spare production,
and real throughput after TCP/TLS/overhead/contention ~ 60% of that:
  effective ~ 3 Gbps = 0.375 GB/sec
  1,000,000 GB / 0.375 GB/sec = 2,666,666 sec ≈ 30.9 days

Roughly a month of continuous, flawless network transfer while contending with production. That is the killer number. It is also why the source-changes-during-migration problem is unavoidable: a month is plenty of time for a large fraction of the data to be rewritten.

Physical transfer as the alternative for the baseline:

AWS Snowball Edge ≈ 80 TB usable per device.
1 PB / 80 TB ≈ 13 devices.
AWS Snowmobile (if it existed for you) or a fleet of Snowball devices.

Fill time is local-LAN/disk bound, not WAN bound:
  local ingest ~ a few days per batch of devices in parallel.
Ship + cloud-side ingest ~ 1 week door to door per batch.

Net: the 1PB baseline lands in the cloud in ~1-2 weeks physically,
versus ~1 month over the wire, AND it does not touch the 10 Gbps link,
leaving the full link for production + delta sync.

How much delta accumulates during migration?

Assume a 5% daily change rate on active data (typical for a working set).
Over ~14 days of physical migration:
  changed bytes ≈ 1 PB * 5%/day * 14 days worst-case ≈ large,
  but overlapping edits collapse: unique changed set is far smaller.
  Estimate unique changed data ≈ 5-10% of total = 50-100 TB.

Delta sync over the wire at effective 3 Gbps:
  100 TB / 0.375 GB/sec ≈ 74 hours ≈ 3 days.

So the deltas - a few tens of TB - do fit over the network in days, even though the full petabyte does not. That is the entire strategy in one line: ship the 1PB baseline physically, sync the ~50-100TB of deltas over the wire.

Final-cutover delta (the freeze window):

Once the delta pipeline has caught up to within minutes of live,
the residual change to flush at freeze is tiny:
  a few minutes of writes at, say, 5% daily rate:
  1 PB * 5%/day / 86400 sec * 300 sec ≈ ~170 GB (very rough upper bound)
  in practice, with continuous CDC, seconds of lag = single-digit GB.
Flush 170 GB at 3 Gbps ≈ 7.5 min. Realistically minutes.

Metadata / file-count pressure:

200M files. Even at 1ms of overhead per object (list, checksum record,
manifest entry) that is 200,000 sec ≈ 55 hours of pure metadata bookkeeping
if done serially. This MUST be parallelized and batched, and tiny files
must be aggregated - per-object cloud PUT overhead dominates for small files.

Verification cost:

Checksum 1 PB once: reading 1 PB off disk to hash it, at ~2 GB/sec aggregate
disk throughput across many parallel workers:
  1,000,000 GB / 2 GB/sec = 500,000 sec ≈ 5.8 days of hashing,
parallelized across the source fleet. Overlap it with transfer, do not
serialize it after.

The BoE settles the architecture before a single box is drawn: network-only is a month and still leaves you inconsistent; physical baseline plus network delta is ~2 weeks with a minutes-long cutover.

High-Level Design (HLD)

Three phases, not one pipeline: (1) Baseline snapshot + physical bulk transfer, (2) Continuous delta sync over the wire while the source stays live, (3) Verify + cutover. The architecture is the machinery that runs those phases and reconciles them.

        ON-PREM DATACENTER                              CLOUD (AWS / GCP)
 ┌───────────────────────────────┐            ┌────────────────────────────────┐
 │        Applications           │            │       Applications (new)        │
 │   (live reads + writes)       │            │   repointed at cutover          │
 └───────────────┬───────────────┘            └───────────────┬────────────────┘
                 │ read/write                                  │
      ┌──────────▼───────────┐                     ┌───────────▼─────────────┐
      │   Source Storage     │                     │   Object Store (S3/GCS) │
      │  (NAS / filesystem /  │                     │   baseline + deltas      │
      │   HDFS / block)      │                     │   = system of record     │
      └───┬───────────┬──────┘                     └───────────▲─────────────┘
          │           │ change events                          │
          │           │ (CDC / inotify / snapshot diff)        │
          │     ┌─────▼──────────┐                             │
          │     │  Change Log /  │                             │
          │     │  CDC Capture   │──── delta objects ──────────┤ (over 10 Gbps
          │     └─────┬──────────┘         │ (5 Gbps budget)   │  WAN link)
          │           │                    │                   │
   ┌──────▼─────┐  ┌──▼────────────┐  ┌────▼─────────┐   ┌──────┴──────────┐
   │ Snapshot / │  │ Delta Sync    │  │ Transfer     │   │ Cloud Ingest +  │
   │ baseline   │  │ Workers       │  │ Manager      │   │ Manifest Merge  │
   │ reader     │  │ (rsync/CDC    │  │ (orchestrator│   │ Service         │
   │ (parallel) │  │  apply)       │  │  + checkpoints│   └──────┬──────────┘
   └─────┬──────┘  └───────────────┘  └──────────────┘          │
         │ bulk read                                     ┌───────▼─────────┐
   ┌─────▼───────────────┐    ship     ┌──────────────┐  │  Verification   │
   │ Physical Appliances │────────────►│ Cloud Import │  │  Service        │
   │ (Snowball x13 /     │   trucks    │ (data center)│  │ (checksum diff, │
   │  Transfer Appliance)│◄────────────│              │  │  manifest audit)│
   └─────────────────────┘             └──────────────┘  └─────────────────┘

        Control plane spanning both sides:
   ┌───────────────────────────────────────────────────────────────────────┐
   │  Manifest DB (every file: path, size, mtime, checksum, phase, status)  │
   │  Orchestrator (phase state machine, checkpoints, retries, cutover gate) │
   └───────────────────────────────────────────────────────────────────────┘

Phase 1 - baseline (physical):

1. Take a point-in-time snapshot of the source (filesystem/LVM/storage-array snapshot) so the baseline is internally consistent and reads do not fight live writes. Record the snapshot marker (a monotonic sequence number, WAL position, or snapshot timestamp) - this is the delta pipeline’s starting line.
2. Baseline readers walk the snapshot in parallel, writing every file into the manifest DB (path, size, mtime, checksum) and streaming bytes onto physical appliances (AWS Snowball / GCP Transfer Appliance), ~13 devices, filled over LAN at local-disk speed.
3. Appliances ship to the cloud provider; provider ingests them into the object store. Cloud-side, the Ingest + Manifest Merge Service records each landed object’s checksum back into the manifest.

Phase 2 - delta sync (network), runs concurrently and after:

4. From the moment of the snapshot marker, a CDC / change-capture mechanism records every create/update/delete on the live source into a change log.
5. Delta sync workers continuously drain that log, transferring changed files (or byte-ranges) over the 10 Gbps link into the object store, updating the manifest. This runs while appliances are still in transit and keeps running until the cloud copy is within seconds of live.

Phase 3 - verify + cutover:

6. The Verification Service audits the manifest: every source file has a cloud object with a matching checksum; no orphans, no missing entries.
7. When delta lag is near zero and verification is green, the orchestrator opens the cutover gate: briefly freeze writes on-prem, flush the final residual delta (minutes), verify once more, then repoint applications at the cloud endpoint. On-prem becomes read-only fallback.

The key insight: the baseline never touches the network, and the network never has to catch a moving petabyte - only its deltas.

Component Deep Dive

The genuinely hard parts are (1) bulk transfer of the baseline, (2) capturing and reconciling changes on a live source, (3) the cutover with minimal downtime, and (4) verifying a petabyte is actually correct. I walk each from naive to scalable.

1. Bulk transfer of the 1PB baseline

This is where the whole design lives or dies, so it comes first.

Approach A: rsync/aws s3 cp over the network (the naive instinct)

Point a fleet of workers at the source, copy everything to the bucket over the 10 Gbps link.

for file in $(find /data -type f); do
    aws s3 cp "$file" "s3://target/$file"
done      # ...parallelized, but still over the WAN

Where it breaks at scale:

• Time. As the BoE showed, ~1 month at a realistic shared-link throughput. A month-long migration is not a plan, it is a liability - a month of drift, a month of failure exposure, a month of the link saturated.
• Production starvation. The migration and live traffic share one 10 Gbps pipe. Push migration hard and you throttle the business; throttle migration to protect the business and the month becomes two.
• Failure amplitude. Over a month, transient network failures are certain. Without perfect resumability every blip risks recopying huge ranges. TCP throughput over a long fat pipe with any loss collapses (bandwidth-delay product), so you rarely even hit line rate.
• 200M small PUTs. Per-object request overhead and rate limits on the cloud side make 200M individual uploads its own multi-day bottleneck, independent of raw bytes.

This works to move a few TB. At 1PB over a shared 10 Gbps link, it is the mistake.

Approach B: physical transfer appliances for the baseline (the fix)

Move the bytes on disks in a truck. Bandwidth of a truck full of drives is absurd; latency is a week. For a static petabyte, throughput wins.

1 PB / 80 TB per Snowball Edge ≈ 13 appliances.

Local fill: baseline readers write to appliances over the LAN
(10-40 GbE internal), not the WAN. Local disk/network bound:
each device fills in hours-to-days; run several in parallel.

Ship → provider ingests directly into the object store.
Door-to-door ≈ 1 week per batch; total baseline in cloud ≈ 1-2 weeks.

Why this is right:

• It frees the WAN link entirely for the delta phase and for production. The petabyte moves on wheels; the 10 Gbps pipe only ever carries deltas.
• Deterministic, bounded time that does not degrade with packet loss or link contention. A week is a week.
• Encryption + tamper evidence are built into these appliances (KMS-managed keys, hardware encryption), so shipping disks is not a security regression.
• Parallel batches: while batch 1 is in transit, batch 2 is filling. You pipeline the physical logistics.

The critical subtlety - snapshot consistency. You cannot fill 13 appliances over several days off a live source and get a coherent image; different files would be captured at different times, some mid-write. So Phase 1 reads from a storage snapshot (LVM/ZFS/array snapshot or an HDFS distcp off a frozen snapshot), giving a single point-in-time baseline. The snapshot is the anchor: everything after its marker is the delta phase’s job. Live writes continue against the real volume, unaffected, because the snapshot is copy-on-write.

Small-file aggregation. 200M tiny files as 200M objects is a metadata disaster (PUT overhead, listing cost). Baseline readers pack small files into large container objects (e.g. TAR/archive shards of ~1GB) with an index, or use the appliance’s native batching, so the object store sees millions, not hundreds of millions, of objects. Large files stream directly, multipart. The manifest records the mapping so a file can still be located inside its container.

2. Capturing changes on a live source (the delta problem)

The source is being written to for the entire ~2 weeks. Whatever the baseline captured is stale on arrival. This is the part that separates a real migration from a copy job.

Approach A: copy twice, diff at the end (the trap)

Copy everything once (physical), then near cutover run a second full rsync that compares and copies whatever changed.

Where it breaks: a second pass still has to scan all 200M files to discover what changed - re-stating a petabyte’s worth of files takes many hours to days and hammers the source’s metadata layer, competing with production. Worse, it only converges if the change rate during the scan is lower than the scan speed; on a busy source the scan can chase a moving target and never quite finish, so cutover downtime becomes unbounded. You have turned “copy 1PB” into “repeatedly scan 200M files,” which is not obviously better.

Approach B: continuous change data capture from the snapshot marker (the fix)

Do not rediscover changes by scanning. Record them as they happen, starting from the exact snapshot marker, and stream them continuously.

The capture mechanism depends on the source:

• Filesystem/NAS: filesystem audit/change journals (inotify/fanotify at scale is fragile for millions of files; prefer the storage array’s native change-block tracking or a filesystem changelog like ZFS/GPFS policy scans, or NetApp SnapDiff). Emit {path, op, new_mtime} events into a durable change log.
• Block storage: changed-block tracking (CBT) - the array knows which blocks changed since the snapshot, so you resync only dirty blocks, not whole files.
• Object storage source: bucket event notifications / versioning.
• Databases (if in scope): real CDC off the WAL/binlog/oplog (Debezium-style), replicating into a cloud replica - a different track, noted here because “1PB” often includes DB volumes.

Snapshot marker S0 taken at Phase 1 start.
Change log records every op with op >= S0:
  { seq, path, op: create|update|delete, size, mtime }

Delta sync workers drain the log in order:
  - coalesce: if a file changed 40 times, transfer its FINAL state once.
  - transfer changed file (or changed byte-ranges for big files) to cloud.
  - apply deletes as tombstones/deletes on the object store.
  - advance the committed sequence checkpoint.

Why this is right:

• No repeated full scans. Cost is proportional to what changed, not to total data size. Discovering changes is O(changes), not O(200M files).
• Coalescing collapses churn: a hot file rewritten 100 times transfers its latest version once, near cutover. This is why the residual delta at freeze is single-digit GB, not the naive sum of every write.
• Continuous convergence. Because it runs the whole time (even while appliances are in transit), by the time the baseline lands and is ingested, the delta pipeline is already streaming and can drive lag toward zero.
• Ordering + idempotency. Each object write is keyed by path and carries the source mtime/version; applying an older event over a newer one is rejected (last-writer-by-source-time wins), so out-of-order or retried events cannot corrupt the cloud state.

The ordering-vs-baseline race. A file might be captured in the baseline snapshot and appear as a delta event (if it changed after S0). That is fine and intended: the delta simply overwrites the baseline version, and because the delta carries a newer mtime, the idempotency guard keeps the newer one. A file deleted after S0 is in the baseline but must be removed in the cloud - the delta delete event tombstones it. Deletes are the easiest thing to forget and the classic migration data-integrity bug: without capturing deletes, the cloud accumulates files the source no longer has.

3. The cutover with minimal downtime

Everything so far exists to make this window small. The cutover is where “minimal downtime” is won or lost.

Naive: freeze, final full sync, switch

Stop all writes, run one last sync to catch everything, verify, switch.

Where it breaks: if “one last sync” means scanning or transferring anything large, the freeze is hours or days. That violates the whole point. Freeze time must be bounded by the residual delta only, which requires the delta pipeline to already be caught up before you freeze.

The fix: drive lag to near-zero first, then a tiny freeze

Cutover is a gated state machine, not a big-bang copy:

Precondition to even attempt cutover:
  - baseline fully ingested + verified in cloud
  - delta pipeline lag < a few seconds (it has caught up to live)
  - verification service reports 0 mismatches on the caught-up set

Cutover sequence (the short window):
  1. Enter read-only / write-freeze on the on-prem source
     (apps can still READ; only writes pause). Downtime starts.
  2. Flush the final residual delta events (single-digit GB, seconds-minutes)
     and wait for the delta pipeline checkpoint to reach the freeze marker.
  3. Final verification: manifest audit - every file present, checksums match.
  4. Repoint applications to the cloud endpoint (DNS/config/endpoint swap).
     Writes resume against the cloud. Downtime ends.
  5. Keep on-prem as read-only fallback for a rollback window (days).

• Write-freeze, not full-freeze. Reads continue against on-prem the entire time, so users mostly see nothing; only writes pause for minutes.
• The freeze is bounded by residual delta only, and continuous CDC keeps that residual tiny. This is the direct payoff of Approach 2B.
• Rollback safety. On-prem stays intact and read-only after cutover. If the cloud shows a problem in the first hours, you can fail back, because you never destroyed the source. Only after a confidence window do you decommission on-prem.
• Dual-write bridge (optional, for near-zero downtime). For systems that cannot tolerate even a minutes-long write freeze, run a brief dual-write period: the app writes to both on-prem and cloud, reads from on-prem, while the delta pipeline drains the last of the backlog. Once cloud is confirmed consistent, flip reads to cloud and stop writing on-prem. This trades application complexity for a near-zero freeze and is the tool to reach for only if the freeze budget is truly sub-minute.

4. Verifying a petabyte is actually correct

“It finished” is not “it is correct.” At 1PB across 200M files and 13 shipped appliances, silent loss or corruption is a real risk, and you must be able to prove the cloud copy is byte-identical.

Naive: trust the transfer tool’s exit code

The cp succeeded, so we are done.

Where it breaks: appliances can have bad sectors, ingest can drop objects, a delta event can be missed, an interrupted multipart upload can leave a truncated object that still “exists.” Exit codes do not catch a flipped bit or a missing file among 200M. At this scale, “probably fine” will contain thousands of corrupt or missing objects.

The fix: manifest-driven, end-to-end checksums

The manifest DB is the source of truth for correctness, populated on both sides:

Source side (during baseline read + each delta):
  for each file: record { path, size, mtime, sha256 } into manifest.
  Checksums are computed while reading (overlap hashing with transfer),
  parallelized across the reader fleet.

Cloud side (on ingest):
  for each landed object: compute checksum, write it back to manifest
  keyed by path. Object stores return per-object ETags/checksums
  (and per-part checksums for multipart) so this is cheap to compare.

Verification service:
  - AUDIT COMPLETENESS: every source manifest row has a cloud row.
    Missing → re-transfer that file (over the network, it is a small set).
  - AUDIT INTEGRITY: source.sha256 == cloud.sha256 for every path.
    Mismatch → re-transfer.
  - AUDIT DELETES: every source delete has a corresponding cloud tombstone.
  - Report: 0 missing, 0 mismatch, 0 orphan == green light for cutover.

• Checksums are computed end to end, not just claimed by the tool. The comparison is a set difference over the manifest - a database operation, not a re-read of a petabyte at verify time.
• Overlap hashing with transfer. The 5.8-day hashing cost from the BoE is paid during baseline reads across many workers, not serialized afterward, so it does not extend the timeline.
• Re-transfer is cheap because the mismatch set is small. You only recopy the specific failed files over the network, which is trivial bandwidth.
• The manifest is also the resumability ledger. Every file has a status (pending / on_device / in_cloud / verified). A crashed reader or worker resumes by querying “what is still pending,” so nothing is recopied and nothing is skipped.

API Design & Data Schema

This is a control-plane system, so the “API” is the orchestration and manifest interface, not a user-facing request path.

Control-plane API

POST /api/v1/migration/jobs
Body:
{
  "source":       "nas://dc1/data",
  "target":       "s3://target-bucket/prefix",
  "snapshot":     "auto",             // take a fresh snapshot at start
  "change_rate":  0.05,               // hint for planning
  "wan_budget_gbps": 5                // cap migration WAN usage
}
Response 201: { "job_id": "mig_7fa3", "phase": "baseline", "snapshot_marker": "S0:0x1a2b" }

GET /api/v1/migration/jobs/{job_id}/status
Response 200:
{
  "phase":            "delta_sync",   // baseline | delta_sync | verify | cutover | done
  "baseline_pct":     100,
  "files_total":      200000000,
  "files_in_cloud":   200000000,
  "files_verified":   199998500,
  "delta_lag_seconds": 4,
  "residual_bytes":   6800000000,     // ~6.8 GB left to flush
  "mismatches":       0
}

POST /api/v1/migration/jobs/{job_id}/cutover
  // gated: rejects with 409 unless lag < threshold AND mismatches == 0
Response 202: { "state": "freeze_requested", "estimated_freeze_seconds": 180 }

POST /api/v1/migration/jobs/{job_id}/rollback   // repoint apps back to on-prem
Response 202

Data store choice: SQL for the manifest, object store for data

The manifest is the heart of the control plane: 200M+ rows, queried by exact path and by status, needing transactional status updates, aggregate counts (COUNT(*) WHERE status='pending'), and set-difference audits. That is a relational / distributed-SQL workload - strong consistency on status transitions matters (two workers must not both claim the same file), and the queries are aggregations and joins, not pure KV lookups. Use PostgreSQL sharded by path-hash, or a distributed SQL store (CockroachDB/Spanner) for horizontal scale at 200M+ rows. The row size is tiny (~200 bytes), so 200M rows is ~40GB - comfortable.

Why not pure NoSQL for the manifest: we genuinely need the aggregate/audit queries (completeness and integrity are set operations) and transactional claim-a-file semantics for resumability. A KV store would push all that logic into the application. The change log itself, by contrast, is an append-only ordered stream - a Kafka/log is the right shape there (ordered, partitioned by path-hash, replayable). And the actual data lands in object storage (S3/GCS) - never in a database.

Manifest table: files

Partition/shard key: hash(path)

path            STRING   PK      -- "/data/2026/img_00417.raw"
size            BIGINT
source_mtime    TIMESTAMP        -- for last-writer-wins on deltas
source_sha256   CHAR(64)         -- computed during read
cloud_object    STRING NULL      -- "s3://.../prefix/..." or container+offset
cloud_sha256    CHAR(64) NULL    -- written back on ingest
container_id    STRING NULL      -- if packed into a small-file archive shard
status          STRING           -- pending | on_device | in_cloud | verified | failed
phase           STRING           -- baseline | delta
last_event_seq  BIGINT           -- highest change-log seq applied to this file
updated_at      TIMESTAMP

Indexes:
  PK (path)                       -- exact lookup + idempotent upsert
  INDEX (status)                  -- "what is still pending/failed" for resumability
  INDEX (phase, status)           -- per-phase progress

Change log (Kafka topic source-changes, partitioned by hash(path)):

{ seq, path, op: create|update|delete, size, source_mtime }
-- ordered per partition; delta workers commit offset = checkpoint.

Job/orchestrator state (small, strongly consistent, e.g. etcd or a jobs table):

jobs
  job_id           STRING PK
  phase            STRING          -- state machine position
  snapshot_marker  STRING          -- S0, the delta start line
  delta_checkpoint BIGINT          -- highest change-log seq flushed to cloud
  freeze_marker    BIGINT NULL     -- seq at write-freeze; cutover waits to reach it
  wan_budget_gbps  INT

Appliance batch tracking:

appliances
  device_id     STRING PK
  batch         INT
  status        STRING            -- filling | shipping | ingesting | done
  bytes_written BIGINT
  file_count    BIGINT

The manifest being SQL is what makes verification a query (SELECT path FROM files WHERE cloud_sha256 IS NULL OR cloud_sha256 != source_sha256) instead of a petabyte re-read.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. WAN bandwidth for the baseline (breaks first, and hardest). 1PB over a shared 10 Gbps link is a month and starves production. Fix: do not put the baseline on the wire. Physical appliances (~13 Snowballs) carry the petabyte; the WAN only ever carries deltas (~50-100TB) and production traffic. This single decision is the difference between a month and two weeks. Cap migration WAN usage (wan_budget_gbps) with traffic shaping so production always wins the link.

2. Source metadata / IOPS during reads. Walking and stat-ing 200M files, plus checksumming, hammers the source storage’s metadata and IO, competing with live traffic. Fix: read from a copy-on-write snapshot (isolates the baseline read from live writes), parallelize readers across the fleet, throttle read IOPS to a budget, and compute checksums inline with reads. Avoid repeated full scans entirely by using CDC for change discovery instead of re-stating.

3. 200M small objects (metadata amplification on both ends). Per-object PUT overhead, listing cost, and request rate limits dominate for tiny files. Fix: pack small files into ~1GB container/archive shards during baseline read; the manifest maps path -> (container_id, offset). Large files stream as multipart. This turns 200M objects into a few million.

4. Delta pipeline hot partitions. A few extremely hot files (a busy log, a growing DB file) generate a firehose of change events on one change-log partition and one path. Fix: coalesce per path - only the latest version transfers, and only near cutover. For huge frequently-appended files, use changed-byte-range / block-level deltas instead of whole-file resend. Partition the change log by hash(path) so hot paths spread, and give the hottest paths dedicated workers.

5. Cutover freeze window (the visible downtime). If residual delta at freeze is large, downtime is large. Fix: drive delta lag to near-zero before freezing (continuous CDC + coalescing), gate cutover on lag < few seconds AND mismatches == 0, and freeze writes only (reads stay live). For sub-minute requirements, add a short dual-write bridge. The freeze is bounded by seconds of residual data, not by total size.

6. Verification at petabyte scale. Re-reading 1PB to verify after transfer would add ~6 days. Fix: manifest-driven, end-to-end checksums computed during transfer, compared as a SQL set difference. Re-transfer only the small mismatch set over the (now free) WAN. Verification is a database query, not a re-read.

7. Resumability / partial failure. An appliance, a reader, or the delta pipeline dies mid-flight; a month-scale job will hit failures. Fix: the manifest is the ledger - every file has a status; a restarted component resumes from status IN (pending, failed). The delta pipeline is a checkpointed log consumer (Kafka offsets), so it resumes from its last committed sequence. Appliances are per-device independent; losing one means refilling one device’s worth, not restarting.

8. Single points of failure in the control plane. The orchestrator or manifest DB going down stalls everything. Fix: orchestrator runs as an HA quorum (state in etcd / distributed SQL); the manifest is a replicated, sharded SQL cluster; the change log is a replicated Kafka. Data-plane workers (readers, delta workers, verifiers) are stateless and horizontally scaled behind the manifest, so any single worker’s loss is a retry, not an outage.

9. Deletes and consistency drift (the silent-corruption trap). Files deleted on the source after S0 get left behind in the cloud, so the cloud slowly diverges from the source. Fix: capture delete events in CDC and apply them as tombstones/deletes on the object store; the verification audit explicitly checks that no cloud object exists whose source path was deleted. Last-writer-by-source_mtime on every apply prevents stale events from resurrecting old data.

Wrap-Up

The trade-offs that define this design:

• Ship the baseline, sync the deltas. The petabyte moves physically on ~13 appliances (bounded ~2 weeks, zero WAN cost); only the ~50-100TB of changes ride the 10 Gbps link. Trying to push 1PB over the wire is the classic mistake - it is a month and it is still inconsistent.
• CDC over re-scanning. Capture changes as they happen from a snapshot marker and coalesce them, so change discovery is O(changes), not O(200M files). This is what shrinks the cutover residual to single-digit GB.
• Cutover is a gated state machine, not a big-bang copy. Drive delta lag to near-zero, then freeze writes only for minutes, verify, and repoint - with on-prem kept read-only as a rollback net. Reads never go down.
• The manifest is the source of truth. SQL manifest with per-file checksums and status makes verification a set-difference query and resumability a WHERE status='pending' - not a petabyte re-read or a full restart.
• Correctness is proven end to end. Checksums computed during transfer on both sides, deletes captured explicitly, last-writer-by-source-time on every apply. “It finished” is never trusted; “the manifest audit is green” is.

One-line summary: physically ship the static 1PB baseline off a consistent snapshot while continuous change-data-capture streams the live deltas over the 10 Gbps link, reconcile both against a checksummed SQL manifest, and cut over with a minutes-long write-freeze once delta lag hits zero and verification is green - keeping the source live and the downtime near zero throughout.