Masking Sensitive IDs Across 600 Million Documents

By Sagar Thakkar

The task sounds deceptively simple: mask the first 8 digits of an identity number on a document, leaving only the last 4 visible, the “masked ID” format many regulators require (for example, the UIDAI masked-Aadhaar standard in India).

The model that detects and masks is already built. That is not the problem.

The problem is the number: 600 million documents, sitting in a Document Management System, reachable over FTP. At that scale the engineering question is not “how do we mask one document?” It is “how do we mask 600 million of them, within a sane timeline, at a defensible cost, without ever losing or double-processing a single file, and without leaking the most sensitive PII you hold?”

This post is about that architecture.

Reframing the problem

This is not a real-time, request/response system. It is a massive offline batch job, closer to a data-migration program than an API.

The flow per document is fixed and simple:

Download from FTP → pass through the masking model → write the masked file back to FTP.

Per document, that’s a few seconds of work. Multiply by 600 million and three concerns dominate everything else:

1. Processing time, how long to clear the backlog?
2. Cost, what does 600M document-operations actually cost?
3. Resilience, a job this long will be interrupted; it must resume without redoing work or skipping files.

Plus a fourth, non-negotiable for regulated PII: compliance and data security.

The architecture

The components

1. Catalog / Manifest Builder

Before processing anything, enumerate every document in the DMS into a Manifest DB, one row per doc_id with a status (pending → in_progress → done / failed). This manifest is the single source of truth for the entire run. Without it, you cannot answer the two questions that matter at this scale: “how far are we?” and “what still needs doing?“

2. Batch Orchestrator

Reads the manifest, shards the work into batches, feeds the queue, tracks progress, and controls how aggressively to scale. It’s also what lets the job resume, on restart it simply picks up everything still marked pending.

3. Work Queue

Decouples “what needs doing” from “who’s doing it.” Batches of document IDs flow in; workers pull them at their own pace. Scaling workers up or down never touches the orchestrator.

4. Masking Worker Pool (the workhorse)

Each worker is ephemeral and stateless and does three things per document:

1. Downloads the file from FTP.
2. Runs the model, detect the ID number, mask the first 8 digits.
3. Uploads the masked file back to FTP and marks the document done in the manifest.

Critical properties:

• No local persistence, the document is processed in memory/temp and wiped immediately. You never accumulate 600M sensitive files on worker disks.
• Embarrassingly parallel, throughput scales linearly with worker count.
• Runs on spot / preemptible instances, this is a batch job with no latency SLA, so use the cheapest interruptible compute available. Interruptions are safe because the manifest makes everything resumable.

5. Review Queue / Dead Letter Queue

Not every document is clean. Some are corrupt, some have no detectable ID, some come back low-confidence. These route to a human QA queue instead of being silently passed or failed. This path is where the real cost and time risk lives, more on that below.

6. Audit & Compliance Log

Every masking operation is logged immutably: which document, when, by which model version, what the outcome was. This is regulated PII, you must be able to prove what happened to every file, and all of it must stay within the required data-residency region.

7. Progress Dashboard

Live throughput, ETA, cost burn, and review-queue size. At 600M scale you do not “check if it’s done”, you watch a curve.

Processing time: the math

State the assumptions, then the method (numbers adjust to real benchmarks):

Assumptions

• Per-document model time: ~2 seconds (download + detect + mask + upload), per worker thread.
• Therefore 1 thread = 0.5 docs/sec = 43,200 docs/day.

Time to clear 600 million = a function of parallelism:

Target completion	Parallel threads needed
30 days	≈ 463 threads
60 days	≈ 231 threads
90 days	≈ 154 threads

threads = 600,000,000 ÷ (days × 43,200)

463 threads is modest, roughly 60 containers at 8 threads each. Compute parallelism is not the constraint. You pick a completion target, and the worker count follows.

Cost: where the money actually goes

Assumptions: avg doc ~200 KB; spot CPU ~$0.03/vCPU-hour; model ~2 sec/doc.

1. Compute (the part everyone over-estimates):

• Total work = 600M × 2 sec = 1.2 billion seconds ≈ 333,000 vCPU-hours.
• On spot at ~$0.03/hr → ≈ $10,000 for the entire backfill.
• Compute is cheap. If the model needs GPU/OCR acceleration, recompute with GPU spot rates, but the order of magnitude stays small relative to the data scale.

2. Data transfer (the silent killer):

• 600M × 200 KB ≈ 120 TB down, 120 TB back up.
• If processing runs in a different network/region from the DMS, egress alone can dwarf compute (~$0.02/GB → thousands of dollars, plus FTP throughput becomes the bottleneck).
• The lever: run the workers in the same region/VPC as the DMS. Keep the data movement free and local.

3. Human review (the real cost driver):

• This dominates everything. If the model auto-handles 98% of documents, the remaining 2% = 12 million documents need human eyes.
• At even 0.5% miss rate, that’s still 3 million manual reviews.
• The single biggest cost and timeline variable is not compute, it’s the model’s accuracy tail. Every 0.1% improvement in auto-handle rate removes ~600,000 manual reviews.

The senior insight: at 600M scale, compute is a rounding error (~$10k). The program’s cost and schedule are decided by data locality and the model’s miss rate, not by servers.

Resilience: a 600M-document job will break

• Worker dies / spot reclaimed? The document was never marked done, so it stays pending and another worker picks it up. Idempotent by design.
• Whole job stops? Restart the orchestrator; it resumes from the manifest. Nothing is reprocessed (which would waste money) and nothing is skipped (which would breach compliance).
• Bad documents? Routed to the review queue, never blocking the line.

Compliance: this is the point of the system

Masking regulated IDs is itself a privacy-compliance act, so the architecture must be airtight:

• Data localization, all processing and storage inside the required region.
• No data at rest on workers, process in memory, wipe immediately.
• Encryption in transit and at rest, strict least-privilege access.
• Immutable audit trail, every document’s masking event, with model version and timestamp, defensible to a regulator.
• Verification sampling, a sampled QA pass to confirm masking quality at scale.

Takeaway

The model was the easy part. Masking 600 million documents is a batch-engineering and cost problem, and it comes down to five moves:

1. Build a manifest so every document is tracked and the job is resumable.
2. Make workers ephemeral, parallel, and spot-based, pick a completion date, derive the worker count.
3. Co-locate compute with the DMS so data transfer stays free and fast.
4. Treat the model’s miss rate as the primary budget line, the human-review tail, not compute, decides cost and schedule.
5. Bake in compliance, in-region, no data at rest, encrypted, fully audited.

Get those right and 600 million documents become a controllable, predictable program instead of an open-ended risk.

---

Written by Sagar Thakkar, solution architect specialising in large-scale data processing, cost-optimised cloud-native systems, and privacy-compliant architecture. More at sagarthakkar.com.