Progressive enrichment of entities at national scale

The analytical value of a public record lies not in its volume, but in the dimensions it leaves implicit. A raw registry dump answers “what exists,” but it does not answer “where,” “linked to whom,” “since when,” or “in what state.” These four questions — spatial, relational, temporal, and of state — are precisely the ones that underpin any non-trivial analysis, and none of them comes explicit in the source.

Beyond the absence of dimensions, authoritative data carries structural debt inherited from decades of legacy systems: locality names in all-caps without diacritics, phone prefixes in historical formats, boolean indicators encoded as characters, numeric codes that require left-padding for correct comparison, and composite identity keys that must be reconciled into a coherent notion of an entity and its establishments.

To enrich, here, has a precise meaning: to systematically add each missing dimension and clean up each structural debt, while fully preserving fidelity and traceability to the original source. It is not interpretation, it is not speculative inference — it is disciplined reconstruction.

Five principles govern every decision in the methodology. They are what separates an enrichment pipeline from a simple data-transformation process.

The flow is linear and unidirectional: the authoritative source enters through a versioned ingestion stage, is reconciled into a canonical model, passes through independent enrichment layers, and is materialized into a query surface. The cadence is idempotent and frequent — the base checks the origin several times a day and absorbs a new reference period in hours, not in manual cycles.

The core of the methodology. Each layer adds exactly one class of dimension and takes as input the output of the previous one. The order is not arbitrary: you normalize before resolving semantics; you resolve semantics before geocoding; you geocode before composing graphs and time series.

The decisive property is orthogonality: because each layer writes to its own space and reads only the contract of the previous one, the base can reprocess geospatial without touching relational, or evolve the semantic taxonomy without rebuilding coordinates. That is what makes it possible to run continuous enrichment over tens of millions of records without global maintenance windows.

Enriching tens of millions of records continuously, while the base serves live queries, is a systems-engineering problem before it is a data problem. Five disciplines sustain it.

Layers that rebuild an entire table do so in a single transaction — the replacement becomes visible to live queries only at the instant of commit, eliminating any window in which the base appears empty or partial. A failure reverts to the prior state; there is never a published intermediate state.

Long enrichments are partitioned into contiguous, disjoint ranges, each with its own resume point. A process crash, machine restart, or interruption of an external dependency does not cost the work already done — the next run continues from the frontier, not from the beginning.

Statement time limits, explicit working-memory sizing, and caching with anti-stampede protection keep tail cost predictable even under concurrency — an expensive query never degrades the service for the rest.

Every schema change is a versioned migration, immutable once applied, tracked by checksum, and executed transactionally by a deterministic runner. The structure of the base is reconstructible and auditable like any other software artifact.

The current reference period is an inspectable state; pipeline failures propagate an exit code and trigger an explicit alert. The graceful-degradation principle (P5) has an operational counterpart: silence is never mistaken for success.

Confidence in an enriched base is an assertion that must be demonstrable, not assumed. The methodology treats verification as part of the product.

Every query is parameterized — a real separation between code and data, with no interpolation — and every output is escaped at the rendering boundary. The highest-risk dynamic surface (combinatorial composition of filters) is verified by a test that asserts every user value travels through binding, never inline.

Pure functions covered by unit tests; behavior against a real database covered by integration tests against a seeded schema; structural invariants protected by automated guards. Each fix is born accompanied by the regression that locks it in.

Maturity is measured by a reproducible procedure: independent reviewers evaluate distinct dimensions (security, correctness, tests, operation, quality, documentation); each finding goes through adversarial verification — a second reviewer tries to refute it — before it counts; a weighted synthesis produces a defensible index. It is the same scientific discipline of peer review, applied to engineering.

The discipline is not in the volume of data you process, but in the methodology by which each dimension is added, verified, and made traceable.

The independent layers exist to be composed. The final value is not each dimension in isolation, but the scenarios that their combination makes queryable.

Multidimensional filters — location, activity, size, time range, tax regime, contact attributes — compose in a parameterized way and are consistent between count and listing: the total displayed and the rows returned derive from the same predicate, eliminating the classic divergence between “how many exist” and “which appear.”

On top of the coordinate layer (L3), proximity and radius scenarios become first-class queries: sector density by region, activity concentration, territorial coverage — questions that the original flat record is incapable of answering.

The orthogonality of the layers is what makes it possible to cross dimensions: cohorts by age and regime (L5) sliced by region (L3) and by sector (L2), or control networks (L4) filtered by size and location. Each scenario is an intersection of layers that were enriched independently and reproducibly.

Although the reference implementation operates over a registry of legal entities, nothing in the methodology is specific to that domain. The layer taxonomy — normalize → resolve semantics → geocode → compose graph → reconstruct time → derive — applies to any authoritative record at scale: property registries, vehicle fleets, health records, person registries, environmental licensing.

What carries across domains is not the code, but the principles and the reliability discipline: primacy of the source, idempotence, provenance, orthogonal layers, graceful degradation, atomicity, resume, adversarial verification. Any public record, treated under this methodology, ceases to be a file and becomes a living analytical base — reproducible, traceable, and verified.

That is the central thesis: enrichment is not a transformation step, it is an architecture. Done as architecture — layered, non-destructive, versioned, and adversarially verified — it scales in data and in trust at the same time.