Schema Design for Public Procurement: The Ontology Crisis

The Ontology Crisis in Your Ingestion Pipeline

You don’t have a scraping problem; you have an ontology crisis. The scraper runs fine. The HTML parses correctly. The data lands in your staging bucket. Then the analytics pipeline crashes because a Gulf Cooperation Council portal omitted supplier identifiers, or an European Union tender lacks evaluation scoring metadata. Building a reliable ingestion system requires accepting a harsh reality about global government spending.

"Governments around the world spend an estimated US$13 trillion through contracts every year."

· source: https://standard.open-contracting.org/latest/en/ That massive financial volume flows through hundreds of disjointed digital portals. Most engineering teams approach this by adopting the Open Contracting Data Standard (OCDS) as a universal translator. The assumption is that strict adherence to this standard solves the normalization problem. That assumption is wrong. Real-world portals implement the standard with critical gaps. When your pipeline expects a universal truth and receives a fragmented reality, the system breaks. The conflict between the desire for a single universal standard and the reality of wildly divergent local legal requirements makes strict adherence impossible.

The Fragmentation Illusion and the Rigidity Trap

The fragmentation illusion assumes that because a portal claims OCDS compliance, the data is uniform. A recent comparative study examining six distinct environments proves otherwise. The research covers the European Union (TED), the UK (Contracts Finder), Singapore (GeBIZ), and three GCC systems: Saudi Arabia's Etimad, Qatar's Monaqasat, and the UAE federal eSourcing portals. GeBIZ demonstrates near-complete lifecycle traceability. The EU and UK systems show moderate gaps. GCC platforms provide limited supplier-level and performance information. Major gaps remain across these environments in supplier identifiers, evaluation scoring, SME or local vendor flags, past performance metrics, and contract implementation data. | Jurisdiction/System | Supplier Identifier Availability | Evaluation Scoring Data | | :--- | :--- | :--- | | EU (TED) | Moderate | Partial | | UK (Contracts Finder) | Moderate | Missing | | Singapore (GeBIZ) | High | High | | GCC (Etimad/Monaqasat/eSourcing) | Low | Missing | Forcing heterogeneous tender data into a rigid core schema causes severe data loss. Consider utility-scale energy storage procurements. These involve complex contracting forms including PPAs, EPCs, BTAs, and battery supply agreements. A rigid schema designed for basic office supplies cannot capture the financeability terms and tariff structures of a power purchase agreement without breaking. When engineers force these complex instruments into a flat, standardized table, the pipeline either drops the nuanced fields entirely or creates a fragile mapping that shatters the moment a local authority adds a new compliance requirement.

The Core-Extension Pattern for Tender Data Normalization

The solution requires shifting from a mindset of mapping everything to the core, to enforcing a strict core with flexible extensions. This is where true tender data normalization actually happens. Most guides assume standard compliance solves the problem. Our analysis reveals that true analytics readiness requires a 'Core + Extension' schema model that explicitly accounts for the supplier ID and evaluation scoring gaps found in 4 out of 6 major global systems. This approach turns schema design from a mere mapping exercise into a comprehensive data preservation strategy. The core schema enforces the absolute minimum required for cross-jurisdictional comparison. It contains the universal truths: tender ID, publication date, basic value, and currency. Everything else · the local legal quirks, the specific evaluation weights, the regional supplier classifications · flows into a structured `local_extensions` object. This pattern mirrors the principles we apply when designing 06 The laws that govern data decisions at Halantir. You define the immutable rules first, then allow localized variations to exist without corrupting the foundational logic. By isolating the heterogeneous data, your public procurement analytics pipelines can aggregate the core fields safely while retaining the ability to drill down into jurisdiction-specific nuances when necessary.

The Normalization Cost and the Verification Loop

Handling missing mandatory fields and inconsistent value units across jurisdictions carries a real engineering cost. When a portal omits a core field, the pipeline must decide whether to halt ingestion, inject a null value, or estimate the missing data. Halting ingestion creates blind spots. Injecting nulls breaks downstream aggregations. Estimating introduces statistical noise. The procurement data schema must include a verification loop. Schema validation is not just for initial ingestion; it is for the ongoing auditability of source changes. Portals update their APIs without warning. A field that was mandatory last month becomes optional today.

AI doesn't remove the need for structured data, it simply makes structuring data faster and cheaper.

· source: AI in public procurement: between experimentation and impact Machine learning accelerates the extraction of unstructured PDFs into JSON, but it does not replace the need for a rigid validation layer. The verification loop compares incoming payloads against the expected schema distribution. When a portal changes its data shape, the loop catches the anomaly before it poisons the analytics database. This boundary enforcement is similar to the challenges explored in How to Engineer Federated Infrastructure for Data Sovereignty, where data boundaries must be strictly policed to maintain integrity across disparate legal jurisdictions. ```json { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "required": ["tenderId", "publicationDate", "value"], "properties": { "tenderId": { "type": "string" }, "publicationDate": { "type": "string", "format": "date" }, "value": { "type": "object", "required": ["amount", "currency"], "properties": { "amount": { "type": "number" }, "currency": { "type": "string" } } }, "local_extensions": { "type": "object", "description": "Jurisdiction-specific fields that do not conform to the core OCDS schema.", "additionalProperties": true } } } ```

Tools for the Core-Extension Architecture

Building this architecture requires tools that enforce strict typing while allowing dynamic expansion. * **JSON Schema:** The foundational layer for defining the strict core and the permissive extension boundaries. It validates the structure before the data ever touches the database. * **Open Contracting Data Standard (OCDS):** Used strictly as the baseline dictionary for the core fields. It provides the semantic definitions for the universal truths. * **Python (pandas/pydantic):** Pydantic enforces the data models at the application layer, ensuring that the `local_extensions` object is parsed correctly without throwing type errors. Pandas handles the downstream aggregation of the core fields. * **Great Expectations:** Essential for the verification loop. It profiles the incoming data distributions and triggers alerts when the statistical shape of the incoming feeds deviates from the historical baseline. These tools operate neutrally. They do not guess the meaning of the data; they enforce the structural rules you define.

How We Hit It: Build-Log and Numbers

Our platform operates with 4.3M tender records across 100+ sources. Reaching this scale required abandoning the idea of perfect upstream compliance. We initially map every local field directly into the core schema. The ingestion pipeline breaks every time a portal changes a single dropdown menu. We reverse course and build the extension layer instead. That scar tissue defines our current architecture. Handling rate limits and fallback mechanisms becomes a daily reality. Companies House API requests are rate-capped at 600 requests per 5 minutes. When primary feeds fail, an FTS API fallback was implemented to backfill 350–440 releases a day for Defence Online. To monitor the health of these pipelines, a distribution-anomaly alert pages on-call when daily release counts drop more than 2σ from the 30-day mean. This ensures that a silent API failure triggers an immediate engineering response. This rigorous approach to data integrity extends to our publishing operations. This site has published 32 articles in the last 90 days. Median time from publish to confirmed Google indexing on this site: 4 days. The verification loop we built for procurement data directly informs how we approach DBM COMPASS: A Developer’s Guide to Forensic Budget Auditing. Auditing requires the same strict separation of universal rules and localized exceptions. You can read more about our structural philosophy on the company page.

Next Steps and Open Questions

At what point does schema flexibility become so permissive that cross-jurisdictional comparison becomes statistically meaningless? If the `local_extensions` object absorbs too much analytical weight, the core schema becomes a hollow shell. This is the open question your team must answer as you scale. To test these concepts in your own environment, execute the following numbered playbook: 1. **Pick two disparate tenders:** Select one infrastructure project and one IT service contract from two different jurisdictions. 2. **Map to core only:** Attempt to force all data points from both tenders into the strict OCDS core schema. Document exactly which data points are lost or distorted in the process. 3. **Build the validator:** Create a simple JSON Schema validator that enforces the core OCDS fields but allows a `local_extensions` object. 4. **Test against raw feeds:** Run this validator against a raw, unmodified feed from a non-compliant portal. Measure the ingestion success rate compared to your previous strict-mapping approach. The goal is not to achieve perfect compliance with a standard that was never designed for the messy reality of global government spending. The goal is to preserve the data exactly as it exists, while maintaining just enough structural rigidity to make public procurement analytics possible.

HEIMLANDR -- Builders of the official layer of the Nordics.

1. Define the Immutable Core: Identify the 10-15 fields (ID, date, value, buyer) that must exist for any record to be analyzable.
2. Map Jurisdictional Gaps: Use comparative frameworks to identify missing mandatory fields (e.g., supplier IDs in GCC systems) and flag them as nullable or extended.
3. Implement Flexible Extensions: Design a JSON Schema that allows jurisdiction-specific objects (e.g., 'energy_storage_terms') without polluting the core namespace.
4. Normalize Units and Currencies: Build a pre-processing step that converts all monetary values to a base currency and standardizes units before ingestion.
5. Validate Against Reality: Run raw samples through your schema validator to ensure extensions capture enough detail to answer specific analyst queries.