Metadata as a Data Model, Part III

The Metadata Application Profile

Part I in this series, Part II in this series

A Metadata Application Profile Template is available HERE for paid subscribers

To pick up where we left off, In my previous articles exploring metadata as a data model, we uncovered fundamental disconnects in how organizations manage meaning across data ecosystems. In part I of Metadata as a Data Model, I noted that ownership and management of metadata is often treated as a hot potato—does the onus of metadata management reside with data engineering, business intelligence (BI), data analytics? Or is metadata every contributor and team’s responsibility? Where did enterprise systems go wrong with metadata?

Metadata Hot Takes: MDM and the Semantic Layer

If an organization takes data and records metadata management seriously, there will be a cross-functional team or governance functions tasked with building and maintaining master data management (MDM) systems, to harmonize data, thereby unifying data and metadata in the process. A core function of MDM is to create golden records and canonical definitions, to ultimately create a metadata “single source of truth”. In a similar vein, data engineering, data analytics and business users implement semantic layers, to bridge the gap between technical data ecosystems and business users, to translate complex data representations into common vocabularies and terminologies for end users. MDM and semantic layers can and do work together although generally, each is unique in its processes, and technical components.

While organizations struggle to imbue data systems with rich context and meaning, MDM and semantic layers have fairly drawn the attention of practitioners and vendors alike. It is no secret that AI demands rich semantics, not syntactic data representations, to understand the context and meaning of data within domains and businesses. MDM and semantic layers promise to deliver the all holy rich context, mainly through text literal labels and metadata. However, true structured and semi-structured semantic data representations are not usually threaded through the underlying code, to drive contextual understanding efforts, for the benefit of AI.

Even though MDM and semantic layers can be effective data architecture and data management components, both systems tend to fall short in truly managing metadata holistically. To manage metadata as a data model, machine readable and interoperable code must accompany metadata models to support a metadata program that has teeth. Beyond metadata encoding structures, a metadata model supports AI’s contextual understanding while also sustaining crosswalks, transformers and translation layers, to negotiate organizational metadata alignment. Not one model to rule them all, but machine readable, interoperable manifestations of metadata and its related contexts.

To address the risks and realities of metadata management, library science has devised a methodology known as the metadata application profile or MAP. In no way a replacement for MDM or semantic layers, the MAP works to provide a roadmap to an organization’s metadata, and methodologies for metadata synchronization and reconciliation. A well crafted MAP supports teams, organizational data systems and AI, all while structuring metadata according to definitions and a metadata reconciliation layer. Consider the MAP living and breathing documentation, able to serve up rich context and meaning, so sorely lacking in most data and business environments.

We Have a Semantic Crisis

Organizations struggle with metadata and semantic consistency. According to a 2021 Gartner report, 75% of MDM programs fail to meet business objectives. The financial impact reaches astronomical levels, when we account for poor data quality, which costs organizations an average of USD 12.9 million to $15 million each year. At the macroeconomic level, IBM discovered that in the US alone, businesses lose $3.1 trillion annually, due to poor data quality.

When we dig into the root causes, metadata and semantic inconsistency emerge as primary culprits. According to O'Reilly research, more than 60% of respondents to a survey selected "Too many data sources and inconsistent data" as their top challenge, followed by "Disorganized data stores and lack of metadata", which was selected by just under 50% of respondents. The operational impact? Data teams spend 30–40% of their time handling data quality issues instead of working on revenue-generating activities.

Organizations have invested millions in an MDM platforms that promise to be the "single source of truth" for data. Meanwhile, the BI team has built elaborate semantic models in tools like dbt or Looker that define what a "customer" means for analytics and business user purposes. These two systems speak different languages, use different standards, and often do not interoperate.

The result? Data engineers spend countless hours writing translation code. Business analysts create spreadsheets to manually map between systems. And executives wonder why their "single source of truth" feels more like multiple sources of confusion.

Enter the Metadata Application Profile

A metadata application profile (MAP) is, at its core, a declaration of which metadata terms an organization uses, how it uses them, and what constraints apply in specific contexts. As Dublin Core elegantly defines it, an application profile is "an assemblage of metadata elements selected from one or more metadata schemas and combined in a compound schema."