What is Data Completeness, and why is it critical for compliance in financial services?

What does Data Completeness mean for the financial service industry? The wrong answer has already cost firms over $2.7 billion in fines for record-keeping failures, frequently caused by supervisory failures on platforms such as WhatsApp, iMessage and Signal.  

In this article we’ll take a look at the concepts, realities, risks and solutions of this critical compliance issue. We will also consider the kind of strategic framework needed for achieving end-to-end data integrity, and how the pursuit of data completeness is not just a defensive necessity but a proactive enabler of brand trust and long-term resilience. 

Before we get to that, though, let us define what Data Completeness means. Within the financial compliance landscape, data completeness refers to ensuring that all relevant records of client interactions, communications, and transactions are fully captured, preserved, and accessible without gaps. For financial institutions, this is critical because incomplete data creates blind spots that can conceal misconduct, undermine risk management, and expose firms to severe regulatory penalties. It is both a regulatory requirement and a cornerstone of institutional integrity, enabling firms to demonstrate accountability and maintain trust.

Known knowns  

Back in 2002, the-then US Secretary of Defense Donald Rumsfeld shared a famous observation during a press briefing: 

“As we know, there are known knowns – things we know we know. We also know there are known unknowns – that is to say, we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know…. It’s the latter category that tends to be the difficult ones.” 

The public reaction at the time thought this was funny, earning Rumsfeld a comedy award for bad English, but anyone in the field of risk assessment immediately recognized it as a “brilliant distillation” of a complex matter.  

In any field where having an exact understanding of your information resources is important, of which the global financial services industry is one of the foremost, data is more than an asset. It’s the definitive record of conduct, the basis of risk management, and the medium through which you’ll be scrutinized by the authorities.  

Consequently, the completeness of this data is not merely an operational objective, but a fundamental pillar of regulatory survival and institutional integrity. Awareness and mitigation of known unknowns is vital, and even more so given the current enforcement climate, characterized by escalating fines and zero-tolerance policies. An incomplete client interaction record represents a blind spot, and it is precisely these areas of unmonitored activity that now attract the most stringent regulatory scrutiny.    

Known unknowns  

The quality of a data set, and specifically its ‘completeness’, is the bedrock of any financial investigation.

Speaking in an interview, Phil Fry, VP & GTM of Financial Compliance Strategy at Verint, said that “it’s the things that go under the radar" that regulators are most interested in. These are the gaps that will contain misconduct, market abuse, or systemic control failures.  

The financial industry’s response to this reality has been tested and, in many cases, found wanting. A recent wave of enforcement actions saw regulators levy penalties totaling over €2.3 billion on financial firms for record-keeping failures and unmonitored communication channels, providing a clear and painful demonstration that this is not a theoretical risk.  

Professionals in regulated sectors need at least a working understanding of why data completeness has become critical for financial institutions, the relevant EU and US laws that enforce it, and the framework through which to forensically reconstruct those events on demand.  

They also need to be able to recognize the common points of technical and operational failure where the ‘unknown unknowns’ can be found, ranging from off-channel comms to broken internal data pipelines that create dangerous compliance gaps.  

Compliance in forensic detail 

Analyzing an organization’s ability to assemble complete data sets is not the academic or abstract principle it initially sounds like, but a concrete legal obligation jointly imposed by the world's most prevalent financial regulations.  

In an almost unprecedented act of harmonious international coordination, regulators in Europe and the US have designed complementary legislation granting themselves full oversight of a firm's operational records. This elevates compliance obligations beyond the simple disclosure of static reports and now necessitates the ability to ‘forensically reconstruct’ the entire lifecycle of a transaction, from initial contact to final execution. 

MiFID II and ‘reconstructability’ 

The Markets in Financial Instruments Directive II (MiFID II) is the defining feature of the European regulatory landscape, establishing ‘reconstructability’ as a core compliance standard. Its provisions move far beyond simple data storage, demanding that firms possess the capability to reproduce a complete, chronological sequence of events for any interaction that even potentially leads to a trade. 

To this purpose, the important section is MiFID II Article 16(6), which states that a regulated business must "arrange for records to be kept of all services, activities and transactions undertaken by it which shall be sufficient to enable the competent authority to fulfil its supervisory tasks."  

The courts rapidly established that the use of the word "all" was not coincidental, and signaled an expectation of completeness, leaving no room for interpretation or permissible gaps.    

This requirement is further sharpened by Article 16(7), which extends the record-keeping obligation to include telephone conversations and electronic communications. Critically, this applies to interactions "that are intended to result in the conclusion of an agreement, even if those conversations or communications do not result in an agreement."  

Just like the use of ‘all’ in the previous paragraph, this is a very specific and very important word. It establishes that even just the intent to transact, as captured in pre-trade communications, is as much a part of the official record as the transaction itself. A casual chat, an instant message query about a price, or an email gauging interest are no longer conversational – they’re mandatory records. Even if absolutely nothing untoward happened, the absence of a complete communication record is itself a violation for potentially obstructing the regulator's ability to investigate possible misconduct. 

The ultimate objective of these rules is to enable ‘trade reconstitution’ where authorities can replay the entire lifecycle of a trade from multiple corroborating data sources.  

MiFID’s predecessor is the Markets in Financial Instruments Regulation (MiFIR), section RTS 24 of which already require financial service providers to ‘retain granular records of all orders and transactions’ but was understood to be all the structured orders and receipts generated by almost any business.   

Instead, investment firms now have to include all of the unstructured and semi-formal data that provides context to doing their daily business, including emails, instant messages, voice recordings from both trading turrets and smartphones, and even the minutes from face-to-face meetings.    

To try to maintain the data integrity of these reconstructions, MiFID II imposes strict technical standards. Records must be stored in a "durable medium" that prevents alteration or deletion, a requirement often met with Write Once, Read Many (WORM) storage tech.  

Furthermore, these records must be kept for a minimum of five years, and up to seven years upon regulatory request, in a searchable format that is "readily available". This combination of completeness, immutability, and accessibility is the main purpose of the MiFID II regime.    

A third iteration of the Markets and Financial Instruments Directive, MiFID III, is expected to be introduced in the EU towards the end of this year. 1GLOBAL will shortly publish a two-part explainer on MiFID III and its expected impact on financial regulation and transaction reporting in the EU.

Prevention of market abuse 

MiFID is far from the only legislative force that makes data completeness an essential concern for financial services. It’s not even the only one in the EU, as the Market Abuse Regulation (MAR) has been using data completeness to prosecute illicit activity since 2016.  

MAR’s purpose is to protect the market by defining three core offenses: insider dealing, unlawful disclosure, and market manipulation. None of these were new offences in themselves, with insider trading having been on the EEC’s book since 1966, but these new rules required complete records as evidence for regulators to police effectively.    

A prime example of MAR's enduring reach is its stringent regulation of "market soundings", which the Regulation’s own onboard glossary defines as “A communication of information prior to the announcement of a transaction in order to gauge the interest of potential investors in a possible transaction and the conditions relating to it, such as its potential size or pricing to one or more potential investors.”  

Even the most amateur legal scholar will be able to spot that the number of times the words ‘potential’ and ‘possible’ get used means that subject firms will have to cover a lot of bases.  

It’s not only quantity that gets mandated but quality too, under the Quality of Execution requirement in Article 6 of the MAR. This details a great deal of meticulous record-keeping for all related comms, from voice captures to written correspondence to detailed, signed minutes with appendices of dates, times, attendees, and the specific information exchanged.    

By now, the purpose of all these records isn’t just to establish what was said to whom and when, but to be able to contextualize an entire firm’s movements when investigating suspicious trading activity to pinpoint a potential leak. If a firm's share price moves unexpectedly ahead of a major announcement, regulators will demand the full ‘market sounding’ records so they can virtually simulate how an organization was acting in minute details up until that event.  

Financial firms should also be aware that the ‘presumption of innocence’ enshrined in almost every legal code since the sixth-century Digest of Justinian officially does not apply to MAR or MiFID, and incomplete or missing communication data will be considered an offence in itself and make it impossible for a firm to defend itself while under investigation.   

American immutability 

Meanwhile, in the US, the Securities and Exchange Commission (SEC) has a parallel regime built on the principle of accurate and complete data. The core rules, 17a-3 & 17a-4, obligate firms to create and preserve equally comprehensive data sets. The flavor of the language is a little more old-school, mentioning “trade blotters, ledgers, and order tickets” but did an impressive job of futureproofing itself by also covering “all communications received, and copies of all communications sent... relating to their business as such".    

Leaving nothing to chance, in 2022 the SEC modernized these requirements with significant amendments and obligated firms to two specific methods of preserving electronic data. The first is the aforementioned WORM format, but also a second slightly more flexible option is to use an electronic recordkeeping system (ERS) that maintains completeness via a system of time-stamps.    

A critical component of the newer SEC ruleset is a "prompt production" mandate. Businesses must have the capacity to readily download and transfer records for regulators in a "reasonably usable electronic format".  

While that sounds like a fairly normal and plain language requirement, it naturally is not. In court actions this has been used to pass judgements of non-compliance for keeping data in siloed, legacy systems with incompatible antique formats. All data must be indexed, searchable, and immediately accessible.  

All of this does not mean that a regulated business is done with old-fashioned hardcopy, either. While investigating Merrill Lynch, the SEC ruled that since the firm used a system that didn’t prevent alteration or deletion of data, and they didn’t keep physical copies in a secure separate location, they were liable for a $1.5 million penalty even without malfeasance. The fine may have appeared nominal as far as Merrill Lynch was concerned, but the court-order to immediately remedy its company-wide recording practices, including upgrading to WORM compliance and data completeness, cost a great deal more.  

Financial service providers in the US and Europe face data completeness requirements that are both dauntingly monolithic, but also surprisingly harmonious in their multijurisdictional requirements.   

MiFID II demands the ability to reconstruct data incrementally. The SEC requires data be unalterable and immediately available. MAR insists that even the smallest data points in notes and discussions be part of completeness.   

Where they unite is the principle that data completeness is the non-negotiable price of doing business. What once were murky basement silos of dumped metadata are no longer secondary information, but primary compliance data points. Incomplete data can break a trade’s validity just as surely as a missing client signature, elevating data enrichment and governance to a core compliance function.    

Common causes of data incompleteness 

While the regulatory obligation for data completeness is abundantly clear, how to actually achieve it in practice is less obvious. There’s a pronounced trend in compliance regulation to increasingly leave the methodology up to the individual firm, as long as the results are clear.  

On the one hand, this can be interpreted as positive non-intervention to allow businesses to run themselves however they see fit. On the other, it can be seen as tacit admission that authorities can’t keep up with the pace of technological innovation such as AI and would rather keep their rules open-ended and interpret them on a case-by-case basis.  

Compliance failures rarely stem from a single, isolated incident. Instead, they’re typically the result of systemic weaknesses in tech, processes, or governance. These weaknesses create data gaps that can accumulate for years, only to be exposed during a regulatory audit or investigation.  

This can have devastating consequences, because depending on how the authority is inclined, fines can just as easily be individual as cumulative. Individually, a lot of regulatory fines can be quite minor. The German Federal Criminal Police Office (Bundeskriminalamt) starts off its warning fines at a modest €50 per infraction. However, if the authority then uncovers a record of half a million finable trades, things rapidly get expensive.  

Understanding the causes of these failures is the first step toward preventing them. 

• Off-channel blind spots 

  The costliest source of data incompleteness in recent years, and one explicitly targeted by authorities, has been the proliferation of off-channel comms. This is the widespread use of unmonitored messaging platforms like WhatsApp, iMessage, and Signal for substantive business discussions.  

  These conversations can occur entirely outside the firm's controlled, recorded, and archived environment, creating a massive compliance blind spot that regulators have been swift to punish. According to Deloitte, U.S. regulators have levied over $2.7 billion in fines specifically for these off-channel data violations, starting with a $200 million penalty against JPMorgan in 2021 to set the tone.  

  By 2023, the total fines had reportedly ballooned to over $1.8 billion across 16 major financial firms, which should be concrete proof for any firm that record gaps and data incompleteness is not a theoretical risk and no organization is too big, or staff too sophisticated, to be vulnerable.  

Leaky Data  

Within the field of Data Integrity, the inability to collect information isn’t the only threat to data completeness. There’s also a problem of keeping it once you’ve got it.  

Data pipelines are any parts of an organizations digital ecosystem that transport information from source (e.g a CRM system) to its destination, such as transcription tool or reporting dashboard. A breakage at any point in this flow can lead to incomplete, inaccurate, or delayed data, with severe compliance implications.    

There are many technical reasons why network architecture can fail in this way. A commonly cited issue is what network analysts call an ‘unplanned schema change’, where file name conventions or formats are updated at source without warning the recipient systems. Other various technical causes include incompatible data, hardware failures, patchy network issues and simple software bugs.  

The 2024 global outage caused by a single faulty update from CrowdStrike cost Fortune 500 companies an estimated $5.4 billion. 

As spectacular and immediate as system failures can be, industry research strongly suggest the majority of data pipeline issues are non-technical. Far more likely are organizational errors such as lack of clear ownership, where no single person or team is accountable for the integrity of a data pipeline from end to end. As soon as the data is off one person’s desk, it’s no longer their problem.   

This extends to a disconnect between business and IT teams, and poorly defined responsibilities.  In 2024 JPMorgan ended up with a $350 million fine even though regulators had “not identified any employee misconduct, harm to clients or the market" but simply that data coming out of one team was getting dumped in storage rather than being fed into the surveillance team’s platform, and it wasn’t either teams job to get the data from A to B.   

The compliance impact of these pipeline failures can be just as severe as off-channel comms, and harder to fix as they’re not a code bug but a broken organizational process.    

Data Integrity via Unified Connectivity  

Addressing the causes of data incompleteness requires a senior, strategy-level response more profound than software fixes and reactive measures. A robust, proactive data compliance posture can only be built upon a holistic framework that combines both policy and the full end-to-end technological competence of a business’s connectivity. 

A compliant organization has to make data completeness the default state of its digital ecosphere, not something to periodically restore as part of IT housekeeping. Rather than an exhaustive list of individual requirements, it’s more illustrative and insightful to consider the best-practice of current leaders in compliance and data completeness, such as 1GLOBAL’s holistic solution.   

Achieving data integrity requires unified connectivity architecture. 1GLOBAL’s compliance solutions are designed to address the core challenges of data incompleteness, specifically in the financial services sector.  

In-Network capture 

Even if app-based mobile recording solutions were not notoriously unreliable, easily disrupted by OS updates or user interference, that they have been proven to be unpopular with regulatory authorities should be reason enough for risk-averse organizations.  

A lasting and effective approach, exemplified by 1GLOBAL's solution, is in-network recording. This captures data flow at the carrier or SIM level, directly within the network infrastructure, before the data ever reaches the end-user's device. This architectural difference has profound implications for compliance.    

It ensures the SEC Rule 17a-4 of highly prized immutability. As capture is automatic on the server-side, and independent of user prompting, it is a fail-resistant and tamper-proof system. This is the principle of "Zero Touch" architecture in action that directly addresses the unreliability of app-based methods and aligns with the immutability principles.  

The responsibility for recording is shifted away from the end-user and their device to a professionally managed, carrier-grade service, giving the organization the central oversight it requires to maintain data quality.  

Message+ 

The true driving force behind all the off-channel communications that have cost the financial service industry so many billions is the simple wish for convenience by employees. 1GLOBAL's Message+ service is a gold-standard example of achieving data completeness by giving staff the path of least resistance and most convenience. It’s a solution embedded directly into the ubiquitous Microsoft Teams platform that allows employees to send and receive SMS and WhatsApp messages using their official corporate number, all within a controlled and fully data-captured environment.  This solution also neatly addresses the BYOD challenge by securely separating personal and professional communications on a single handset.    

The Core Network 

For a global financial institution, even the most sophisticated data-capture tools are only as good as the network they operate in. As connectivity becomes unreliable, slow, or fragmented across regional providers, this ‘pipeline’ also accumulates potential points of failure.    

A data-complete compliance strategy requires a complete connectivity strategy. 1GLOBAL addresses this by operating a single, globally distributed core network. This network has geographically distributed points of presence (POPs) that route traffic intelligently and locally, rather than backhauling it to a home country, and it leverages partnerships with over 600 local networks in more than 190 countries.    

This uniquely distributed architecture offers failover resilience against outages and lower latency for data transfer, which is critical to ensure that compliance capture tools are always online. The operational dependability extends to management too, as a financial service provider can manage its entire device fleet through a single contract, a single bill, and a unified management platform. This dramatically increases operational streamlining, shrinks overheads, and minimizes the risk of data getting lost or siloed by organizational complexity.     

Next steps 

Even if market analysis hadn’t unanimously established that data completeness is a non-negotiable, existential requirement for survival in the financial services industry, then the scale of the fines alone would be a good enough motivator.  

The regulatory mandate for total visibility and forensic reconstructability, enforced by MiFID II, MAR, and the SEC is absolute. 

The causes of these failures can be complex – a mix of technological lag, legacy systems, operational dysfunction, and ambiguous responsibilities – but the significant effort required to achieve this end-to-end data integrity should be considered in context.  

Although driven by the stick of fines and reputational damage, the carrot of radically improved operational efficiency makes it worthwhile. Providing risk managers and stakeholders with complete, high-quality data improves the accuracy of their insights and the quality of their decisions. A firm’s decisions are only as good as it’s available intelligence, and a data-complete record is an invaluable strategic resource.  

Find out how 1GLOBAL can help your organization to comprehensively meet its data completeness and compliance obligations - contact our team today.