Why using an off-the-shelf generative AI tool to validate your regulated documents could cost you far more than a software licence.
By Hubert Stoop – Chief Innovation Officer, MyData-TRUST
The arrival of AI plugins in legal tools is sending a legitimate shockwave through the industry. After several days of intensive testing of Anthropic’s Claude plugin, applied to standard contract review workflows, but also to highly regulated documents such as Clinical Trial documents, the verdict is clear: the tool is powerful, but power without proper governance is a risk.
This article is for those asking the essential question: what could go wrong when AI works without domain experts on high-stakes documents?
⚖️ What AI Does Well and What It Does Not
Claude’s general legal responses are impressive. For a standard contract review workflow, verifying boilerplate clauses, identifying deviations, summarising positions, the tool is effective and delivers significant time savings.
But as soon as you enter the domain of regulated documents, let’s use Clinical Trials and Data Protection regulations as examples, raw capability is no longer enough. To be fair, today’s models are not technically limited in their access to information: tools like web search, integrated into platforms such as Claude Desktop, Google Gemini or ChatGPT, allow them to retrieve Data Protection regulations, national health authority templates, and privacy guidance, whether GDPR, CCPA, LGPD, PIPL or others in real time. Some are even capable of identifying the legal qualification of the roles of the various stakeholders in a Clinical Trial within the specific jurisdictions concerned by the study. For example, determining who qualifies as controller, joint controller, processor, sponsor, investigator, or site under the applicable national and supranational frameworks.
The problem is that they do not do so spontaneously. A generic model does not know what to look for, when to look for it, or how to structure a compliance assessment unless it is explicitly guided to do so. Using the same example, it will not instinctively cross-reference an Informed Consent Form (ICF) against the latest national health authority guidance, relevant case law, or a mandatory template issued by the competent authority or ethics committee. The same applies to Clinical Trial Agreements (CTAs): without specific direction, the model will not identify which Data Protection requirements to check or how to assess them. The information is accessible, but the judgement to seek it out, and the expertise to interpret it correctly, must come from somewhere.
This is precisely why real efficiency and real reliability require customised “skills”, bespoke configurations that embed domain-specific rules, regulatory checklists, and compliance logic, combined with the human expertise needed to direct the tool towards the right questions in the first place.
👀 The Question Nobody Is Asking: Who Bears the Liability?
This is where the issue becomes critical. Using the same example, when a generative AI tool is used to validate a CTA or an ICF, and a non-compliance is discovered after the fact, the liability does not fall on the tool’s vendor. It falls on you.
The terms and conditions of these tools are unambiguous: no guarantee of results, no contractual liability for the accuracy of outputs, no coverage for damages arising from decisions made on the basis of a generated response. In plain terms: the vendor provides the engine, not the pilot.
In the regulatory context of clinical trials, this reality has very tangible consequences:
- A non-compliant ICF can trigger sanctions from the applicable Data Protection authority, up to 4% of global annual turnover under GDPR (Art. 83), with equally severe penalties under CCPA (§1798.155), LGPD (Art. 52), and other frameworks. [1][2]
- Inadequate Data processing agreements are actively sanctioned. In 2022, France’s CNIL fined Dedalus Biologie €1.5 million for Art. 28 GDPR violations as a Data processor handling health data from 28 medical laboratories, exposing records of nearly 500,000 patients. [3] If a software provider faces such consequences for deficient processing agreements, the exposure for a global pharmaceutical sponsor is orders of magnitude higher.
- A delay in detecting non-compliance can halt a Clinical launch. According to a 2024 peer-reviewed analysis published in Therapeutic Innovation & Regulatory Science by Tufts CSDD, the estimated direct daily cost to conduct Phase II and III clinical trials is approximately $40,000 per day, with Phase III trials averaging higher. When combined with lost prescription drug sales (approximately $500,000 per day), total delay costs can exceed $540,000 per day. [4]
- In the event of litigation, reliance on an unqualified AI tool could be held as a breach of the sponsor’s or CRO’s duty of care, a risk amplified by the EU AI Act (Regulation 2024/1689), which classifies AI systems in Healthcare and Clinical Research as high-risk and imposes specific obligations on deployers. [5]
Sources:
[1] GDPR Art. 83, General conditions for imposing administrative fines
[2] CCPA §1798.155, Administrative enforcement
[5] EU AI Act, Regulation (EU) 2024/1689, Annex III (high-risk AI systems)
⚠️ The Trap of False Assurance
The most insidious risk is not that the AI gets it blatantly wrong. It is that it gets it subtly wrong, with a confident tone and convincing phrasing. A generic language model can produce a legal opinion that “looks” correct, coherent structure, appropriate terminology, plausible references, yet omits a critical requirement or misinterprets a rule.
For a non-specialist, this kind of error is virtually undetectable. And that is precisely where the danger lies: the tool creates an illusion of compliance without delivering the guarantee.
In a domain where every word matters, “the participant will be informed” versus “the participant must be informed”, “Data may be used” versus “Data shall be used exclusively for”, a false sense of security can have irreversible consequences.
🤝 The Responsible Approach: AI + Domain Expertise
Should we then abandon AI altogether? Absolutely not. The wave is real, and it is already changing behaviours. But it must be ridden with method.
The responsible approach rests on three pillars:
Deep customisation of the tool. An AI tool used in a regulatory context must be configured with domain-specific rules: compliance checklists, Data Protection frameworks (GDPR, CCPA, LGPD, PIPL, national regulations), relevant case law, supervisory authority guidances, codes of conduct, and validation criteria tailored to each document type. This is what we call “skills”, layers of domain expertise embedded within the tool.
Qualified human oversight. AI accelerates the work; it does not replace it. Every output must be validated by a professional with mastery of the applicable regulatory framework. AI is a co-pilot, never the captain.
Traceability and auditability. In the event of an inspection or dispute, it must be possible to demonstrate that the tool was used within a controlled framework, with documented validation rules and traceable human oversight. The absence of such traceability is itself an aggravating factor.
🔍 The True Cost of Apparent Savings
Using an off-the-shelf generative AI tool for regulated documents may seem cost-effective: no consultant, no expensive configuration, an answer in seconds. But this saving is a mirage.
The cost of non-compliance detected late in a clinical trial is measured in regulatory fines, launch delays, reputational damage, and sometimes direct harm to participants. Comparing these risks to the cost of expert tool configuration is meaningless. It is like comparing the price of a fire extinguisher to the cost of a fire.
👉 Conclusion: Speed Without Direction Is a Risk
Generative AI gives us unprecedented speed. But speed only matters when you know where you are going. Using such a system alone, without embedded domain expertise, without deep customisation, without qualified oversight, means taking a legal, regulatory, and ethical risk that few organisations can afford.
The question is no longer “should we use AI?” but “how do we use it responsibly, within a framework that protects the organisation, its partners, and Data subjects?”
At MyData-TRUST, that is exactly the question we work on every single day.
