Artificial intelligence (AI) has rapidly transformed from a futuristic concept into an everyday tool for insurance claims professionals and legal practitioners. While AI platforms offer unprecedented capabilities in claims analysis, legal research, and document review, they also present significant risks that demand careful attention. Professionals who want to harness AI’s potential can implement simple practices to mitigate its inherent risks.

AI Applications in Claims Handling
The central risk of AI lies not in inherent unreliability but in professionals placing undue trust in AI-generated outputs without exercising independent judgment. AI can fail through “hallucinated” data (information fabricated from whole cloth), flawed pattern recognition, or biased training sets. When professionals accept AI outputs at face value without verification, the consequences can prove severe: incorrect coverage determinations, bad faith exposure for carriers, and professional liability for individual practitioners.

Consider AI as the most capable summer intern your office has ever employed—one who is incredibly bright, producing work at a prolific rate, but still requiring oversight. No matter how impressive the output appears, no prudent professional would stake their reputation on unverified work from someone new to the industry. AI is no different. While platforms may produce work product that includes industry-specific terminology and engage in familiar looking analysis, they often lack the ability to recognize nuance—a skill only experienced claims professionals can bring.

Claims professionals commonly deploy AI tools in four primary areas: automated coverage analysis, reserves and valuation, fraud detection, and litigation outcome prediction. These platforms excel at digesting policies, endorsements, and complaints to generate initial coverage assessments. However, they cannot replace the specialized judgment that comes from professional experience.

AI tools function as “people pleasers” in that they tend to tell users what they want to hear rather than providing critical analysis. This confirmation bias can lead to overconfidence in flawed conclusions. If an AI tool consistently validates your position without ever identifying potential weaknesses, treat that as a red flag. Treat the AI output as a starting point for your own analysis and bring a healthy skepticism to its conclusions and recommendations.

Bad Faith Implications for Insurers
Insurers face particularly high stakes. If an AI-generated coverage determination proves incorrect—for example, recommending denial of a claim that should have been covered—the carrier faces potential bad faith exposure, including assigned judgments and direct litigation. Coverage attorneys representing policyholders have grown increasingly aggressive, and penalties for bad faith handling continue to escalate across jurisdictions.

AI prompts and queries may become discoverable in bad faith litigation. A prompt asking the tool to “find a way to deny coverage” could constitute powerful evidence of bad faith intent. Claims professionals should approach AI interactions with the understanding that their queries could become exhibits in future litigation. Make queries open ended, without any suggestion that a specific result is expected or anticipated. Be mindful that the final recommendation is yours to make, not the platform’s.

Defense Counsel and Rule 11 Obligations
In the same way claims professionals are subject to the limitations of AI, so too are the defense counsel they hire. The legal profession has witnessed a surge in sanctions arising from AI-generated court filings containing fabricated case citations. Under Federal Rule of Civil Procedure 11, attorneys certify that they have conducted a reasonable inquiry into the facts alleged and the law cited. Filing an AI-generated brief without verifying case law constitutes a Rule 11 violation that can expose both counsel and client to significant consequences.

The landmark Mata v. Avianca case first brought widespread attention to AI hallucinations in legal filings. Since then, courts have lost patience with such errors. Sanctions now routinely include monetary penalties, mandatory continuing legal education, disciplinary referrals, and procedural consequences including case dismissal or striking of pleadings. Many federal and state courts have adopted standing orders requiring disclosure of AI use in court filings.

For claims professionals who supervise defense counsel, these developments carry direct implications. Insurers may face bad faith or negligent claims handling allegations if their retained counsel’s AI-related failures compromise an insured’s defense. As the use of AI by counsel increases, it is no longer wise to simply trust that counsel is using it in a responsible, compliant manner. Claims professionals are increasingly placing strict limits on how AI may be used and requiring adoption of codes of conduct in an effort to minimize exposure to carriers.

Confidentiality and Privilege Concerns
The distinction between “closed” and “open” AI systems presents significant privilege implications. Closed systems—proprietary platforms used under secure enterprise agreements—maintain confidentiality of inputted information. Open systems, such as publicly available versions of ChatGPT, Claude, or Gemini, do not provide the same protections. Because open systems use submissions by one user to refine the experience of another, submission of privileged information provides access to the substance of that submission to countless users. In this way, uploading privileged documents or confidential client information to an open AI platform may waive attorney-client privilege or work product protection.

Accordingly, best practice for both claims professionals and their counsel is to use open AI systems only for general inquiries and subject matter education—never for analysis of specific case materials or confidential information.

Key Takeaways: Verification Is Non-Negotiable
With these concerns in mind, there are simple protocols that may be implemented to mitigate risk inherent in generative AI use:

1. Verification remains non-negotiable.
Every AI output must be independently verified against actual policy language, complaint allegations, and applicable law. AI serves as a starting point, never a conclusion.

2. Document the human element. 
Claims notes should reflect independent professional judgment at every critical decision point. When you disagree with an AI output, document that disagreement and the reasoning behind your ultimate decision. Avoid formulaic notations; detailed documentation of your analytical process provides crucial protection in potential bad faith litigation.

3. Hold defense counsel accountable. 
Claims professionals should inquire about counsel’s AI policies, verification procedures, and the specific tools in use. Review significant briefs before filing and ensure the cited legal authorities are accurate and properly applied.

4. Understand your tools. 
Know the capabilities and limitations of the specific AI platforms you use. Proprietary, industry-specific tools trained on carrier claims history will generally produce more reliable outputs than general-purpose consumer platforms.

AI represents a powerful addition to the professional toolkit, but it does not—and cannot—replace the judgment, experience, and critical thinking that define competent claims handling and legal practice. The professionals who thrive in this new environment will leverage AI’s efficiency while maintaining the rigorous verification standards that have always marked excellence in these fields. As the regulatory landscape continues to evolve and courts grow less tolerant of AI-related failures, the imperative for responsible use has never been clearer.

This article is based on the Wilson Elser Forum webinar, When AI Fails: Risks of Overreliance on Artificial Intelligence in Insurance Claims and Defense Litigation. Watch the playback. (Passcode: Q$6^Kn=S)