Ed. Note: This is the third article in our series, Smarter AI Vendor Contracts: Legal Strategies for Protecting Your Business. Read Part 1 and Part 2.
A contract is only as strong as the diligence that informs it. Before a single term is negotiated, the procurement team, including the organization’s legal counsel, should conduct structured due diligence that maps the vendor’s capabilities, practices, and risks to the protections the contract will need to provide.
This article outlines the key diligence workstreams and the operational safeguards that should accompany any AI vendor engagement.
Data Inventory and Use Mapping
The first question is deceptively simple: what data will be shared with the vendor?
The answer, however, requires a granular inventory that goes beyond broad categories. If the engagement involves sensitive data (the kind of data that could facilitate identity theft or cause significant harm if exposed) the organization should ask whether synthetic data can be used instead. Synthetic datasets that preserve the statistical properties of real data without exposing actual personal or proprietary information can materially reduce risk without materially reducing the utility of the AI tool.
Equally important is the question of data return. Will the customer need the data back? What about prompts — do they constitute customer data, and can the customer retrieve them? These questions have implications for data portability, deletion, and the customer’s ability to transition to a different vendor.
The diligence process should also map how data will be processed, retained, de-identified, or used for training and fine-tuning (if allowed). When the parties discuss “de-identified data,” the de-identification standards should be specified and tested, and not just assumed. If the vendor claims that data is anonymized before it is used in training, the customer should probe the methodology for anonymization and assess whether re-identification is technically feasible. It is important to note that AI tools have made it easier to re-identify data by being able to gather and connect multiple data points. In the context of training an AI tool with de-identified or anonymized data, the discussion surrounding the potential for re-identification should be subject to close analysis.
If applicable to the AI tool, data localization and cross-border flows should round out this workstream. Organizations should question: where will data be processed and stored? Will it transit through jurisdictions with different privacy regimes? Are there contractual or regulatory restrictions on data transfer that the vendor’s infrastructure may not accommodate?
Model Governance
Model governance diligence examines the AI system itself.
The customer should understand the model type, for instance: is the AI model a proprietary model, an open-source model, or a fine-tuned version of a third-party foundation model? Each architecture has different implications for control, auditability, and intellectual property (“IP”) risk.
Training data sources should be disclosed by the vendor and evaluated. An assessment of the training data should evaluate whether the vendor has conducted provenance reviews of its training data, and if there are any known risks of copyrighted, biased, or otherwise problematic content in the training data.
The customer should review the results of safety testing, bias assessments, and red-teaming results to gain insights into the vendor’s maturity and the model’s readiness for the customer’s use case. The customer should request documentation of these activities and evaluate whether the testing reflects the customer’s deployment context, for instance: a model that has been safety-tested for consumer chatbot use may not be appropriate for healthcare decision support.
Additionally, with AI tools, change management and versioning practices are critical. How does the vendor handle model updates? Is there advance notice? Can the customer test new versions before they are deployed? What happens to the old version? Are there assurances that new versions of the tool will maintain similar functions? If not, customers may find they are saddled with an AI tool that no longer serves the functions for which it was originally purchased.
The risks associated with inaccurate, biased and/or unreliable outputs make human-in-the-loop controls (i.e., mechanisms that require or enable human review of AI outputs or training data before they are acted upon) crucial, and should be non-negotiable in certain use cases. Accordingly, human-in-the-loop controls should be evaluated both as a feature of the vendor’s product and as a requirement of the customer’s internal processes.
Security Program Review
As with any software tool, the customer should carry out a security review to ensure appropriate safeguards are in place to protect the customer’s data. However, the security diligence workstream for AI vendors should build on the standard security review but add AI-specific dimensions. Additionally, due to the technical nature of reviewing security safeguards, information technology (“IT”) professionals or third-party advisers may need to be involved as part of the security program review.
Similar to software tools, the customer should evaluate:
- Certifications such as SOC 2, ISO 27001, and industry-specific frameworks, which provide a baseline for security. However, reliance on certifications is, generally, not sufficient, and additional review of the vendor’s safeguards is recommended.
- Penetration testing and vulnerability management practices to ensure they are current and relevant to the vendor’s AI infrastructure, not just its general IT environment.
- Whether data processed by the AI tool is encrypted to cover data at rest, in transit, and in use (where technically feasible).
- Access controls and key management practices to ensure they reflect the sensitivity of the data being processed.
In addition to the above criteria, the customer should also evaluate certain criteria relevant to AI tools, such as:
- Ensuring the secure development lifecycle extends to model development, not just software development.
- How are models tested before deployment?
- What safeguards prevent data leakage during training?
- How are model artifacts stored and protected?
The customer should inquire about and explore the technical constraints that the vendor may have in place to prevent the model from producing certain categories of output. Such guardrails should be tested, and not taken on faith.
While it is important to confirm the security measures employed by the vendor, such review will also, ultimately, inform contractual provisions and internal operational controls in using the AI tool.
Subcontractor Ecosystem
AI vendors rarely operate in isolation. The subcontractor ecosystem (e.g., model providers, hosting providers, annotators, content filters, etc.) introduces additional layers of risk and complexity.
If subcontractors have access to customer data, it is important to confirm the trustworthiness of such subcontractors. It may not be possible to directly inspect or audit subcontractors, but the vendor should be able to provide certain information regarding subcontractors to allow the customer to gauge the acceptability of such subcontractors. At the least, inquiries into subcontractors can inform contractual terms placing the obligations of conducting adequate due diligence into subcontractors on the vendor and placing responsibility on the vendor for all acts and omissions of subcontractors.
Diligence should identify all material subcontractors, assess the flow-down of contractual obligations, and evaluate the transparency of the vendor’s subcontractor relationships. If the vendor relies on a third-party foundation model, the customer’s data may be subject to that third party’s terms of service unless the vendor has negotiated appropriate protections.
Operational Controls
Operational controls bridge the gap between the contract and day-to-day use of the AI tool.
Usage policies should define how the tool may and may not be used within the customer’s organization. Rate limiting and content moderation features should be evaluated for adequacy. Guardrails — technical constraints that prevent the model from producing certain categories of output — should be tested, not taken on faith.
Monitoring capabilities allow the customer to track usage patterns, detect anomalies, and identify potential misuse. Incident response plans should address AI-specific scenarios, including model failures, data leakage through outputs, and adversarial attacks.
Customer configuration responsibilities should be clearly documented. Many AI tools offer significant configurability, and the allocation of responsibility between vendor defaults and customer choices has direct implications for liability.
Vendor Viability
In a rapidly evolving market, vendor viability is a legitimate diligence concern.
In the AI space, acquisitions, pivots, or failures are common, and customers should consider how long the vendor may continue to exist in its current form (e.g., is it a well-established company introducing a new AI tool that will most likely consider to operate in a similar capacity, or is it a small start-up that might sell to a larger company or close its operations?). The answer affects not only the commercial relationship but also the contractual treatment of assignment. In a traditional vendor engagement, customers will typically resist permitting assignment of the contract. In the AI space, given the fluidity of a vendor’s existence, there may be good reason to permit assignment under defined conditions, or to ensure that the contract includes robust transition and data portability provisions that protect the customer regardless of the vendor’s fate.
Operational Controls
Operational controls bridge the gap between the contract and day-to-day use of the AI tool.
As a customer evaluates the AI Tool, and comes across certain risks that cannot be mitigated (e.g., turning off a particularly risky feature), or if a tool or contract cannot be modified (such as an off-the-shelf AI tool), the customer should consider usage policies that define how the tool may and may not be used within the customer’s organization.
The customer should inquire whether the AI tool has monitoring capabilities that allow the customer to track usage patterns, detect anomalies, and identify potential misuse. Such monitoring capabilities can provide customers with a snapshot of the health of the AI tool at a glance and allow the customer to report issues to the vendor or modify its usage accordingly.
The customer should consider adding AI-specific scenarios to its incident response plans (including model failures, data leakage through outputs, and adversarial attacks) to ensure, if an incident occurs, it can quickly and effectively respond to it through an established process instead of trying to formulate a plan in the moment.
Many AI tools offer significant configurability. However, customer choices that vary from vendor defaults may result in allocation of responsibility between the vendor and customer and have direct implications for liability. Customer configurations and the parties responsible for them (whether in the customer’s organization or the vendor) should be clearly documented.
Aligning Stakeholders and Implementing Usage Policies
Due diligence is not solely a legal function. Aligning internal stakeholders (e.g., information security, privacy, compliance, the business unit sponsoring the engagement, IT, etc.) ensures that the findings from diligence are translated into contractual requirements and operational practices across the organization.
Internal usage policies that complement contractual controls are essential. A contract that prohibits the vendor from training on customer data is undermined if employees are uploading proprietary information to an unapproved AI tool. Policies, training, and technical controls must work in concert with the contract.
From Diligence to Contract Terms
The ultimate purpose of pre-contract diligence is to inform the contract. Findings from each workstream should be translated into specific terms, conditions, representations, warranties, service level agreements, and audit rights, for instance:
- A gap identified in the vendor’s security program becomes a security commitment in the contract.
- A risk identified in the vendor’s subcontractor ecosystem becomes a subprocessor approval right.
- A limitation identified in the model’s performance becomes an acceptance criterion.
The next article in this series will present the contract architecture for AI vendor engagements, showing how the risks identified in Article 2 and the diligence conducted in this article come together in a structured set of contract terms.
