The Question Clients Are Actually Asking
Managing partners are not asking whether AI is useful. That conversation is largely settled. The question arriving in our conversations now is more specific: can we run models powerful enough to matter without routing client data through a third-party API?
For accounting and law firms, that question is not theoretical. Client confidentiality is a professional obligation. For consulting firms, proprietary client methodologies and engagement data carry similar sensitivity. The compliance exposure from an inadvertent data disclosure is real, and the reputational exposure is worse.
The honest answer is yes. But the conditions matter.
What "Sending Data to an API" Actually Means
When a user submits a prompt to a hosted model, their input typically travels to the provider's infrastructure, is processed there, and the response is returned. The model itself lives on the provider's servers.
Most major providers offer enterprise agreements that include data-processing terms, no-training commitments, and regional data residency options. Those agreements reduce risk materially. But they do not eliminate it, and for some firms and some data types, residual risk is not acceptable.
The alternative is to run the model yourself: on your own hardware, in a private cloud tenant your team controls, or in a hybrid configuration where inference happens inside your perimeter. This is what private inference means.
Open-Weight Models Have Closed the Performance Gap
For most of the past three years, the practical objection to private inference was performance. The best models lived behind APIs, and self-hosted alternatives were noticeably weaker.
That gap has narrowed. A current generation of open-weight models, meaning models whose weights are publicly available and can be deployed without a licensing agreement that mandates cloud hosting, now performs at a level that is genuinely useful for the document-heavy, reasoning-intensive work that characterises professional services.
Code-generation benchmarks and legal reasoning benchmarks both show competitive performance from open-weight models at parameter counts that are practical to run on a single server or a small private-cloud instance. [DATA NEEDED: current benchmark scores for leading open-weight models on legal/document reasoning tasks, sourced from published evals such as LegalBench or MMLU-Pro]
This does not mean open-weight models are identical to the frontier hosted models in all respects. For some tasks, particularly complex multi-step reasoning chains, hosted frontier models still hold an edge. The relevant question is not which model wins a benchmark. It is whether a model is strong enough for your specific use cases. For most document review, research summarisation, drafting assistance, and internal Q&A, the answer is yes.
Three Deployment Patterns Worth Understanding
Firms considering private inference usually end up in one of three configurations:
On-premise inference. The model runs on hardware within the firm's own data centre or server room. No data leaves the building. This offers the strongest sovereignty guarantee. The trade-offs are capital expenditure, GPU availability, and internal operational overhead.
Private cloud tenant. The model runs in a cloud tenant the firm controls, on dedicated capacity rather than shared infrastructure. Major cloud providers offer this configuration, often under what they call private or isolated compute. Data residency is contractually bounded. This is the most common choice for mid-market professional services firms because it combines strong isolation with manageable operational overhead.
Azure AI Foundry (or equivalent). If a firm is already standardised on Microsoft Azure, Foundry provides a structured environment for deploying both Microsoft and third-party models within the firm's Azure tenant. Governance, logging, and access controls integrate with the firm's existing Azure Active Directory and compliance posture. This is frequently the fastest path to governed private inference for firms already inside the Microsoft ecosystem.
What Firms Actually Need Before They Deploy
Private inference is not a single decision. It is a chain of decisions. Firms that treat it as a procurement exercise, rather than a governance programme, discover this the hard way.
Several things need to be in place before private inference is genuinely safe and useful:
A clear data classification policy. Which data can the model see? Which cannot? Without this, private infrastructure does not protect you. A model running inside your perimeter can still be prompted with data it should not process if users have not been trained and the system has not been scoped.
Access controls that mirror your existing permissions. The model layer should not collapse the permission structure your firm already maintains around client files, matter documents, and engagement data. Role-based access at the model and retrieval layer is essential.
Logging and audit trails. Regulators and professional standards bodies want evidence of what happened and when. Inference logs, prompt records, and output audit trails are governance requirements, not optional extras.
A retrieval layer that understands your data. Raw model deployment without a knowledge retrieval layer, one that connects to your document management system, your email archive, your matter files, is a model that can reason but cannot find. The retrieval layer is where most of the firm-specific value comes from, and it is also where most of the governance complexity lives.
Tested evaluation and output review processes. Private inference does not mean unsupervised inference. For professional services outputs that carry legal or regulatory weight, human review workflows need to be designed in, not bolted on afterward.
A Note on Hybrid Architectures
Firms do not need to choose one approach for everything. A practical pattern is to route highly sensitive data, specific client files, matter documents, non-disclosure-covered information, through a private inference layer, while using hosted models for lower-sensitivity internal tasks: drafting marketing content, summarising public research, generating internal briefings.
This hybrid posture is often more cost-effective than full private inference and more protective than full hosted-API reliance. Getting the routing logic right is a governance design problem, not purely a technical one.
Practical Takeaways
If your firm is evaluating private inference, these are the questions that matter most right now:
- Do you have a data classification policy? If not, no inference architecture protects you. Start there.
- Are you standardised on Azure? If yes, Azure AI Foundry is the fastest path to governed private inference and deserves a structured evaluation before you build anything custom.
- What use cases are you actually solving? Performance requirements vary by task. Document review and research summarisation have lower model-strength requirements than complex multi-step legal analysis. Scoping this honestly shapes your architecture decision.
- Who owns the governance layer? Private inference does not manage itself. Someone in your firm, or a partner sitting alongside your team, needs to own access control, logging, and policy enforcement across the model stack.
- What is your retrieval strategy? A model without retrieval is a capable reasoner with no access to your firm's knowledge. The retrieval design, what it connects to, how it respects permissions, how it handles confidential matters, is where most of the practical value and most of the risk concentration sits.
ClearFuture works with professional services firms as an embedded AI Office, advising on architecture, deploying the stack, and staying on to govern and optimise it. If you are evaluating private inference and want a structured view of your options before committing, book a 30-minute call or reach out to jp@clearfuture.ai.