How many people does an AI CoE need at the start?

Two to four people is enough: a lead who owns the use-case list and standards, one or two engineers who have shipped AI to production, and a part-time security reviewer. Add a champion inside each product team before hiring more central staff.

Who should an AI CoE report to?

Usually the CTO or CIO, with a business sponsor on a steering group. Microsoft's Cloud Adoption Framework recommends monthly progress reviews with sponsors and direct access to C-level decision makers. A CoE buried three levels down can publish standards but can't enforce them.

Does an AI CoE need its own budget?

Yes. Microsoft's Cloud Adoption Framework says executive sponsorship supplies the budget and authority a CoE needs to enforce standards. At minimum, fund the shared platform, model and tooling costs, and the core team's time. Product teams can still pay for their own builds.

Is an AI CoE the same as an AI governance committee?

No. A governance committee sets policy and approves high-risk uses on a schedule. A CoE is a working team that also runs the platform, trains people and helps ship workflows. The AWS guide treats a steering committee and an ethics board as parts of a CoE's structure, not the whole of it.

Can a company outsource its whole AI CoE?

Not the parts that need authority. Strategy, standards and budget decisions must sit with employees, because an outside firm can't enforce rules on your teams. Engineering, evaluation and training can come from a partner in year one. Origins AI's services page, for example, lists build-operate-transfer partnerships alongside dedicated AI teams. The Microsoft Cloud Adoption Framework (CAF) defines the CoE as an internal team.

When should an AI CoE be dissolved or merged back?

Shift it rather than dissolve it, once product teams run AI safely and the platform enforces the rules automatically. Microsoft's guidance calls this moving from centralized control to an advisory team. In its hub-and-spoke model, platform, identity and security stay central; delivery moves out.

What Is an AI Center of Excellence and Do You Need One? (2026)

Quick Answer: An AI center of excellence is a small cross-functional team that sets AI guardrails, shares tools and turns successful pilots into production capability. You need one once several teams build with AI and start duplicating work or skipping review. Microsoft's Cloud Adoption Framework advises starting centralized, then shifting to an advisory role as teams mature.

Most companies don't plan an AI CoE. They notice three teams using three model providers, nobody tracking which prompts touch customer data, and a months-old pilot still stuck short of production. An AI center of excellence exists to fix that pattern, not to own every AI project.

Here's what it owns, who sits on it and what it should ship first.

What is an AI center of excellence?

An AI center of excellence is a standing team that decides where the company uses AI, sets the rules for using it safely, and gives product teams shared tools so no project starts from zero.

IBM describes it as an operational structure for the adoption, optimization and governance of AI across an organization. Microsoft's Cloud Adoption Framework guidance frames it as an internal team of experts that prevents fragmented or ungoverned AI adoption.

The work falls into five jobs:

Strategy. A ranked list of use cases, each with an owner, a data source and a success metric.
Governance. Rules on approved models, data classes, human review and logging, applied before a pilot reaches customers.
Enablement. Training, templates, prompt libraries and reference architectures.
Adoption. Moving proven pilots into production and turning one team's success into a reusable pattern.
Measurement. Adoption, cost and business outcomes per use case, reported to leadership.

It isn't a research lab or a committee that only says no; its value shows up as more AI workflows running in production.

What does an AI CoE own and what does it hand to product teams?

The CoE owns the shared layer: platform, identity and access, the security baseline, standards and a registry of what's running. Product teams own delivery, meaning they build, run and improve their own AI features inside those guardrails.

Microsoft's guidance on CoE operating models calls this split, a central platform with federated delivery or hub and spoke, the most common arrangement at scale.

Pick an AI CoE operating model by maturity and risk, not by your org chart.

Operating model	Typical core team	The CoE owns	Product teams own	Fits when
Lightweight	2 to 4, some part-time	Use-case list, use rules, approved tools, reuse library	What they build, after a CoE review	One or two teams build with AI
Hub-and-spoke	4 to 8, plus a champion per team	Platform, identity, security baseline, standards, registry	Building and running their features	Several teams share data and need consistency
Centralized	8 or more, mostly full-time	Strategy, platform and most delivery	Requirements and acceptance	Early maturity with high risk, such as customer-facing agents

Team sizes are rules of thumb. Most mid-size companies start lightweight and move toward hub-and-spoke as more teams build.

When does a company need an AI CoE instead of a partner?

You need a CoE when AI is becoming a permanent capability across several teams. A partner fits when you need a few workflows built quickly. For most product teams the answer is both: the CoE owns standards and reuse, and a partner builds under them. A CoE is one part of a broader program, and what AI transformation involves in 2026 sets out the workstreams around it.

Set up a CoE when three or more teams build AI at once, AI is part of what customers pay for, security reviews have become a case-by-case bottleneck, or nobody owns the bill for overlapping model providers.

Use a partner alone when one or two workflows support internal operations, nobody on staff has shipped retrieval or agent work, or a deadline rules out recruiting first.

Do both when the first workflows must ship now and you want to own the capability later. A two-person CoE can write the rules while a partner builds, with handover planned from the start.

You may not need a new team. Microsoft's framework says to fold AI into an existing Cloud Center of Excellence where one exists. Your position on an AI maturity model is a useful tiebreaker.

How do you staff an AI center of excellence?

Staff it with a named lead, an executive sponsor and a few engineers who have shipped AI to production, then borrow security, legal and domain expertise part-time. Roles you can't recruit quickly can come from a partner while you hire.

Role	Responsibility	In-house or partner
Executive sponsor	Budget, authority, cross-team decisions	In-house
CoE lead	Use-case list, standards, reporting	In-house
AI or ML engineer	Reference implementations, design reviews	Either; often a partner at first
Platform or MLOps engineer	Model access, gateway, deployment, cost tracking	Either
Evaluation specialist	Test sets, quality metrics, regression checks	Partner early, in-house later
Security, privacy and legal	Data classes, vendor review, use rules	In-house, part-time
Product champions	One per team; bring use cases, carry standards back	In-house

Microsoft's role list adds senior data scientists, AI governance experts and AI operations professionals. The AWS guide to an AI/ML CoE adds product strategists, domain experts and project managers.

Engineering roles are the slow ones to fill. Placing engineers from an AI development company inside the CoE while you recruit is a common stopgap; the trade-off is covered in AI development partner vs in-house team.

How should an AI CoE work with an outside AI agency?

The CoE sets the rules and the agency builds to them. Agree five things before the first sprint:

Same standards, no side door. Approved models, logging, data classes and review gates apply to the agency as they do to internal teams.
Company repositories and accounts. Code, prompts, model keys and cloud resources sit in accounts the company controls from the first commit.
A shared evaluation set. The CoE owns the test cases that define a good answer, and the agency reports against them on every release.
Contributions to the reuse library. Connectors, prompt templates and evaluation harnesses go into the CoE library, documented.
A named handover. Each workflow gets an internal owner who takes over runbooks and on-call on an agreed date.

A build-operate-transfer contract goes further: it makes handover a contract term.

What should an AI CoE deliver in its first 90 days?

In its first 90 days an AI CoE should ship a ranked use-case inventory, written use rules, one shared platform, two workflows in production and a dashboard tracking them. Use this as a scorecard.

Days 1 to 30. List every AI tool already in use, including unofficial ones, and rank use cases. Publish use rules and an AI governance policy naming who approves new use cases.
Days 31 to 60. Choose one stack: model access, a gateway for keys and cost tracking, logging and an evaluation harness. Start the top two workflows on it.
Days 61 to 90. Put both workflows in front of real users with monitoring on, and track adoption, cost, quality and the business metric each was meant to move.

A CoE that reaches day 90 with a policy but no production workflow has become a committee.

What mistakes should you avoid when setting up an AI CoE?

Most failures turn the CoE into a gate, or leave it without authority or money.

Becoming the approval queue. Microsoft names approval delays and knowledge bottlenecks as signs central control is hurting adoption.
No production mandate. A CoE measured on pilots produces pilots.
No budget of its own. Without money for a platform and engineers, it can only write documents.
Security and legal joining late. Bring them in at intake, not launch review.
All data scientists, no engineers. Production needs platform, integration and evaluation skills.
No exit plan. Decide early which duties move to product teams as they mature.

How Origins AI works alongside internal AI teams

Origins AI (originshq.com) is an AI-augmented engineering company that builds AI workflow automation for product teams. Next to a CoE, it can act as the build partner: engineers who ship the first workflows under the CoE's standards, then hand them to internal owners. Its AI services page lists AI consulting, automation and AI product development, integrated through APIs and custom connectors.

For staffing gaps, the AI specialist talent page describes LLM, computer vision, NLP and MLOps specialists, engaged for a three-month project or a long-term seat. The services page lists dedicated-team, project-based, time-and-materials and build-operate-transfer engagements, plus training workshops for internal teams.

Origins AI does not publish a rate card. Its security FAQ describes encryption at rest and in transit, secure authentication, continuous monitoring and least-privilege access. Case studies on its site include YesMadam and RagaAI, where Origins AI reports it worked as a founding member on an AI testing and deployment platform.

Talk to an engineer

Setting up a CoE and need engineers to ship the first workflows? Bring your use-case inventory and the 90-day checklist above. Book a call.

Written by Apoorva Kumar, Co-Founder & CEO, Origins AI.