How to Structure Artificial Intelligence at Scale in Companies

In the first scenario, distinct teams can consume model APIs, implement RAG pipelines, create copilots, or experiment with agents independently. In the second scenario, the organization can do all of this in a repeatable manner, across different business domains, while maintaining consistent standards of architecture, security, identity, observability, evaluation, costs, governance, and operation.

This difference may seem irrelevant when there are two or three experiments. It becomes an architectural problem when there are 20, 50, or 100 concurrent initiatives.

It is at this point that a central function of AI Center of Excellence (AI CoE) becomes necessary.

Not necessarily a formal department titled "CoE." But some organizational structure needs to own the AI control layer of the company.

The Microsoft Cloud Adoption Framework explicitly positions the AI CoE as a structure aimed at eliminating fragmented and ungoverned AI adoption [¹]. AWS works with centralized and federated models of AI/ML CoE [²][³]. The discussion, therefore, is not simply "centralize or decentralize," but what should be centralized and what should remain distributed.

The central thesis of this article is:

"

Centralize the control layer. Distribute the execution layer.

The problem begins when each squad builds its own AI platform

Consider a company with multiple business units.

Team A needs intelligent support and integrates directly with the OpenAI API. Team B needs to analyze contracts and adopts Claude. Team C implements a RAG with Gemini. Team D builds an agent that executes internal operations. Team E creates its own embedding pipeline.

Technically, none of these moves is necessarily incorrect in isolation.

The problem materializes months later when the company starts to operate simultaneously:

• five distinct authentication strategies;
• four independent implementations of RAG;
• three vector databases without standardization;
• multiple versions of embedding models;
• different chunking strategies;
• prompts stored directly in the source code;
• provider credentials distributed without centralized management;
• models being invoked directly by applications, without an abstraction layer;
• no common taxonomy for evaluation;
• different auditing mechanisms;
• different policies for prompt and inference data retention;
• different mechanisms for PII masking;
• no central token budget;
• agents operating with different permission levels;
• no consolidated view of operational risk.

This constitutes architecture drift, and will almost inevitably be followed by governance drift.

The organization ends up with several AI systems but lacks an AI architecture.

The CoE should not be a project factory

There is a frequent antipattern in the implementation of Centers of Excellence: the company centralizes all AI specialists and determines that any AI project must go through this team.

This approach resolves part of the fragmentation but introduces another problem: throughput. The CoE becomes a bottleneck, and when dozens of areas start demanding AI simultaneously, the predictable result is operational bottleneck.

For this reason, the most suitable model for the CoE is a combination of Platform Team + Architecture Function + Governance Function + Enablement Function. The main work should not be to build all the AI products of the company. It should be to build the paved road, the infrastructure that allows these products to be built consistently, securely, and governed.

"

CONTROL LAYER (AI CoE)

• Architecture
• Platform Engineering
• AI Security
• Governance
• GenAIOps / LLMOps
• Evaluation
• FinOps

⬇️ (provides the foundations for)

EXECUTION LAYER (Product Squads)

• Squad A: Fraud / Risk
• Squad B: Customer Experience
• Squad C: Operations

The domain remains with the squads. The AI fundamentals do not need to be rebuilt by each squad.

The most important asset of the CoE is an AI Platform

When AI starts to be used transversally in the organization, allowing direct and unrestricted access of systems to model providers creates strong coupling and hinders governance.

A sustainable corporate architecture must introduce an intermediary layer, an AI Gateway, which serves as an integration and enforcement point between applications and providers:

"

Flow with Enforcement:

1. Applications / Agents / Copilots
2. ⬇️ AI Gateway (enforcement point)
3. ⬇️ Orchestrated Services: Policy Engine, Model Router, Semantic Cache
4. ⬇️ Providers: OpenAI, Anthropic, Gemini, Vertex/Bedrock, Self-hosted
5. ⬇️ Governance & Security applied to every request

An important point: the AI Gateway functions as an enforcement and integration point, but should not own all the capabilities it orchestrates. In a real architecture, it delegates to specialized services:

• Policy Decision Point
• Model Router
• Guardrail Service
• Identity / AuthZ
• Cost Metering
• Observability Collector
• Semantic Cache
• Model Registry

This distinction prevents the gateway from becoming a monolith, a God Service that concentrates all complexity. It coordinates; the services resolve.

The capabilities that pass through the gateway include authentication and authorization, tenant isolation, model allowlist, rate limiting, cost attribution, distributed request tracing, prompt filtering (PII detection), DLP, intelligent model routing, circuit breaker, caching, and auditing.

With this layer, replacing the model used by a specific workload no longer requires changes in dozens of applications. The routing decision can be resolved dynamically based on:

routing_policy = capability + data_classification + latency_requirement + cost_budget + region + compliance + context_window + model_quality

In this model, "which LLM do we use?" ceases to be a static architectural decision and becomes a runtime policy.

Control layer versus execution layer

An effective way to organize responsibilities is to explicitly separate two operational planes and, more importantly, show how these components connect at runtime.

The flow of a request follows the direction of workloads towards providers, passing through the platform:

"

1. CONTROL LAYER (CoE)

• Developer Portal
• AI Platform APIs
• Registries (Prompt, Model, Agent)
• Policy Engine & Release Gates

⬇️

2. EXECUTION LAYER (Runtime)

• Workloads: Squads A, B, C (Applications, Agents, Copilots)
• ⬇️
• AI Gateway
• ⬇️
• Model Router (routing to GPT, Claude, Gemini, etc.)

Although Gateway and Model Router are capabilities managed and governed by the CoE, they operate in the runtime of the execution layer. The control layer defines policies, configuration, and approval criteria; the execution layer applies these decisions during requests.

In practical terms, this distinction translates to:

Predominantly under the responsibility of the CoE:

• Model Registry, Prompt Registry, Agent Registry
• Evaluation Registry, Observability Standards, Risk Classification
• Policy Engine, Release Gates, Cost & Security Controls
• Reference Architectures and Governance

Predominantly under the responsibility of product teams:

• Business Use Cases, Domain Logic, Domain APIs
• User Experience, Domain Knowledge
• RAG Collections, Agent Workflows
• Product SLOs, Business Metrics, and Business Outcomes

This separation resolves a recurring conflict. The CoE does not need to have deep knowledge about fraud, credit, HR, logistics, or customer service. But it also does not make sense for each area to reinvent LLM observability, agent security, model routing, or evaluation framework.

The AI Gateway is just one piece. The real product is the AI Platform

A corporate AI platform with an adequate level of maturity has multiple shared services that go beyond the gateway. A complete AI PLATFORM includes:

• AI Gateway and Model Router
• Registries (Model, Prompt, Agent)
• Evaluation Platform and Experiment Tracking
• Guardrails and Semantic Cache
• Observability and Cost Management
• Feature / Context Services and RAG Services
• Policy Engine, Secrets Management, and Audit

This platform transforms AI from a collection of SDKs and point integrations into a corporate capability.

But there is a fundamental principle that many organizations overlook:

AI Platform is also a product

The platform is not successful because it exists. It is successful when teams prefer to use it rather than build their own solutions. This requires treating it with the same discipline as any product:

• Platform Product Owner and Roadmap
• Internal Customers and Adoption Metrics
• Developer NPS and platform SLOs
• Backward Compatibility and Deprecation Policies
• Versioned APIs and Platform Documentation

Without product thinking, there is a predictable antipattern: the CoE becomes an internal framework factory that no one adopts.

Developer Experience is what connects platform to adoption

A mature enterprise AI Platform needs to explicitly invest in Developer Experience (DX). The catalog of shared services only generates value if squads can consume it with low friction:

• Developer Portal and CLI
• SDKs (Platform, Observability, Agent, RAG, Evaluation)
• Templates, Scaffolding, and CI/CD Templates
• Golden Paths and Reference Implementations
• Policy Libraries

This materializes what platform engineering calls the golden path: the path that delivers compliance, security, and observability by default, without requiring each team to figure out how to implement each of these dimensions.

The developer executes:

ai-platform create rag-service

And natively receives:

• ✅ Tracing and Observability
• ✅ Authentication and Secrets
• ✅ Evaluation and Guardrails
• ✅ Cost tracking and Audit
• ✅ CI/CD and Model routing

The developer remains free to solve the business problem but does not need to rebuild the foundation of security, observability, and governance.

RAG requires dedicated platform and governance

Another recurring antipattern is treating Retrieval-Augmented Generation as a trivial implementation of documents ➔ embedding ➔ vector database ➔ LLM.

In production, the complexity is considerably greater. An enterprise-grade RAG platform needs to handle, at a minimum:

"

RAG Pipeline: Sources ➔ Connectors ➔ Parsing ➔ Normalization ➔ PII Detection / Data Classification ➔ Chunking ➔ Embedding ➔ Indexing ➔ ACL Propagation ➔ Retrieval ➔ Reranking ➔ Context Assembly ➔ Generation ➔ Evaluation

And it needs to be able to answer critical operational questions:

• Which version of the document produced a certain response?
• Which version of the embedding model was active at that time?
• Which chunking algorithm was used?
• Which index was queried and which documents were retrieved?
• What was the retrieval score for each document?
• Did the user have authorization to access the retrieved documents?
• Was the retrieved content effectively used in generating the response?
• Is it possible to reproduce the inputs, context, versions, and conditions of that execution for investigation and behavior comparison?

These properties (traceability, reproducibility, access control) need to be part of the architecture, not added later.

The OWASP Top 10 for LLM Applications (2025) classifies Vector and Embedding Weaknesses as a separate risk category, alongside Prompt Injection, Sensitive Information Disclosure, Excessive Agency, and Unbounded Consumption [⁴].

RAG is not just a retrieval technique. It introduces a new attack surface and a new dimension of governance.

Evaluation needs to be infrastructure, not manual verification

One of the cle