1.Introduction

Building a proof of concept with Artificial Intelligence has become relatively simple.

A model can be loaded, connected to a dataset, and used to classify information, generate responses, query data, produce reports, or execute some automated workflow.

In an experimental environment, an architecture like this may be sufficient:

Interface
   ↓
Data
   ↓
Model
   ↓
Processing
   ↓
Result

The challenge arises when this solution needs to stop being an experiment and start operating as part of a corporate platform.

At this moment, requirements come into play that typically do not appear during the prototype construction:

• authentication;
• authorization;
• data governance;
• security;
• APIs;
• contracts;
• observability;
• auditing;
• persistence;
• scalability;
• isolation;
• cost control;
• versioning;
• model evaluation;
• CI/CD;
• operation;
• failure recovery.

The architecture needs, therefore, to evolve from a functional AI pipeline to an AI-oriented software system.

This difference is fundamental.

A pipeline primarily answers the question:

"

How to execute a certain AI capability?

A production architecture needs to answer:

"

How to transform this capability into a secure, governable, testable, observable, and sustainable system?

This article presents a reference architecture to guide this evolution.

2. The problem this architecture solves

AI applications typically start focused on four responsibilities:

Input
  ↓
Preparation
  ↓
Inference
  ↓
Output

This structure is valid.

In fact, the separation between data loading, model management, inference execution, and result presentation represents an important first step in engineering.

The problem arises when these responsibilities also start to concentrate on:

business rules
authorization
infrastructure
credentials
persistence
error handling
access control
governance
orchestration
telemetry

The result tends to be a strongly coupled application.

For example:

UI
 ↓
loads model
 ↓
fetches data
 ↓
executes inference
 ↓
applies rule
 ↓
writes file
 ↓
generates report

This design may work well for a local experiment.

But it creates difficulties when we need to:

• change the interface;
• provide an API;
• change the model;
• switch the data source;
• run in the cloud;
• process large volumes;
• add authentication;
• execute asynchronous tasks;
• audit decisions;
• implement multiple agents;
• test parts in isolation.

The reference architecture seeks to decouple these responsibilities.

3. From experimental pipeline to production system

The evolution typically goes through stages.

Experiment
    ↓
Structured Pipeline
    ↓
Modular Application
    ↓
API and Contracts
    ↓
Governance and Security
    ↓
Observability and Auditing
    ↓
Operational Automation
    ↓
AI Platform

These stages do not need to signify different systems.

The important thing is to realize that architecture maturity is not equivalent to the number of services.

A solution can remain as a single deploy and still have a good architecture.

What needs to exist are clear boundaries between responsibilities.

4. Central architectural principle

The main recommendation of this architecture is:

"

Separate probabilistic intelligence from deterministic control.

AI models are probabilistic components.

They excel at:

• interpretation;
• classification;
• planning;
• generation;
• extraction;
• contextualization;
• synthesis.

But they should not be the authority responsible for:

• authentication;
• authorization;
• IAM;
• access control;
• corporate policies;
• financial limits;
• integrity;
• auditing;
• security;
• privileged execution.

The correct relationship should be:

AI
 ↓
proposes
 ↓
Software
 ↓
validates
 ↓
authorizes
 ↓
executes

And not:

AI
 ↓
decides
 ↓
executes

This separation should exist regardless of whether we use classification, RAG, agents, SQL, tools, APIs, or other mechanisms.

5. Reference architecture

An overview can be organized as follows:

┌──────────────────────────────────────┐
│          Experience Layer            │
│                                      │
│   Web / Mobile / Chat / API Client   │
└──────────────────┬───────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│             API / BFF                │
│                                      │
│ Auth · Validation · Rate Limit       │
│ Contracts · Streaming · Errors       │
└──────────────────┬───────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│          Application Layer           │
│                                      │
│ Use Cases · Coordination · Lifecycle │
└──────────────┬─────────────┬─────────┘
               │             │
               ▼             ▼
┌────────────────────┐ ┌──────────────────────┐
│    Domain Layer    │ │       AI Layer       │
│                    │ │                      │
│ Policies           │ │ Models               │
│ Rules              │ │ Prompts              │
│ Invariants         │ │ Chains               │
│ Constraints        │ │ Agents               │
└────────────────────┘ │ Workflows            │
                       │ Evaluations          │
                       └──────────┬───────────┘
                                  │
                                  ▼
┌──────────────────────────────────────────────┐
│          Integration / Data Layer            │
│                                              │
│ Data Sources · APIs · Tools · Vector Store   │
│ Warehouse · Model Providers · Storage        │
└──────────────────────┬───────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────┐
│             Infrastructure Layer             │
│                                              │
│ Compute · Identity · Secrets · Persistence   │
│ Messaging · Cache · Networking · Storage     │
└──────────────────────────────────────────────┘

There are also transversal capabilities:

Security

Identity

Governance

Observability

Audit

LLMOps

Cost Management

CI/CD

These capabilities do not belong exclusively to a single layer.

They span the entire architecture.

6. Experience Layer

The experience layer represents the channels used by users or consuming systems.

It can include:

• web applications;
• mobile applications;
• conversational interfaces;
• integrations;
• public or internal APIs.

Its responsibility is interaction.

It should not concentrate critical business rules.

It should also not have direct access to:

databases
Data Warehouse
model providers
credentials
service accounts
secrets
infrastructure

The recommended flow is:

Client
  ↓
API / BFF

and not:

Client
  ↓
LLM

or

Client
  ↓
Database

7. API / BFF

The API represents the boundary between consumers and the system.

Among its responsibilities are:

• HTTP contracts;
• authentication;
• input validation;
• rate limiting;
• idempotency;
• correlation IDs;
• versioning;
• streaming;
• polling;
• error handling;
• pagination;
• cancellation.

An API should not become a business layer.

The role of the API is to transform:

HTTP Request

into:

Application Command

and transform the result back into an external contract.

8. Application Layer

The Application Layer organizes use cases.

For example:

AnalyzeData

RunInference

GenerateReport

AskQuestion

ExecuteAnalysis

GetExecution

CancelExecution

It defines the necessary sequence to perform an operation.

Example:

Request
   ↓
Validate Context
   ↓
Authorize
   ↓
Load Resources
   ↓
Execute AI Workflow
   ↓
Persist State
   ↓
Audit
   ↓
Return Result

The application layer knows the process.

But it should depend on abstractions to access:

• models;
• databases;
• APIs;
• Data Warehouse;
• queues;
• persistence;
• external services.

9. Domain Layer

The Domain represents the stable behavior of the system.

It concentrates:

• rules;
• policies;
• invariants;
• constraints;
• allowed states;
• deterministic decisions.

A healthy domain should not depend directly on:

FastAPI

React

LangChain

LangGraph

cloud SDK

database SDK

persistence framework

This allows critical rules to be tested without infrastructure.

The desired direction is:

Infrastructure
      ↓
Application
      ↓
Domain

And not the other way around.

10. AI Layer

The AI layer concentrates what truly depends on models.

For example:

Model Adapters

Prompt Registry

Structured Output

Chains

Agents

Tools

Workflows

Retrieval

Evaluations

This isolation has two main advantages.

First, it prevents AI frameworks from contaminating the entire system.

Second, it makes it easier to swap components.

For example:

Model A
   ↓

Application

   ↑
Model B

Or:

Local Model
    ↓

Application

    ↑
Cloud Model

The application should not need to be rewritten because of the choice of a provider.

11. Model Management

Model management should be treated as an explicit responsibility.

It can involve:

• loading;
• initialization;
• authentication with the provider;
• configuration;
• model selection;
• versioning;
• quantization;
• device management;
• fallback;
• health check.

In local inference:

Runtime
  ↓
Hardware Detection
  ↓
Model Loading
  ↓
Model Ready

In remote inference:

Runtime
  ↓
Provider Configuration
  ↓
Authentication
  ↓
Model Endpoint

The upper layer should not need to know these differences.

The ideal is to work through an abstraction:

ModelPort

with specific implementations.

12. Local versus Cloud

The architecture should not necessarily assume that models will be executed in the cloud.

There are scenarios where local inference may be interesting:

• privacy;
• absence of external data transmission;
• cost control;
• low local latency;
• disconnected environments;
• specialized models;
• edge computing.

On the other hand, managed providers offer:

• elasticity;
• larger models;
• less operational management;
• integration with cloud services.

Therefore:

"

Local or cloud should be an implementation decision, not a structural decision of the application.

The application should depend on an inference contract, not the provider.

13. Data Layer

Data also needs to be treated as abstractions.

A solution can consume:

files

datasets

APIs

Data Warehouse

Data Lake

relational databases

Vector Databases

Streams

A DatasetLoader or DataProvider can encapsulate these differences.

For example:

Application
     ↓
Data Port
     ↓
┌────────────┬───────────────┬─────────────┐
│ File       │ Data Warehouse│ External API│
└────────────┴───────────────┴─────────────┘

This avoids application rules being coupled to a specific source.

14. Data Preparation

Data preparation is an independent step.

It can involve:

• field selection;
• cleaning;
• normalization;
• validation;
• segmentation;
• enrichment;
• masking;
• chunking;
• transformation.

In simple pipelines:

Raw Dataset
    ↓
Validation
    ↓
Preparation
    ↓
Inference

In corporate systems:

Data Source
    ↓
Governance
    ↓
Authorization
    ↓
Preparation
    ↓
AI Workflow

AI should