Model APIs should be treated as external, replaceable resources

An eight-factor AI application treats language models as attached resources, accessed through abstraction layers that enable provider flexibility. A model provider is any service offering language model capabilities, whether cloud-based APIs or local deployments. The interface to these providers should be clearly defined and independent of specific implementations.

Good provider abstraction has several key characteristics:

  • No provider-specific code in core business logic
  • Unified interface for model interactions
  • Consistent handling of provider capabilities and limitations
  • Clear separation between model access and model use

The application should require no code changes to switch between providers:

# Configuration (environment or config file)
MODEL_PROVIDER=openai
MODEL_CONFIG={
    "model": "gpt-4",
    "temperature": 0.7
}

# Different providers, same interface
providers = {
    "openai": OpenAIProvider,
    "anthropic": AnthropicProvider,
    "local": LocalModelProvider
}

provider = providers[MODEL_PROVIDER](MODEL_CONFIG)

Bad practice - tight coupling to provider:

import openai

def generate_text(prompt):
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7
    )
    return response.choices[0].message.content

Good practice - provider abstraction:

class BaseModelProvider:
    def generate(self, 
                prompt: str,
                context: Optional[List[Message]] = None,
                **kwargs) -> GenerationResult:
        """Generate text from the model."""
        raise NotImplementedError

class OpenAIProvider(BaseModelProvider):
    def generate(self, prompt, context=None, **kwargs):
        messages = self._format_messages(prompt, context)
        response = self.client.chat.completions.create(
            model=self.config.model,
            messages=messages,
            **self._translate_params(kwargs)
        )
        return GenerationResult.from_openai(response)

class AnthropicProvider(BaseModelProvider):
    def generate(self, prompt, context=None, **kwargs):
        messages = self._format_messages(prompt, context)
        response = anthropic.messages.create(
            model=self.config.model,
            messages=messages,
            **self._translate_params(kwargs)
        )
        return GenerationResult.from_anthropic(response)

Provider abstraction should handle:

  • Authentication and credentials management
  • Rate limiting and quotas
  • Error handling and retries
  • Response parsing and normalization
  • Parameter translation between providers
  • Capability detection and fallbacks

A proper abstraction layer enables:

  • Seamless provider switching
  • Multi-provider strategies
  • A/B testing between providers
  • Graceful fallback handling
  • Cost optimization
  • Provider-specific optimizations behind a common interface

Providers should be treated as attached resources:

  • Provider configuration should be external to application code
  • Credentials should be managed through environment variables
  • Provider health should be monitored
  • Usage should be tracked and logged
  • Costs should be monitored and controlled

Some applications may use multiple providers simultaneously:

class MultiProvider(BaseModelProvider):
    def generate(self, prompt, **kwargs):
        for provider in self.providers:
            try:
                return provider.generate(prompt, **kwargs)
            except ProviderError:
                continue
        raise AllProvidersFailedError()

This pattern enables:

  • Load balancing between providers
  • Cost-based routing
  • Capability-based provider selection
  • High availability through redundancy
  • Progressive enhancement based on provider capabilities

The abstraction layer should be treated as critical infrastructure:

  • All provider interactions should be traced
  • Performance metrics should be collected
  • Costs should be tracked per provider
  • Errors should be monitored and analyzed
  • Usage patterns should be studied for optimization