Appendix B: AI Stack Comparison Reference
This appendix provides a structured comparison of the major technology choices that B2B SaaS companies face when building an AI stack for the use cases described in this book. It is not a comprehensive survey of every available tool or platform. It is a practitioner’s reference for the decisions that matter most in the context of the sprint-based AI transformation approach this book describes.
The comparisons are organized by stack layer. Each layer covers the primary choices, the key criteria for selection in the B2B SaaS context, and the tradeoffs that are most consequential for programs of the scale and complexity this book addresses.
All cost estimates are approximate and based on typical usage patterns for a mid-size B2B SaaS company in the 50 to 200 employee range. Actual costs will vary significantly based on volume, model selection, and usage patterns.
Layer 1: Language Model Provider
The language model is the core generative capability in most AI systems described in this book. The selection criteria in the B2B SaaS context are: output quality for the specific task type, API reliability and latency, pricing at production volume, and data handling commitments.
OpenAI (GPT-4o, GPT-4o mini)
Strengths: Consistently high output quality across a wide range of task types; well-documented API with extensive community resources; the largest ecosystem of integration tooling and documentation; strong enterprise data handling commitments available through the Enterprise tier.
Weaknesses: Pricing is higher than some alternatives at volume; subject to capacity constraints during peak periods, though this has improved significantly; vendor lock-in risk is real given the depth of integration most production systems develop.
Best for: Organizations that want proven quality with minimal integration friction, are willing to pay a quality premium, and are building their first production AI systems where the debugging support of a large ecosystem is valuable.
Approximate cost at mid-scale (100K API calls/month, mixed short/long requests): $400 to $1,200 per month depending on model tier and context length.
Anthropic (Claude 3.5 Sonnet, Claude Haiku)
Strengths: Strong performance on instruction-following tasks; Claude’s extended context window is valuable for use cases involving long documents; competitive pricing at volume; strong content safety controls that reduce the risk of harmful outputs in customer-facing applications.
Weaknesses: Smaller ecosystem than OpenAI; some integration tooling assumes OpenAI API format and requires adaptation; API availability has been more variable historically, though this has improved.
Best for: Organizations where content safety in customer-facing outputs is a priority; use cases that involve long document processing where extended context windows reduce the need for complex chunking strategies; organizations that want to reduce single-vendor dependency by running a multi-provider architecture.
Approximate cost at mid-scale: $350 to $900 per month.
Google (Gemini 1.5 Pro, Gemini Flash)
Strengths: Competitive pricing, particularly at volume; strong multimodal capability if use cases involve image or document inputs; tight integration with Google Cloud infrastructure if that is the existing cloud provider; Gemini Flash provides very low latency for simpler tasks.
Weaknesses: API and developer tooling have been less mature than OpenAI or Anthropic; model behavior can differ from other providers in ways that require re-evaluation of prompts developed on other platforms; integration with existing developer workflows can require additional configuration.
Best for: Organizations already on Google Cloud infrastructure; use cases that involve multimodal inputs; high-volume, latency-sensitive applications where Gemini Flash’s speed-cost tradeoff is advantageous.
Approximate cost at mid-scale: $200 to $700 per month.
AWS Bedrock (multi-model managed service)
Strengths: Access to multiple model providers (Anthropic, Meta, Mistral, Amazon Titan) through a single API; native integration with AWS infrastructure; unified billing and IAM access controls; no data retention by default, which simplifies data handling compliance.
Weaknesses: Adds a layer of abstraction that can introduce latency; not all model capabilities are available through Bedrock that are available through direct provider APIs; cost includes Bedrock overhead on top of underlying model costs; some provider models are available only with regional constraints.
Best for: Organizations on AWS infrastructure that want multi-provider flexibility without managing multiple API integrations; organizations where unified cloud billing and IAM controls are important for compliance or security posture.
Approximate cost at mid-scale: 10 to 20% higher than direct provider access for equivalent model usage, offset by infrastructure consolidation savings.
Selection Recommendation for Sprint Phase
For most B2B SaaS companies beginning their first sprint, OpenAI or Anthropic as a primary provider with a secondary provider available for fallback represents the lowest-risk configuration. The quality difference between the top providers is smaller than it was eighteen months ago; the primary selection criteria should be fit for the specific task type (instruction-following vs. document analysis vs. code generation), data handling requirements, and existing team familiarity.
Avoid building provider lock-in into the architecture: abstract the model call behind an interface layer that allows provider switching without significant refactoring, even if you expect to use one provider for the foreseeable future.
Layer 2: Retrieval Infrastructure (Vector Database)
The vector database stores the embeddings that enable semantic search over the organization’s document corpus. The selection criteria are: query latency, ease of integration with existing infrastructure, operational overhead, and cost at the data volumes the sprint use cases require.
Pinecone
Strengths: Fully managed; minimal operational overhead; good performance at scale; well-documented integration with LangChain and other orchestration frameworks; production-grade reliability.
Weaknesses: Cost scales significantly at high vector counts; less flexibility than open-source alternatives for organizations with specific infrastructure requirements; data lives outside the organization’s infrastructure, which requires review of data handling implications.
Best for: Organizations that want a production-ready vector database with minimal operational investment and are willing to accept the cost premium for managed service convenience.
Approximate cost: $70 to $300 per month for sprint-scale usage (1 to 5 million vectors).
pgvector (PostgreSQL extension)
Strengths: Runs inside an existing PostgreSQL instance, eliminating the need for separate infrastructure; zero additional cost if PostgreSQL is already in the stack; data stays within the organization’s existing database infrastructure, simplifying data handling compliance; well-supported in most cloud providers’ managed PostgreSQL offerings.
Weaknesses: Query performance degrades relative to dedicated vector databases at large vector counts (above 1 to 2 million vectors depending on hardware); operational tuning requires more effort than managed alternatives; index types (IVFFlat, HNSW) require configuration decisions that affect the quality-performance tradeoff.
Best for: Organizations with an existing PostgreSQL infrastructure that want to minimize stack complexity for sprint-phase work; use cases with moderate vector counts where PostgreSQL’s query performance is adequate; organizations where keeping data within existing infrastructure is a compliance or security requirement.
Approximate cost: Infrastructure cost only (no additional licensing); typically adds less than $50 per month to existing database infrastructure costs at sprint scale.
Weaviate
Strengths: Open source with cloud-hosted option; supports hybrid search (keyword + semantic) natively, which is valuable for use cases where keyword matching and semantic similarity are both relevant; good performance at medium scale; can be self-hosted on existing infrastructure.
Weaknesses: More complex configuration than simpler alternatives; self-hosted option requires operational investment; the hybrid search capability adds value only for use cases that genuinely need it.
Best for: Use cases where pure semantic search produces poor results because the query language is highly specific (technical identifiers, product codes, proper nouns); organizations that want open-source flexibility with a managed hosting option.
Approximate cost (cloud hosted): $50 to $200 per month at sprint scale.
Selection Recommendation for Sprint Phase
For most B2B SaaS sprints, pgvector (if PostgreSQL is already in the stack) or Pinecone (if a managed service is preferred) are the right starting points. The performance difference between options is not material at sprint-scale data volumes, and the operational simplicity of starting with infrastructure you already operate (pgvector) or a fully managed service (Pinecone) reduces the time spent on infrastructure work relative to evaluation and improvement work.
Avoid over-engineering the retrieval infrastructure in the sprint phase. The vector database is a component, not the system, and optimizing it before the retrieval strategy is validated is a common source of sprint phase delays.
Layer 3: Embedding Model
The embedding model converts text into the vector representations that the retrieval system uses for semantic search. The primary selection criteria are: embedding quality for the domain and language of the corpus, cost at the volume the use case requires, and latency for real-time embedding generation.
OpenAI text-embedding-3-small / text-embedding-3-large
Strengths: Strong general-purpose embedding quality; well-integrated with the OpenAI API if it is already in use; text-embedding-3-small provides a good quality-cost tradeoff for most B2B SaaS use cases.
Weaknesses: Cost adds up at high embedding volumes; vendor dependency if already using OpenAI for generation.
Approximate cost: text-embedding-3-small: $0.02 per million tokens; text-embedding-3-large: $0.13 per million tokens.
Cohere Embed
Strengths: Strong multilingual embedding support, which is relevant for B2B SaaS companies with international customer bases; competitive pricing; provider diversification if OpenAI or Anthropic is already the primary generation provider.
Weaknesses: Smaller ecosystem than OpenAI embeddings; requires managing a second provider relationship.
Approximate cost: $0.10 per million tokens (Embed v3).
Open-source (BGE, E5, Sentence-Transformers)
Strengths: Zero per-token cost after infrastructure; can be fine-tuned on domain-specific data; data does not leave the organization’s infrastructure.
Weaknesses: Requires hosting and operational overhead; fine-tuning requires labeled data and ML expertise; out-of-the-box quality is generally below top API-based options for most domains without fine-tuning.
Best for: Organizations with high embedding volume where API costs are significant; organizations with domain-specific language that would benefit from fine-tuning; organizations with strict data residency requirements.
Selection Recommendation for Sprint Phase
OpenAI text-embedding-3-small provides a good quality-cost tradeoff for most sprint use cases. Evaluate domain fit by running a sample of representative queries against the embedding model during the sprint’s technical foundation phase; if semantic search quality is poor even with a well-designed prompt, consider whether fine-tuning or a different embedding model is needed before investing further in the retrieval system.
Layer 4: Orchestration Framework
The orchestration framework manages the sequence of operations in an AI system: context retrieval, prompt construction, model calls, output parsing, and error handling. The primary selection criteria are: ease of integration with existing engineering practices, quality of debugging and observability tooling, and community support for the use case types being built.
LangChain
Strengths: Large community; extensive documentation; built-in integrations for most vector databases and model providers; good support for RAG (retrieval-augmented generation) patterns that are central to most B2B SaaS AI use cases.
Weaknesses: Can introduce unnecessary abstraction overhead for simple use cases; frequent API changes in the library have historically caused upgrade friction; debugging complex chains can be difficult without dedicated observability tooling.
Best for: Teams that want a high-level framework with broad integration support and are willing to accept some abstraction overhead; first-time AI system builders who benefit from the opinionated structure LangChain provides.
Direct API integration (no framework)
Strengths: Complete control over the system’s behavior; no framework-introduced abstraction overhead; easier to debug and modify; no dependency on a third-party library’s API stability.
Weaknesses: Requires writing more boilerplate; does not provide built-in integration with vector databases or model providers; each integration must be built and maintained.
Best for: Teams with strong engineering discipline that prefer explicit control over framework-provided abstraction; use cases that are simple enough that framework overhead is not justified; organizations where long-term maintainability without framework dependency is a priority.
LlamaIndex
Strengths: Strong focus on document ingestion and retrieval, which makes it well-suited for RAG use cases; good support for complex document structures; active development with useful primitives for evaluation and retrieval experimentation.
Weaknesses: More opinionated than direct API integration; some overlap in functionality with LangChain creates a choice that is not clearly differentiated; community is smaller than LangChain’s.
Best for: Use cases centered on document retrieval and question answering where the indexing and retrieval primitives LlamaIndex provides reduce development time significantly.
Selection Recommendation for Sprint Phase
For the sprint phase, the engineering team should use whatever approach minimizes the time to a working evaluation, not the approach that would be optimal for a mature production system. For most teams, that means LangChain for the first sprint if the team has familiarity with it, or direct API integration if the team values explicitness and the use case is architecturally straightforward. Avoid spending sprint time evaluating frameworks; pick one, build the system, and refactor if the choice proves limiting in the mid-term phase.
Layer 5: Evaluation Tooling
Evaluation tooling supports the systematic assessment of AI system quality that the sprint evaluation framework requires. The primary selection criteria are: support for the evaluation methodology described in Chapter 7, ease of integration with existing development workflows, and cost.
Custom evaluation scripts (Python)
For most B2B SaaS sprints, a set of Python scripts that run the evaluation benchmark against the production system, calculate the metrics in the evaluation framework, and output results in a format that the team reviews regularly is sufficient. Custom scripts have zero licensing cost, are fully controllable, and can be version-controlled alongside the system code.
The limitation of custom scripts is that they require the team to build and maintain the tooling rather than using established primitives. For sprint-phase evaluation, this is an acceptable tradeoff; for mid-term and long-term evaluation programs, investing in more structured tooling is worth considering.
Arize AI / Weights and Biases / LangSmith
Dedicated evaluation and observability platforms provide structured tracking of evaluation results, automatic detection of performance regression, and tooling for managing evaluation datasets. These platforms are most valuable in the mid-term and long-term phases when the team is operating multiple AI systems simultaneously and manual evaluation tracking becomes inadequate.
For the sprint phase, the overhead of configuring and integrating a dedicated evaluation platform is typically not justified by the benefit relative to custom scripts. Consider these platforms as a planned investment for the month-six to month-nine period when the evaluation infrastructure needs to scale beyond what custom scripts efficiently support.
Quick Reference: Sprint-Phase Stack Recommendation
For a typical B2B SaaS sprint targeting retrieval-augmented generation use cases, the following stack configuration minimizes operational overhead while providing adequate capability for a production-quality first system:
- Generation model: OpenAI GPT-4o mini (cost-quality tradeoff) with GPT-4o available for complex cases
- Embedding model: OpenAI text-embedding-3-small
- Vector database: pgvector (if PostgreSQL is in the existing stack) or Pinecone (if managed service is preferred)
- Orchestration: Direct API integration for simple architectures; LangChain for complex retrieval chains
- Evaluation: Custom Python scripts against a defined benchmark set, run at least weekly
Estimated total AI infrastructure cost for a well-scoped sprint use case: $400 to $1,500 per month, varying primarily with generation model selection and call volume. This estimate excludes infrastructure costs for hosting the vector database if self-hosted, and excludes any fine-tuning compute costs.