M.A.P.P.E.D. Framework

AI projects are some of the most complex-dense software projects imaginable. Even seemingly small use cases bring organizational challenges that demand multi-disciplinary teams, leadership support, third party integrations, security audits, and cross team alignment.

It doesn’t get any easier on the technical side either. You still need all the bells and whistles of a classical software project: from full-stack dev capabilities, to mature DevOps, quality assurance, and support. On top of that come the unique quirks of AI. These include but are not limited to modeling black box behavior, navigating a constantly evolving ecosystem, and massive lack of best practices.

At v9Labs, we strive to deliver the same level of consistency and reliability when navigating client projects that we strive for in our AI implementations. To make that possible, we created the M.A.P.P.E.D. framework as a tool for us to keep track of all of the mission critical dimensions for a successful AI project, and to have a standardized way to measure progress and steer projects proactively.

Models

Our first concern is ensuring timely access to the latest models. Many organizations have vetting systems in place that significantly delay the adoption of newer models. While recent model releases have not drastically expanded what they can do, the how in terms of reliability and performance is an entirely different ball game. If you’re still using GPT-4 or Sonnet-4, it’s like you are trying to win the olympics with your local high-school team.

Once timely access is facilitated, we evaluate task-relevant capabilities, cost, and performance. Especially with reasoning models, the cost per token is not just tied to the amount of input and output tokens, but is mainly driven by the intermediate reasoning tokens (see “Cost to Run AI Index” by Artificial Analysis). The true cost can only be forecasted with task-specific benchmarks that we co-create with our clients.

Core metrics:

• Capabilities
• Cost
• Speed
• Security

Architecture

Advancing model capabilities are only as valuable as the architecture of an AI system is able to adapt to them. It’s rarely as simple as swapping the LLM endpoint: different models have distinct strengths and weaknesses, that influence how they fit into your overall system.

We help clients design and implement an architecture, that ensures their systems are quick and easy to adapt to changes in the ecosystem.

The other key concern in terms of architecture is the quickly emerging complexity of AI projects. Between experimentation, deployment, and adaption to multiple use cases, keeping complexity in check is paramount to the quality, reliability, and speed of iteration. At v9Labs, we argue that “configuration is king” (see related article): you need a modular, configuration-driven system where models, processing steps, and technologies can be swapped or tuned without having to rewrite major parts of your code base.

Deliberate architecture design is not a technical luxury, it is a necessity for projects that don’t want to see their progress grind to an excruciating halt halfway between prototype and production.

Core metrics:

• Feature cycle time
• Issue resolution time
• Service availability

Prompts & Tools

When we look at prompts and tools, it is all about ensuring the LLM inputs are designed in a way to reduce ambiguity and increase reliability. Models have become much better at handling complex prompts and multi-faceted instructions. This has led many teams to treat prompts as a secondary concern. The reality is that all benchmarks across the board testify that smaller instruction sets still yield better results with greater reliability.

Tools enable your agents to perform a range of tasks, from web search to working on third party systems. Many organizations that go “all in” on agentic AI often hit a wall when every API call and interaction is encoded as a separate tool. Giving the AI as little as 20 tools to choose from can often lead to tool confusion and result in the wrong tools being called with the wrong parameters.

At v9Labs, we’ve developed proven patterns for delegation, routing, and composition, to break down complex instructions and tool calls into nodular building blocks that are accurate, reliable, and testable.

Core metrics:

• Success rate
• Context size
• Instruction complexity

Process

AI projects are inherently multi-disciplinary. AI is there to support a core business function, which requires deep domain expertise to identify, implement, and evaluate. Implementations often interface multiple third party systems, with ownership dispersed throughout your organization, demanding different approval and integration requirements.

Without clear alignment on process and priorities, then even the most promising AI initiative will eventually be dropped before ever seeing the light of day.

Through our implementation experience, we provide guidance and foresight around expectations, commitments, and organizational requirements. We ensure challenges are resolved proactively, rather than firefighting reactively when misalignment causes delays and frustration.

Core metrics:

• Cross-team alignment
• Decision speed

Evaluations

Evaluations (“evals”) are a structured set of questions or tasks that your AI system must perform, each with an expected outcome (“ground truth”). Actual outcomes are then compared with the ground truth.

Evals are where everything converges:

• The process to ensure that your evals are up-to-date and correctly reflect the business domain
• The architecture to allow for easy testing and documentation of experiments
• The prompts and tools that are structured to keep black boxes small and outcomes verifiable

Evals are your goal post and report card in one. They are the key differentiators between systems that degrade over time and the ones that improve with every iteration. Their quality is measured by how well they reflect real-world user experience. If evals are “green” but users are frustrated, they’re not measuring what matters.

Designing an efficient and effective eval process is one of the biggest hurdles in AI projects, and one that comes up repeatedly over the the project’s lifetime. At v9Labs, we ensure that evals become your launchpad for reliable releases, continued improvements, and rapid iterations, rather than bureaucratic obstacles to overcome.

Core metrics:

• Coverage
• Accuracy
• Reliability
• Alignment

Data

Saving the big one for last. Data is the foundation for everything: LLMs can only act on what is contained in their context window. Bloat the window, and the critical information is overlooked and prices skyrocket. Miss the correct information, and the agent is prone to hallucinate.

Data can include documents from your sharepoint, human feedback, user data, or responses from third-party systems, the list just goes on. Data is not just about “what” is put into the context window, but also the “how”. Full documents may cause bloat, excerpts might miss context. On top of that come challenges around access permissions, data privacy and security concerns, compatibility, versioning, and many more.

Our AI Toolkit provides some of the core infrastructure components that focus on data retrieval, so your team can focus on engineering the context, rather than worrying about the glue that holds it all together.

At v9Labs, we’ve built a catalogue of battle-tested patterns across a range of use cases on how to engineer context in a way that it is cost efficient and effective in performing the desired task.

Core metrics:

• Retrieval precision
• Relevancy
• Context utilization

How we use the M.A.P.P.E.D. framework

When we start a project, we will map out the current state along each dimension. This creates a shared understanding of where things stand, clarifies next steps, and highlights areas where we can deliver the most value quickly, without losing sight of the long term goals.