AI Product Management Glossary (2026)

A controlled experiment that compares two variants to estimate the causal impact of a change on a target metric.

Specific, testable conditions that define when a user story or requirement is complete and acceptable.

The mechanisms an agent uses to remember and reuse past interactions or facts across turns and sessions.

Coordinating how agents, tools, and models are invoked, sequenced, and supervised within a product.

A product flow where an agent chains reasoning, tool use, and checkpoints to achieve a user goal with minimal hand-holding.

A system where an LLM plans and executes actions toward a goal using tools, memory, and feedback loops.

How you split documents or histories into pieces for indexing so retrieval balances relevance, completeness, and speed.

Showing which sources support an answer, with links or identifiers users can verify.

Screening inputs and outputs for toxicity, abuse, violence, or other policy-violating content.

Reducing and reshaping context (summaries, salience scoring, deduplication) so key facts fit within token and latency budgets.

Context engineering is the system-level discipline of deciding what information a model may use at runtime, which sources are eligible, how they are selected, ordered, scoped, transformed, and refreshed, and what happens when sources conflict, go stale, exceed budget, or cross permission boundaries. Prompt engineering designs the instructions; context engineering designs the whole information environment those instructions run in.

The maximum token length a model can attend to at once across input and output.

Total variable cost (tokens, tool calls, infra) to complete a user task with your AI feature.

Sensitive information being exposed to unauthorized users or external systems through model inputs, outputs, or logs.

Tracking the origin, transformations, and permissions of data used for training, retrieval, or responses.

Vector representations of text or data that capture semantic meaning, enabling similarity search, clustering, and ranking.

A repeatable pipeline that scores model or agent outputs against test cases and business metrics before and after changes.

Few-shot examples are input-output demonstrations supplied in model context so the model can infer desired task, style, and format patterns without changing model weights. Each exemplar demonstrates one behavior or failure-mode response, a common case, an ambiguous input, missing data, a refusal, a formatting rule, or a tool-use decision, with an explicit input/output boundary and a coverage role. They are runtime behavior controls with token, latency, maintenance, and privacy costs, proven by eval comparison and ablation rather than by count: they demonstrate how to answer, they do not supply fresh facts (retrieval), enforce machine contracts (structured outputs), retrain the model (fine-tuning), or substitute for system-level safety controls.

An API pattern where the LLM returns a structured call to a specified function, often validated and executed by your code.

A curated collection of test cases with trusted answers used to judge model quality over time.

Responses that are explicitly supported by retrieved or verifiable sources, reducing hallucination risk.

Policies and technical controls that constrain what an AI can say or do, preventing harmful or out-of-scope behavior.

How often a model produces unsupported or incorrect facts relative to total responses.

A crafted input designed to bypass safety constraints and make the model produce disallowed content or actions.

A framework that explains why users hire a product to make progress in a specific situation.

Keeping the information an AI feature relies on up to date, and detecting when stale data harms quality.

A measurable indicator used to track progress toward a specific product or business objective.

The maximum response time you can spend across model calls, tools, and orchestration while meeting UX and business goals.

Using a model to score another model’s outputs against criteria, often faster and cheaper than human labeling.

Infrastructure to run, monitor, and resume agents that operate over minutes to hours with checkpoints and persistence.

A minimum viable product (MVP) is the smallest implementation that lets real users complete a meaningful task, so a team can observe what happens and decide what to do next.

A protocol for connecting models to external tools and data sources in a standardized, secure way.

A setup where multiple specialized agents collaborate or compete to solve a task, often with coordination rules.

The single metric that best captures long-term customer value delivered by your product.

Quality tests run on recorded or synthetic data without live users, giving fast, safe feedback on changes.

A goal setting framework that pairs a qualitative objective with measurable key results.

Live experiments that measure model changes with real user traffic, often via A/B tests or shadow deployments.

A planning distinction where outputs are shipped artifacts and outcomes are measurable user or business changes caused by those artifacts.

Detecting and removing personally identifiable information from inputs, outputs, or stored data to prevent exposure.

Splitting an agent into a planning component that outlines steps and an executor that performs them, often with feedback.

A structured document that explains the user problem, product goals, scope, and success metrics for a feature.

The set of constraints, behaviors, and success conditions a solution must satisfy to solve the target problem.

A communication tool that shows strategic direction, problem areas, and planned outcomes over time.

A state where a defined segment repeatedly gets enough value from your product that retention and referral become durable.

Prompt engineering is the discipline of designing, testing, and maintaining the instructions, examples, and constraints a product sends to a language model to elicit a desired behavior for a specific task. It covers what the prompt asks the model to do and how that request is framed, not the broader runtime information environment those instructions run in (context engineering), the schema mechanism that enforces machine-readable outputs (structured outputs), or the test infrastructure that proves the prompt works (evaluation harness). Prompt behavior is empirical: it is measured against representative cases, not assumed from wording alone.

A user or document attempt to override system instructions, causing the model to act outside intended bounds.

A prompt library is a managed collection of reusable prompt assets, organized so teams can discover the right prompt by Product job, understand its inputs and dependencies, use a known version, and see its ownership and evaluation status. Stable identity, an explicit job, an owner, a status and version, declared variables and dependencies, recorded model and tool compatibility, and linked eval evidence are what separate a library from a folder of saved prompts. A library governs many assets, including prompt templates, system prompts, tool instructions, few-shot sets, and eval fixtures, but it is not itself one template, a security boundary, or a substitute for evaluation, permissions, or guardrails.

A prompt template is a reusable instruction structure that keeps stable task rules fixed while exposing explicit variables for the context that changes from one run to another. The stable part states what the model must do, must never do, and what output and failure behavior are expected; the variable part declares named inputs, required or optional, with admissible values and visible defaults, that a caller supplies per run. A template is a parameterized contract, not a guarantee: the same template can behave differently across models, providers, and tool or context configurations, so it needs a version and eval evidence before it is reused in production. It can render into any instruction layer a runtime supports, but it is not itself a system prompt, a security boundary, or a prompt library.

Systematic attempts to break or exploit an AI system to uncover safety and security weaknesses before attackers do.

A pattern where the model critiques or scores its own output (or an agent’s step) before finalizing or retrying.

Running automated checks to ensure a change doesn’t reintroduce past bugs or quality drops in model behavior.

A second-pass model or heuristic that orders retrieved items by relevance before feeding them to the LLM.

Pairing LLM reasoning with external retrieval so responses cite up-to-date, relevant sources instead of relying on model memory.

Structured outputs are schema-constrained model responses that downstream software can treat as an application contract: the provider or runtime constrains generation to a declared JSON Schema (shape, types, enums, required fields) and the application validates, handles refusal and no-data paths, and versions the contract. A schema guarantees structure, never truth, a perfectly valid object can still carry wrong values. Use native response-schema or strict tool-argument features for machine-readable shape; use validation, explicit unknown and refusal states, bounded retries, fallbacks, field-level evals, and compatibility handling for everything the schema cannot prove.

A system prompt is the high-priority, persistent instruction layer a product sends to a language model to establish stable behavior, role, policy, and constraints for an interaction, above or separate from task-specific user content. Exact role semantics vary by provider: some APIs expose a system role, others expose higher-priority developer instructions or a top-level instruction parameter, so system prompt is best read as the conceptual layer rather than one universal API field. It is one instruction surface inside prompt engineering and one governed source inside context engineering, not a security boundary, a secret store, or a substitute for tool permissions, schema validation, or guardrails.

The percentage of user or agent tasks completed correctly without human rework or retries.

The maximum tokens you allocate per request across prompt, tools, and output to control latency and cost.

Allowing a model to invoke predefined functions or APIs with structured arguments during its reasoning loop.

Explicit guidance given to a model about when and how to call tools or APIs, including constraints and safety rules.

How often model-invoked tools succeed, how they fail, and how gracefully the system recovers.

Crafting clear input/output definitions for tools exposed to the model to ensure safe, correct, and efficient calls.

A short statement of user intent written from the user perspective with clear acceptance criteria.

A clear statement of why a specific customer segment should choose your product over alternatives.

A storage and query engine optimized for vector similarity search, often combined with metadata filtering and hybrid search.