Product Analytics Instrumentation: Build a Trustworthy Tracking Plan

Most "product analytics instrumentation" advice starts with the wrong artifact. It hands you a list of events to track, a naming convention, a property schema and a dashboard, and treats the hard part as remembering to add the SDK calls. Then a metric moves, nobody can explain why, and weeks disappear into arguing about whether the number is real.

The hard part is not naming events. The hard part is making the data mean the same thing every time it is read, by people and systems who never spoke to each other. Instrumentation is the implementation contract between a Product decision and the evidence used to make it. If that contract is vague, every downstream analysis, activation, cohorts, retention, experiments, inherits the vagueness and looks like insight while being noise.

This guide owns that contract. It starts from the Product decision, not the event list, and works through the semantics that decide whether the resulting data can be trusted: the unit and the eligible population, the entity and identity model, the source closest to the truth, event and property semantics, time and duplicates, QA, schema change, privacy and the trust gate before anyone acts on the number.

Instrumentation exists to answer Product decisions reliably. If you cannot name the decision a signal changes, you do not yet know what to instrument.

An event is an observation. A metric is a computation. A campaign to "track more" is not a measurement plan.

Trust is the product. Data that nobody can defend is worse than no data, because it manufactures confidence.

Table of contents

• Instrumentation exists to answer a Product decision
• From Product decision to trusted metric
• Event, property, entity state and metric are four different things
• The Measurement Contract
• Choose the entity before the event
• Identity: anonymous, authenticated and the merge between them
• One user, many workspaces
• Capture at the source closest to the truth
• The source-of-truth map
• Event naming is a semantic problem, not a style contest
• Organize the taxonomy around domains and decisions
• Event tiers: decision-critical, diagnostic and exploratory
• Autocapture is useful, not sufficient
• Properties: required, optional and the silent-null problem
• Time, order and late arrival
• Duplicates, retries and idempotency
• Derived events and derived metrics
• Eligibility and denominator events
• Experiment instrumentation: eligible, assigned, exposed, converted
• The Event Spec Clinic
• The Event Spec template
• The Tracking Plan worksheet
• QA is more than "the event fired"
• The golden path as an analytics acceptance test
• The Instrumentation Trust Gate
• Schema changes: version, deprecate, migrate
• The Schema Change Ledger
• Metric reconciliation
• Data quality monitoring after release
• Privacy and data minimization for instrumentation
• Tools and vendors illustrate patterns, not prescriptions
• Who owns what: a lightweight instrumentation ownership model
• Instrumentation acceptance criteria
• Worked case 1: activation in a B2B workspace
• Worked case 2: a payment is not a button click
• Worked case 3: experiment exposure
• Worked case 4: one user, three workspaces
• Worked case 5: a schema change to a project event
• Common failure modes
• Instrumentation decision clinic
• References and further reading

Instrumentation exists to answer a Product decision

The most expensive instrumentation mistake is not a typo in an event name. It is building a firehose of clean-looking data that answers no question the team actually owns. Every event, every property and every query is a cost: build time, review time, privacy exposure, storage, and the analytical debt of a field nobody trusts.

So start one step earlier. Before "what should we track?", answer three questions:

1. What decision does this data change? Not "what would be interesting to know." A decision with an owner, a threshold and a consequence if the number is wrong.
2. What is the metric or state that decision depends on? Written as a sentence with a unit, a population and a window, for example, "the share of eligible workspaces that complete a first shared workflow within 14 days."
3. What is the smallest set of events, properties and entity records needed to compute it correctly? Then, and only then, choose the capture points.

This is the difference between instrumentation as evidence and instrumentation as collection. The first is bounded and reviewable. The second grows until the pipeline is too expensive to trust and too noisy to query.

The page intentionally does not begin with page_view, button_clicked, signup and purchase. Those may be useful. They are only instrumentation once they are tied to a decision and given a stable meaning.

Several jobs sit upstream of this page and this page routes to them rather than redefining them: choosing the strategic metric belongs to the North Star guide and the North Star Metric Finder; breaking a metric into drivers belongs to the KPI Tree Builder; deciding what counts as first value belongs to the activation definition method. This page owns what happens after the definition exists: turning it into data anyone can defend.

From Product decision to trusted metric

Instrumentation is a chain, and a break at any link corrupts everything downstream. Work it left to right; do not start at "events."

Product decision
      ↓
Metric / state definition          (unit + population + window + threshold)
      ↓
Entity + eligibility               (who or what can be counted)
      ↓
Events / properties / state        (what must be observed)
      ↓
Source of truth                    (client / server / backend system / warehouse)
      ↓
Identity + time semantics          (stable IDs, event time, timezone)
      ↓
Schema + version                   (meaning, owner, change history)
      ↓
QA                                 (trigger, negative, identity, property, end to end)
      ↓
Metric computation                 (the derived number)
      ↓
Analysis                           (cohort / funnel / experiment)
      ↓
Decision

Feedback loop:
Product or schema change → back to the contract and the schema change ledger.

Read the map as a set of failure points. "The dashboard says activation fell" can break at the metric definition, the unit, the eligible population, the capture source, identity merge, a timestamp, a duplicate, a schema change, or the QA. Naming events nicely fixes exactly one of those.

The chain also tells you what to keep small. You do not need lineage tooling to reason about a decision. You need to be able to trace a metric back through the computation, the events and entities, the source systems and the owners. If you can, debugging is minutes. If you cannot, every disagreement about a number becomes a political argument.

Event, property, entity state and metric are four different things

Teams conflate these constantly, which is why "one event equals one metric" quietly becomes the operating model. It is wrong, and it is expensive.

Two consequences follow.

First, a metric is not an event. Retention, activation rate, conversion and revenue are usually computations, often over several events and entity records. You should almost never emit a client event called user_is_retained or account_activated just because the computation is annoying. That is a derived metric wearing an event's name, and it will drift from the logic that is supposed to produce it.

Second, the same event can drive many metrics, and the same metric can need many events. workflow_shared might feed activation, collaboration depth, frequency of value and an experiment outcome. One event, several metrics. Do not create a new event for every chart.

The Measurement Contract

This is the primary artifact. Fill it in before writing tracking code, and treat an empty field as an open decision rather than a formality. A contract that cannot be completed honestly is the finding: the question is not yet answerable.

# Product Measurement Contract

Product decision:
[What decision changes, who owns it, and what happens if the number is wrong]

Metric / state:
[The metric or state in one sentence, with unit, population and window]

Why this metric changes the decision:
[...]

Analysis unit:
[User / Account / Workspace / Transaction / Order / Other]

Eligible population:
[Who or what is allowed into the denominator]

Entity identifiers:
- user:
- account / workspace:
- anonymous / pre-auth:
- object / transaction:

Entry / denominator event:
[What puts a unit into the population]

Success / value event or state:
[What counts as the outcome]

Required properties:
- [...]

Optional diagnostic properties:
- [...]

Source of truth:
[Client / Server / Backend DB / Warehouse / External system / Mixed]

Event time definition:
[When the event is considered to have happened]

Timezone:
[...]

Deduplication key:
[What makes two records the same real-world occurrence]

Expected ordering / dependencies:
[What must happen before what, if the metric depends on sequence]

Privacy / minimization notes:
[What is deliberately not collected]

Owner:
[Who is accountable for the meaning]

QA owner:
[Who verifies the implementation]

Release verification:
[How production is checked after a change]

Version:
[Schema / definition version and effective date]

Known limitations:
[What this data cannot yet support]

Revisit trigger:
[What would force this contract to be re-opened]

Three fields deserve emphasis.

Owner. An event with no named owner decays. The owner is the person who answers "what does this mean?" six months from now, not the person who happened to write the tracking call.

Version and effective date. Definitions drift, an event is renamed, a filter changes, a plan is excluded. Without a version, every historical comparison is a guess dressed as a trend.

Known limitations. Writing down what the data cannot do yet prevents the team from using it for a decision it cannot support. Limitation is not failure; hidden limitation is.

The contract deliberately has no "quality score" and no "confidence percentage." Inventing a number like "78% reliable" adds precision the data does not have. Use descriptive states instead, as the trust gate does.

Choose the entity before the event

Instrumentation breaks when the entity model is unclear. An "event" is always an event about something: a person, an account, a workspace, an order, a subscription, a device. Choose the entity that matches the decision, because attaching everything to a user is a decision in disguise, and often the wrong one.

A useful sanity check:

If the thing that receives the value cannot experience it on its own, the unit is probably too small.

A person cannot "activate" a shared workspace by themselves. A workspace cannot "log in." Pick the unit the decision is about, write it into the contract, and do not silently swap it later. When the unit is unclear or contested, the cohort comparison method owns the deeper comparison design.

Identity: anonymous, authenticated and the merge between them

Identity is the single most common source of silently wrong product data. Getting it wrong does not produce an error; it produces a plausible number that is quietly measuring the wrong population.

Three identifiers usually exist and must be kept conceptually separate:

• Anonymous / pre-auth ID. A stable identifier generated before login or signup. It is what lets you analyze a visit that has not become a user.
• Authenticated user ID. The durable ID of a known person, ideally independent of email, device or login method.
• Entity ID(s). The account, workspace or object the person acts within at the moment of the event.

The merge between the first two is a contract, not an implementation detail. Decide and document:

• When does the merge happen? At signup, at first login, at email verification, at subscription activation, different products legitimately choose differently.
• Is it one-way? Most systems merge the anonymous history into the known user. Reversing a bad merge is hard or impossible; do not treat it as reversible unless the platform says so.
• What about shared devices? A family tablet or a demo kiosk can produce one anonymous session that several people use. Do not assume anonymous equals one person.
• What about logouts and multi-account users? One person may use more than one account, or one account may be shared by a team. Logout does not guarantee a fresh anonymous identity, and login does not guarantee a single actor forever.
• What is the failure mode of a duplicate merge? Two unknown sessions may later belong to the same person, or one session may map to two accounts. Decide what the analysis should do when the merge is ambiguous.

Platforms implement this differently, and the differences matter. As documented examples rather than recommendations: Mixpanel distinguishes a simpler merge from its original ID-merge behavior, and its $identify-style call is what transitions a user from anonymous to known; Segment's identify call updates a user's traits. The conceptual contract is the same everywhere: decide when unknown becomes known, what happens to the history, and what happens when the merge is wrong.

Do not paste a vendor's alias call into this page's advice as if it were universal. Read the current documentation for the platform you use, and write your merge rules into the Measurement Contract.

One user, many workspaces

B2B instrumentation fails in a specific, predictable way: everything is attached to user_id, and the account or workspace reality disappears. Consider one person who belongs to three workspaces with a different role in each. They invite a teammate in workspace A, complete a review in workspace B and only read in workspace C.

Question: did the user act, or did a workspace activate?

Both are real, and the answer depends on the decision. What is not acceptable is inferring the workspace later from current membership. Membership changes. People leave, roles move, workspaces get renamed. If the workspace context at the time of the event was not captured, no later join can reconstruct it faithfully.

For B2B products, capture at minimum:

• user ID, who acted;
• account / workspace ID, where the action happened;
• role or membership at event time, what authority the actor had then;
• account-level events and state, invite sent, seat added, workspace plan changed, first shared workflow completed.

This is the difference between a user metric that looks alarming and an account metric that is actually stable. It is also why this page insists the entity model comes before the event list. The activation definition method decides whether the account or the user is the activation unit; this page makes sure that decision survives contact with the tracking code.

Capture at the source closest to the truth

Choosing between client-side and server-side capture is routinely presented as a binary where server-side wins. That framing is incomplete and produces its own class of bug.

The useful rule:

Capture the event at the source closest to the authoritative truth for that event.

That single rule prevents both common errors: treating a browser click as proof of a business outcome, and dismissing the client entirely so that exposure, context and UI-level behavior are lost.

Two practical patterns follow.

Mixed model for outcomes. Log the exposure or intent client-side (the user saw the paywall, the user started checkout) and log the outcome server-side (the payment settled, the subscription activated). This gives you the full journey without pretending a click is revenue.

Server-side is not automatically better. A server event that fires on a request that later fails can be less correct than a client event tied to what the user actually saw. Source-of-truth is about which layer knows the truth of this fact, not about which layer sounds more senior.

Whatever you choose, write it in the contract, and make sure the event carries enough identity and context to be joined back to the entity that owns the decision.

The source-of-truth map

Different facts live in different systems, and the analytics tool is rarely the canonical owner of any of them. A short source-of-truth map prevents a large share of "the numbers disagree" meetings.

The map has one job: when two systems disagree about a number, you know which one is allowed to win. For revenue, billing and payment systems usually win; for exposure and UI behavior, the client is the only witness. Do not let a convenient analytics field quietly become the canonical truth just because it is easy to query.

Event naming is a semantic problem, not a style contest

Event naming gets more ink than any other part of instrumentation and matters less than the semantics underneath it. There is no universally correct style. Segment's guidance uses the Object + Action framework with Title Case event names and snake_case properties; Amplitude's default convention is Noun + Past-Tense Verb in Title Case from the user's perspective. Both are defensible. What is not defensible is having two conventions in one product, or a convention that hides meaning.

Choose one convention and hold it. Then enforce what actually matters:

• One semantic meaning per event. If signup_completed sometimes means "form accepted" and sometimes means "account usable," it is two events wearing one name.
• A stable definition. If the trigger changes, that is a schema change (handled later), not a quiet edit.
• No hidden context in the name. button_clicked and cta_clicked tell you nothing about the business event. workspace_invitation_sent, payment_completed and analysis_exported do.
• No dynamic or value-carrying names. Do not put dates, IDs or categories in the event name (User Signed Up 11-01). Values belong in properties.
• No duplicate synonyms. project_created and project_added should not both exist for the same occurrence.
• Human-readable and machine-stable. A new engineer should guess the meaning; a query should not break because someone "improved" the name.

The distinction that resolves most naming arguments:

A UI diagnostic event is not a business metric. upgrade_cta_clicked may be useful for diagnosing a funnel; it must not be the number that counts paying customers. When in doubt, ask what decision would change if the event count doubled, if the answer is "none," it is a diagnostic.

Organize the taxonomy around domains and decisions

A taxonomy is a way to know what you have. Organize it around Product domains and decisions, not around screens. Screens change; the customer job usually does not.

A workable grouping (adapt it; do not adopt it as law):

• Identity and account, signup, verification, account and workspace lifecycle, roles, membership;
• Acquisition, visit, channel, campaign, referral, first-touch context;
• Activation, the early-value milestone and the eligibility that precedes it;
• Core value, the core action the product exists to perform;
• Collaboration, invites, sharing, participation, multi-actor workflows;
• Transaction / monetization, checkout, payment, subscription, refund, plan change;
• Experiment, assignment, exposure, outcome;
• Error and reliability, client errors, failed operations, retries.

The test of a taxonomy is not whether the list looks complete. It is whether a new engineer, given a feature, can place its events in the right domain without asking, and whether an analyst can find every event that affects a metric they care about.

Avoid two default failure modes: one event per click (a screen recorder, not a measurement system) and track everything and analyze later (a bill, a privacy surface and an analysis debt). Autocapture mitigates the first but does not replace semantics, as the next section argues.

Event tiers: decision-critical, diagnostic and exploratory

Not every event deserves the same governance. Spending equal review effort on a revenue event and a hover event is how instrumentation reviews become theater.

The tier is a statement about how much a wrong value would cost. A rough click count may be sufficient for an exploratory question and is not sufficient for a pricing decision, an experiment's primary outcome or a board metric. Tiering lets you instrument exploration without pretending it is evidence.

This is also the honest answer to "how many events should we track?" There is no magic number. Instrument the value flows and decisions you own; tier the rest. A team with 30 well-governed events is in better shape than one with 3,000 unowned ones.

Autocapture is useful, not sufficient

Autocapture, automatically collecting clicks, pageviews, navigation, dead clicks and similar interactions, is genuinely valuable for exploratory work. It answers questions you did not know to ask, it survives some UI churn, and it can reveal where users are stuck. PostHog, for example, documents interaction, navigation, clipboard, dead-click and lifecycle autocapture across SDKs, with privacy controls and the ability to rename and organize captured events.

What autocapture does not do is give you stable semantic business events for free. Its limits are structural:

• Semantic instability. The captured event describes the UI (clicked span with text "Delete"), not the business meaning ("project deleted"). Redesigns change the capture, and history becomes incomparable.
• UI dependence. A button rename, a component refactor or a DOM change can silently move or drop what you were measuring.
• Privacy surface. Autocapturing everything captures things you did not intend to collect. PostHog, for instance, deliberately does not capture form values by default for exactly this reason.
• Governance difficulty. Thousands of unowned interactions are hard to review, version or reconcile.

The working rule:

Use autocapture for exploration and diagnostics; use explicit, reviewed semantic events for any metric that informs a decision.

A vendor claiming autocapture removes the need for a tracking plan is describing a smaller governance problem, not a solved one. Evaluate the limitation honestly before accepting the marketing.

Properties: required, optional and the silent-null problem

Properties are where taxonomy quietly rots. Two disciplines keep it healthy.

Required vs optional. For each event, separate:

• Required properties, needed to interpret or compute the metric. payment_completed without currency and amount is not analysable.
• Optional diagnostics, help segment or debug (source_page, app_version) but do not change the metric's meaning.

Start minimal. Every required property must earn its place, because a required property that is missing 30% of the time makes the metric unusable for the decisions that need it, which is worse than a smaller, complete schema.

Controlled values vs free text. Stable analytical dimensions should use controlled values. plan = Pro / pro / PRO / premium fragments one dimension into four and produces wrong counts. Define enums for plan, platform, status and similar dimensions. Genuinely open data (a search query, a free-text note) can stay free, but it is a different kind of field and should not be treated as a dimension you can rely on.

The silent-null problem. These are not the same thing, and dashboards routinely merge them:

• the property was omitted (the event predates the field, or the client did not send it);
• the property is explicitly null (present and empty by design);
• the value is unknown / not captured (the source did not know);
• the value is not applicable (the dimension does not apply to this event);
• enrichment failed (a join or lookup broke downstream).

Treating all of these as "other" is how a metric becomes confidently wrong. Decide which states are legal, document them, and make obvious in the data layer when a value is missing rather than merging it into a real category.

Avoid property explosion and over-collection. Do not store every UI string, raw user input, secret, token or sensitive free text just because it is within reach. Collect what the decision needs and nothing that creates risk, which is also the privacy rule later in this guide.

Time, order and late arrival

"My data changed" is often "my time semantics changed." Distinguish at least four timestamps:

Product metrics should generally be computed on event time, because that is what describes user behavior. Processing time is a pipeline fact and drifts with retries, backfills and batching. If a cohort, retention curve or experiment is silently keyed on processing time, late events distort it: a user who acted on Monday but whose event arrived Wednesday can land in the wrong cohort window.

Questions to answer explicitly for each decision-critical event:

• Can the event be produced offline (mobile, kiosk, unreliable network) and delivered later?
• Is there clock drift on the device, and does the server timestamp override or coexist with the client timestamp?
• Which timezone defines day, week and month boundaries for the metric, and does it match how the business operates?
• Do retries create new event times, and are they deduplicated?

Order deserves the same discipline. Ingestion order is not real-world order. If a metric depends on sequence, checkout_started before payment_completed, invited before joined, document that dependency in the contract, and prefer entity state (the order's status) over inferred event order when the decision is critical.

Duplicates, retries and idempotency

A surprising share of inflated revenue and conversion numbers is simple double counting. Duplicates come from everywhere: network retries, webhook redelivery, app relaunch after a failed upload, batch resend, offline queues replaying on reconnect, and server functions invoked more than once.

For decision-critical events, define a deduplication contract:

• What is the identity of one real-world occurrence? A client-generated event_id, a backend transaction_id, an order ID, or a natural key.
• Who is responsible for uniqueness? Client, API gateway, ingestion pipeline or the warehouse model.
• How are retries marked? Retries should carry the same dedupe key so downstream can collapse them.
• What does the metric do with duplicates? Count distinct keys, not raw rows.

There is no single universal implementation, but there is a universal requirement: any metric you would act on must be able to answer "could this be counted twice?" in one sentence. If it cannot, it is not yet a trustworthy metric. Payments and subscriptions almost always ship with a transaction identity from the provider, use it rather than reconstructing one from click events.

Derived events and derived metrics

Some "events" are not observed; they are computed. workspace_activated, user_is_retained and account_churned are typically derived from multiple raw events and entity state, with a rule and a window.

That is fine. What is not fine is emitting a client event named like a conclusion. The difference matters for three reasons:

• Recomputability. If the rule changes, a derived metric can be recomputed from raw events. A client event that asserted the conclusion cannot be recomputed; it is frozen and wrong.
• Clarity. A derived event hides its inputs and its version. The next analyst cannot tell what "activated" required at the time.
• Correctness. A client cannot reliably know that activation happened; only the rule can.

So document derived state as what it is:

• Raw source events, what is actually observed;
• Computation, the rule, window and filters;
• Derived state / metric, the output;
• Version, the definition's effective date.

Emit a derived event only when it is genuinely useful operationally (for example, a backend lifecycle event that triggers lifecycle messaging), and even then treat it as a record of the computation, not as an independent observation. Retention, in particular, is generally derived, not emitted, the retention system owns how it is defined.

Eligibility and denominator events

Most product metrics fail on the denominator, not the numerator. "Activation improved" means nothing if you cannot say who was eligible to activate.

Instrumentation should preserve who had the opportunity to succeed:

• eligible for onboarding (the user/account actually received it);
• exposed to the feature (not just assigned to it);
• invited to a workspace (as opposed to merely signed up);
• eligible for a trial (plan available in their region and tier);
• eligible for the experiment (met the targeting conditions).

If eligibility is not captured, the metric silently changes meaning as the product changes. A new signup flow that routes enterprise accounts elsewhere can make activation "improve" simply because a harder population left the denominator.

For project-critical metrics, make the eligibility rule explicit in the contract and capture the event or condition that establishes it. This single discipline prevents a large share of activation, funnel and experiment misinterpretation.

Experiment instrumentation: eligible, assigned, exposed, converted

Experiments are where instrumentation errors become false causal claims. Four distinct states must be kept separate:

The critical mistake is treating assigned as exposed. A unit can be assigned to a variant and never open the screen, never reach the step, or never load the changed component. Analysing on assignment when the treatment only affected exposed units dilutes the effect and can reverse the conclusion. Analysing on exposure when assignment is what randomised can reintroduce selection bias. Neither is automatically right; the design determines which analysis population is valid.

The instrumentation page's job is narrow but essential: make sure eligibility, assignment and exposure are each captured with the right entity, at the right moment, with the right identity, otherwise no analysis is trustworthy. The design of the experiment itself belongs to the A/B Test Plan Generator and the low-traffic experimentation method.

The Event Spec Clinic

This is the second artifact. Weak specs look fine until something changes. Strong specs survive a redesign, an identity merge and a new analyst.

Weak: signup_complete

Event: signup_complete
Trigger: fires when the user finishes signup
Properties: user_id, email

Problems hiding in plain sight: What is "complete", form submitted, account created, or email verified? Is the entity the user or the account? What about OAuth and invited users? Is it client or server? Can it fire twice? What does a marketer analysing "signups" actually get?

Stronger: account_created

Event: account_created
Business meaning: a new, durable account record exists and is usable
Exact trigger: after the backend persists the account and returns its ID
Must NOT fire: on form open, validation error, OAuth redirect, duplicate retry, or invited-user join
Entity IDs: user_id, account_id
Source: server (backend account service)
Required properties: account_id, user_id, signup_method, plan_at_creation
Optional properties: referrer_source, campaign
Event time: backend persistence time; timezone UTC (store), business timezone for reporting
Deduplication: account_id (one account_created per account)
Ordering: precedes any onboarding eligibility event
Owner: Growth engineering
Version: v2 (v1 was client-side form submit; see change ledger)
QA cases: new email signup, OAuth signup, invited user, duplicate submit, failed validation
Downstream metrics: new-account count, activation denominator, cohort entry

The same approach for the other families. Each is a hypothesis about meaning, written so a reviewer can falsify it.

Activation: first_shared_workflow_completed

• Weak version. activated fired client-side when the user opened the workspace and clicked around.
• Stronger version. Fired server-side when a workflow is shared with at least one other workspace member and is saved; entity is the workspace; unit is the workspace; required properties include workspace_id, workflow_id, participant_count; deduped per workspace_id + workflow_id; window and eligibility defined by the activation contract.

Transaction: payment_completed

• Weak version. purchase_button_clicked counted as revenue.
• Stronger version. Fired from the payment webhook after settlement, carrying transaction_id, amount, currency, plan_id, account_id; deduplicated on transaction_id; refunds and cancellations captured as separate state, not by decrementing an event count. Worked in detail below.

Collaboration: workspace_invitation_sent

• Weak version. invite_clicked.
• Stronger version. Fired server-side after the invitation is persisted and (where applicable) delivered, carrying workspace_id, inviter_user_id, invitee_ref, invite_channel; separate from invitation_accepted; deduplicated per invitation.

Experiment exposure: experiment_exposed

• Weak version. experiment_assigned treated as exposure.
• Stronger version. Fired at the point the variant is actually rendered or applied, carrying experiment_key, variant, unit_id and the entity IDs; separate from assignment and from the outcome event.

The Event Spec template

Copy this for each decision-critical event. Right-size it: a Tier 1 event earns every field; a Tier 3 interaction does not.

# Event Spec

Event:
[...]

Business meaning:
[One sentence a new analyst would understand]

Exact trigger:
[...]

Must NOT fire when:
[...]

Entity IDs:
- user:
- account / workspace:
- object / transaction:

Source:
[Client / Server / Backend DB / Warehouse]

Required properties:
- [...]

Optional diagnostic properties:
- [...]

Event time:
[When it is considered to have happened]

Deduplication:
[What makes two records the same occurrence]

Owner:
[...]

Version:
[...]

QA cases:
1. [...]
2. [...]
3. [...]

Downstream metrics:
- [...]

Known limitations:
[...]

The Tracking Plan worksheet

The tracking plan is the table view of the specs, and it should be reviewable in one screen. Keep it to the columns that change a decision.

Field tools formalise the same idea. Segment's Tracking Plan and Protocols, for instance, exist to make the plan a single source of truth rather than a stale spreadsheet. The tool is optional; the discipline is not. A plan that lives only in a vendor UI and is never reviewed by the team that owns the metrics is a spreadsheet with better branding.

QA is more than "the event fired"

"The event fired" is the weakest possible test and it passes for broken instrumentation all the time, wrong entity, missing property, duplicate count, wrong source, wrong compressed timestamp. Build QA as a set of questions, not a single check.

Negative QA is the most neglected and the most valuable. Instrumentation that over-fires is often harder to detect than instrumentation that under-fires, because over-firing produces a larger number and rarely triggers an alert. Test the cases where the event should be silent.

The golden path as an analytics acceptance test

Define a small, fixed set of known user journeys with expected analytics output. This is the analytics equivalent of a smoke test, and it catches integration breaks that unit tests miss.

For a B2B collaboration product, one golden path might be:

1. Anonymous visit to pricing
2. Signup → account_created (server)
3. Create workspace → workspace_created (server)
4. Invite teammate → workspace_invitation_sent (server)
5. Teammate accepts → invitation_accepted (server)
6. First shared workflow saved → first_shared_workflow_completed (server, derived condition)

Expected:
- exact event names, in order
- correct user_id, account_id, workspace_id on each
- anonymous session merged into the new user once
- first_shared_workflow_completed fires exactly once per workspace + workflow
- activation metric increments by one for the eligible workspace

Write the expected event sequence, IDs, account linkage, order and properties. Then make it part of release QA. Run it after a tracking change the same way you would run a regression test, because it is one.

The golden path is also the fastest way to onboard a new engineer or analyst. Reading real expected output teaches the system faster than reading the taxonomy.

The Instrumentation Trust Gate

This is the third artifact. Before data supports a Product decision, walk the gate. It is descriptive, not a score, and it is deliberately boring.

# Instrumentation Trust Gate

DEFINITION     Is the metric/event meaning documented and owned?
ENTITY         Are user / account / object IDs stable and correct?
ELIGIBILITY    Is the denominator measurable?
SOURCE         Is the event captured at the authoritative layer?
DELIVERY       Could events be lost, retried or duplicated?
TIME           Are event time, timezone and ordering correct?
SCHEMA         Are required properties present and valid?
VERSION        Has the definition changed since the period being read?
QA             Was end-to-end behavior verified, including the negative case?
PRIVACY        Are only the needed fields collected?
OBSERVABILITY  Will a failure be noticed?

Produce one of four qualitative outputs:

• Trustworthy enough for this decision.
• Usable with stated caveats.
• Not trustworthy yet.
• Unknown, the gate has not been run.

The gate is decision-scoped, not global. Data can be trustworthy enough for an exploratory question and not trustworthy for a pricing decision. Say which, and say why.

Schema changes: version, deprecate, migrate

Instrumentation is not built once. Products change, and so must events. The failure mode is not change itself; it is silent change, reusing an event name with a new meaning so that a trend becomes a comparison of two incompatible definitions.

When a definition changes, decide explicitly:

• What exactly changed? Trigger, entity, properties, source, or the derived rule.
• Does the change break historical comparability? If yes, say so; do not plot across the break as if nothing happened.
• Can history be backfilled? Sometimes the raw data allows recomputation; often it does not.
• Who depends on it? Dashboards, alerts, cohorts, experiment readouts, finance reports.

There is no single correct versioning strategy. Common options:

• a version property on the event (schema_version);
• a new event name with the old one deprecated;
• a warehouse transformation that maps old records to the new meaning;
• a semantic-layer change where the metric definition moves but the raw event stays.

Pick one, document it, and keep the old definition readable for anyone analysing history. This page adds the Schema Change Ledger as the record that makes the decision reviewable.

The Schema Change Ledger

The fourth artifact. One entry per meaningful change.

# Analytics Schema Change Ledger

Event / metric:
[...]

Current definition:
[...]

Proposed change:
[...]

Reason:
[...]

Effective date:
[...]

Owner:
[...]

Downstream metrics / reports:
- [...]

Historical comparability:
[Preserved / Partial / Broken]

Backfill:
[Yes / No / Not possible]

Migration plan:
[...]

Deprecation date for old definition:
[...]

Validation:
[How the change was verified]

Communication:
[Who was told, and where]

Two rules keep the ledger honest: no silent reuse of a name, and no fake governance score. The value is the record, not a rating.

Backfill deserves care. If the raw source data exists, an event can often be recomputed into the new definition. If it does not, because the old event never captured the needed property, then the metric has a genuine break, and pretending otherwise is an invented history. Mark the break and move on.

Metric reconciliation

Critical metrics should be reconciled against another trusted source when one exists. Not because the analytics tool is wrong, but because reconciliation is how you find out how it is wrong.

Do not require perfect parity. Systems legitimately differ by timing, scope and definition. What you require is a documented expected discrepancy: "analytics counts a payment at settlement; billing counts at invoice creation; a 1–2 day lag is expected." An unexplained gap is a defect; an explained one is a known limitation.

Data quality monitoring after release

Instrumentation trust is not a one-time QA pass. The events that were perfect at launch degrade quietly. Watch the cheap signals that catch most breakage:

• event volume anomalies, a sudden step change in a core event;
• null-rate changes, a required property that used to be complete now is not;
• enum drift, new, unexpected values in a controlled dimension;
• duplicate rate, dedupe keys firing more than once;
• missing identity, events with no user/account ID;
• schema validation failures, malformed payloads reaching the pipeline;
• source drops, a client or server path stops reporting;
• delay, a growing gap between event time and arrival.

You do not need a dedicated observability platform to do this. You need alerts or a recurring check on the handful of Tier 1 metrics. The rule is simple:

A metric you would act on needs some way to notice when the data behind it breaks.

Privacy and data minimization for instrumentation

This page is not a privacy policy, and it does not give legal advice. It does teach the instrumentation hygiene that prevents most privacy incidents from being created in the first place.

• Collect only what the decision needs. Every extra field is cost and risk.
• Never send secrets, passwords, tokens or full credentials to an analytics pipeline.
• Avoid sensitive free text by default. Support-message content, notes, addresses and health or financial detail do not belong in an event payload unless there is a documented, reviewed need.
• Take PII seriously at the schema level. If an email or name is genuinely required, treat the field as sensitive: restrict access, define retention, and prefer identifiers over raw values.
• Review analytics vendor and autocapture settings. Default capture may collect more than you intend.
• Honor the current consent and legal requirements that apply to you, and route legal questions to the owner who makes those determinations.

Retention deserves a distinction, because "we delete analytics data after X" is often conflated with account data:

• analytics-event retention, how long raw events are kept;
• account-data retention, how long the product keeps the customer's data;
• warehouse retention, how long transformed tables persist;
• backup retention, how long backups retain copies.

These are different policies with different owners. Do not invent legal durations on a product education page. State the distinction, implement the organization's actual policy, and route legal disclosure to the privacy policy rather than rewriting it here.

Tools and vendors illustrate patterns, not prescriptions

The analytics market is deep, and the correct choice depends on the team, the stack and the constraints. This page deliberately does not rank tools. Vendors are referenced below only because each illustrates a pattern you may need to solve, and vendor behavior is not a universal law.

Three cautions. First, do not reproduce volatile pricing or feature comparisons; documentation changes faster than any article. Second, separate the conceptual method from the vendor implementation, "define the decision, then the event" is method; a specific alias call is implementation. Third, verify current documentation before relying on any specific syntax. The method is portable; the syntax is not.

Where a tool genuinely removes work, use it. A tool that lets the team review events, validate schemas and see ownership is worth real money. A tool that lets the team skip the semantics is just a faster way to collect unowned data.

Who owns what: a lightweight instrumentation ownership model

Instrumentation fails as an ownership problem more often than as a technical one. A workable split, not a universal RACI:

Two boundaries matter. Product Ops may own the shared definitions, access and recurring analytics process, but it does not become the central analyst for every Product question, the instrumentation methodology and measurement contract stay with the team that owns the decision. And every Tier 1 event needs one named owner, even if the work is shared. Shared ownership without a name is how schemas decay.

When a conflict arises between Product's intent and Engineering's implementation, the Measurement Contract is the referee. That is its main practical value.

Instrumentation acceptance criteria

"Track activation" is not a ticket. It is a wish. Strong instrumentation work is specified so that it can be verified and rejected.

A ticket should state:

# Instrumentation Acceptance Criteria

Event(s) and canonical names:
[...]

Business meaning:
[...]

Exact trigger and negative case:
[Fires when... / must not fire when...]

Entity IDs on every event:
[user_id, account_id, workspace_id, transaction_id...]

Required properties and valid values:
[...]

Source and delivery:
[client/server; retry behavior]

Expected count behavior:
[once per X; may repeat per Y]

Deduplication key:
[...]

QA environment and method:
[...]

Expected result in destination / warehouse:
[...]

Downstream metric it feeds:
[...]

Owner and version:
[...]

The acceptance criteria are also the contract. If a reviewer cannot verify each line, the implementation is not done. This is where instrumentation stops being a vague engineering task and becomes a Product artifact with a definition of done.

Worked case 1: activation in a B2B workspace

All numbers and names here are hypothetical. This is a reasoning trace, not a benchmark.

Decision. Should we invest in the collaboration invite flow? We need to know what share of eligible workspaces reach first shared value.

Metric. Share of eligible workspaces that complete a first shared workflow within 14 days of workspace creation.

Unit. Workspace.

Eligible population. Workspaces created in the period, excluding internal/test accounts and workspaces created by users who are not the owner of the workspace (they cannot invite).

Source events. workspace_created (server), workspace_invitation_sent (server), invitation_accepted (server), first_shared_workflow_completed (server, derived condition).

Source-of-truth split. All server-side, because the outcome is a durable backend state, not a UI interaction. A client invite_clicked is a diagnostic only.

Identity. user_id for the actor; workspace_id for the unit; account_id where the workspace belongs to a parent organization. The workspace context is captured at event time, not inferred from current membership.

Required properties. On workspace_created: workspace_id, owner_user_id, created_at, plan_at_creation. On first_shared_workflow_completed: workspace_id, workflow_id, participant_count, completed_at.

Deduplication. One workspace_created per workspace; one first_shared_workflow_completed per workspace (the first matching workflow), keyed on workspace_id.

Timing. Event time from the backend; UTC storage; the 14-day window computed in event time, not ingestion time.

QA. Trigger QA (fires only on a real shared, saved workflow); negative QA (a solo draft does not count); identity QA (correct workspace); property QA (participant_count ≥ 2); dedupe QA (re-saving does not re-fire); end-to-end QA (the metric reconciles with a hand-checked sample).

Derived activation state. workspace_activated = true when the rule is satisfied. It is a derived state with a version, not a client event.

Downstream use. The workspace cohort goes to the cohort comparison method and the repeated-value system in the retention guide.

The takeaway: "track activation" became a unit, a population, four events, a dedupe rule, a window and a QA plan, before anyone wrote a tracking call.

Worked case 2: a payment is not a button click

The problem. purchase_button_clicked is counted as revenue. Then the number is wrong when users retry, when payments fail, when the page is refreshed, and when a refund happens.

Why the click is not authoritative. The click is an intent signal from the client. It cannot know whether the charge settled. Two clicks can be one purchase. A click can be a payment that later fails. A refund can arrive days later.

The authoritative event. payment_completed, fired from the payment provider webhook (or the backend after settlement), carrying:

• transaction_id (provider identity, the dedupe key);
• amount and currency (never sum across currencies without conversion);
• plan_id / product_id;
• account_id (the paying entity, not just the user who clicked);
• payment_method_type;
• settled_at (event time) and the provider's status.

Retries and duplicates. Webhooks redeliver. Dedupe on transaction_id; a retry with the same ID is the same purchase.

Refunds and cancellations. Model them as separate state transitions (payment_refunded, subscription_cancelled) rather than decrementing the completion count. Gross revenue and net revenue are different metrics with different rules.

Reconciliation. Reconcile the analytics payment count against the payment provider on a schedule. Expect a small timing discrepancy and document it.

The instrumentation lesson is not billing architecture. It is the source-of-truth rule: capture the outcome where the truth about the outcome lives.

Worked case 3: experiment exposure

The problem. An experiment is analysed on assignment, but a large share of assigned users never reached the changed screen. The measured effect is diluted and the readout is wrong.

The four states. Eligible (met targeting), assigned (allocated), exposed (actually saw the variant), converted (outcome occurred).

Why assignment is not exposure. Assignment is a logical allocation that can happen before a screen loads, before a feature flag is evaluated in the client, or for a user who never returns. Exposure is the moment the variant is actually applied or rendered.

What to instrument. An experiment_exposed event fired at the point of real exposure, carrying experiment_key, variant, the analysis unit ID and the relevant entity IDs. Keep assignment separate. Keep the outcome event defined by the metric owner.

Why it matters. If exposure is missing, the analysis population is ambiguous, and neither assignment-based nor exposure-based analysis can be defended. Fix instrumentation before reading the result. The design and statistics belong to the A/B Test Plan Generator and the low-traffic method.

Worked case 4: one user, three workspaces

The problem. A single power user belongs to three workspaces with different roles. A user-level activity metric looks healthy, but two of the three workspaces are stalled.

Why user_id alone fails. The user acting is not the same as the workspace progressing. And the role that permitted the action differs by workspace.

The fix. Every event carries user_id and workspace_id (and account_id where relevant), plus the role at event time. Account-level events and state are captured separately. Dashboards can then show user activity and workspace health independently.

Why current membership cannot fix it later. If the workspace context was never captured, a join on today's membership attributes historical actions to a workspace the user may not have belonged to then. Roles change and membership churns. Attribution must be captured at event time or it is fiction.

This is the case that most clearly separates competent B2B instrumentation from a user-only schema.

Worked case 5: a schema change to a project event

The situation. project_created originally meant "a draft was started." After a redesign, the team wants it to mean "a valid, saved project." The dashboard trend would become meaningless if the meaning changed silently.

Bad response. Change the trigger and keep the name. Every historical comparison is now two different definitions pretending to be one trend.

Good response.

• Record the change in the Schema Change Ledger.
• Bump the definition: either a schema_version property or a new name (project_saved) with project_created deprecated.
• Assess comparability: partial, the old events cannot distinguish drafts from saved projects.
• Backfill: not possible from the old event alone unless a draft/saved flag already existed in the backend.
• Update downstream metrics and dashboards by the effective date; annotate the trend at the break.
• Communicate to every consumer before the change ships.

The metric now has an honest break. That is better than a smooth, false trend.

Common failure modes

Instrumentation decision clinic

The situations teams actually bring, and what this framework suggests.

References and further reading

• Twilio Segment, Data Collection Best Practices. Documents the Object + Action naming framework, Title Case events with snake_case properties, avoiding dynamic event names and property keys, limiting the plan to a small core set with rich properties, and the Tracking Plan as a shared source of truth. Vendor methodology, not universal law.
• Amplitude, Plan your taxonomy. Three-step taxonomy (business objectives → key metrics → optimize events and properties), Noun + Past-Tense Verb convention, event/user property distinction, and anonymous-user reconciliation before login.
• Snowplow, Server-Side vs Client-Side Tracking. The client/server distinction and the hybrid approach; used here to argue for capturing at the source closest to the truth rather than treating server-side as universally superior.
• PostHog, Autocapture. Documents interaction, navigation, clipboard, dead-click and lifecycle autocapture, SDK differences, actions for organizing captured events, and the deliberate omission of form values by default for privacy.
• Mixpanel, identity-merge documentation and $identify. Used as an example of the anonymous→known transition and of platform-specific merge behavior that teams must verify in current docs.
• Data-contract and schema-versioning practice (producer/consumer agreements, backward compatibility and deprecation windows), referenced conceptually; not attributed to a single source.
• Related CraftUp owners: Activation Metric (what first value means), Cohort Analysis (comparison design), Retention Strategy (repeated-value system), Product Funnel Analysis (funnel diagnosis), A/B Testing With Low Traffic (feasible experiments), North Star Metric (durable value anchor), KPI Tree Builder (decomposition) and the Metrics & Experiments topic hub.

If the next job is deciding what the metric should be, use the North Star guide or the activation definition method. If the metric is defined and now trustworthy, use the cohort comparison method to compare populations or the funnel guide to locate a drop-off. If the next job is a controlled test, use the A/B Test Plan Generator. And if you are preparing for a metrics interview rather than instrumenting a real decision, the PM metrics interview guide owns that job.