AI Lab — Experiment

"We're seeing a lot of trial signups for our reporting tool, but only 22% of trials connect a data source in the first session. I think the setup flow is too long before people see any value. Help me turn this into an experiment."

Experiment Brief

Connect-First Onboarding

Only 22% of trial users connect a data source in their first session, and connecting is strongly linked to converting to paid. We'll test whether moving the data source connection to the start of onboarding, and asking the profile questions afterward, increases the share of trials that connect.

Problem

Most trial users never connect a data source. Without data, they can't see the product's core value, and they rarely convert to paid. (Known)

Context

• The current flow is: segment step → agency name and URL → intent → client count → connect data source. (Known)
• Some users connect in a later session, but most who don't connect in the first session never do. (Known)
• The product is a reporting tool, and the profile questions suggest the audience is agencies. (Known / inferred)

Evidence

• 22% of trials connect a data source in the first session. (Known)
• Connecting a data source is associated with converting to paid. (Known) — a correlation, not proof of cause.
• Gap: no evidence yet that flow length is the cause — no drop-off data, research, or feedback. (Needs validation)

Desired Outcome

More trial users connect a data source early, which should lead to more trial-to-paid conversions.

Hypothesis

(Suggested) If we move the data source connection to the start of onboarding for new trial signups, and ask the profile questions afterward, then the share of trials that connect a data source will increase — because users currently answer four questions that give them nothing back before reaching the step that leads to value.

Supporting Hypotheses

(Suggested, candidates for follow-up tests, not part of this experiment)

• Sample data first. Users who aren't ready to connect would engage more if they could see a sample-data dashboard before connecting.
• Invite a teammate. Some sign-ups don't have credentials or permission to connect. Letting them invite a colleague who does would recover part of the drop-off.

Experiment

An A/B test on the onboarding flow. Only the order of steps and when profile questions appear changes — the connection step, its UI, and available data sources stay the same, so a clean result can be attributed to reordering.

Target Audience

New trial signups entering onboarding after launch. (Suggested) Exclude existing accounts, internal/test accounts, and invite-based joins.

Control

The current flow: segment → agency name and URL → intent → client count → connect data source.

Variant

The data source connection comes first, after any fields required to create the account. The four profile questions come after, either in the same session or as a short prompt later.

Open question: does anything in the profile steps affect the connection step (e.g. does segment change which data sources show)? If so, that step may need to stay first, or the connection screen needs a default.

Experiment Type

(Suggested) A/B test, 50/50 split at signup — trial volume appears meaningful and the change is low-risk and easy to reverse.

Primary Metric

(Suggested) The percentage of trials that successfully connect a data source within 7 days of signup.

• "Successfully connect" should mean the connection worked, not just that the user clicked "connect" — otherwise errors can inflate the number.
• A 7-day window catches users who return later, adding safety without much delay since most users don't come back after that.

Secondary Metrics

• First-session connection rate. Current benchmark — shows whether connections happen earlier.
• Trial-to-paid conversion. The real business outcome; treat as a directional signal, may lack power.
• Step-by-step drop-off in both versions — fills the evidence gap.
• Time from signup to first connection.

Guardrail Metrics

• Profile question completion rate in the variant — agree beforehand on acceptable data loss.
• Connection error or failure rate.
• Support contacts during onboarding.

Success Criteria

(Suggested, confirm before launch) The variant succeeds if the 7-day connection rate improves by at least a pre-agreed minimum, with statistical significance, and no guardrail gets meaningfully worse.

Decision Framework

• Connection up, guardrails OK, paid conversion flat/up → ship, then test sample-data or invite-a-teammate.
• Connection up, but conversion down or data loss unacceptable → iterate on which questions come upfront.
• No meaningful change → check drop-off data; move to sample-data or invite-a-teammate tests.
• Connection down → stop and look at where variant users drop off.

Risks & Limitations

• The cause might be elsewhere — without drop-off data, reordering may miss the actual problem.
• More connections from lower-intent users may not produce more paid customers.
• Losing profile data could affect sales follow-up or personalized onboarding.
• May lack power to detect changes in trial-to-paid conversion.
• Timing effects — run for full weeks to avoid seasonal or campaign skew.

Instrumentation & Analytics

• Viewed / completed / abandoned events for each onboarding step.
• A distinct "successful connection" event, separate from clicking "connect."
• Connection error records.
• Experiment version stored on the account for later analysis.
• Sample size (needs validation) — illustratively, ~1,800 trials per version to detect 22%→26%, ~1,150 per version for 22%→27%.

Design & Engineering Requirements

• Reorder the flow behind a feature flag with random assignment.
• Design where/how profile questions appear after connection.
• Handle any dependency of the connection step on profile answers.

Open Questions

• Where do users actually drop off today? (needs validation)
• Which profile fields do sales/onboarding truly need, and how soon?
• Does any profile answer change the connection experience?
• What is the weekly trial volume?
• What improvement would be worth shipping?

Next Steps

• Pull step-by-step drop-off data and use it to confirm or rethink this test.
• Confirm with sales/onboarding which profile data is essential.
• Agree on the minimum improvement worth shipping; estimate test length from trial volume.
• Add the missing tracking events, then launch.