surface↓analysis↓ab_test
Analysisscalar · returns json

AB_TEST

Bayesian A/B test (Beta-Binomial) returning full posterior + recommendation

Per-row — runs once for each row.

analysisdeterministicnumeric

Arguments

nametypedescription
variantsJSONArray of variant labels (use ARRAY_AGG in caller)
conversionsJSONArray of binary outcomes (parallel to variants)

About

Run a Bayesian A/B test directly in SQL. Point it at a column of variant labels and a column of 0/1 conversion outcomes and get back the probability each variant is better, the expected lift, and a 95% credible interval — all in one aggregate call. Useful for quick, honest experiment analysis without exporting data to a notebook: conversion-rate tests, button-click tests, plan-selection experiments, feature-flag comparisons. Because it returns a full posterior rather than just a p-value, you can make calls like "B is 92% likely to be better, good enough to ship" without fighting significance thresholds. Runs in-process with scipy — no LLM, no zoo, no external service, no cost beyond CPU. Works best on binary outcomes (did they convert: yes/no). For conversion metrics (binary 0/1 outcomes), this uses a Beta-Binomial conjugate prior. With a flat Beta(1, 1) prior, the posterior for each variant is Beta(1 + successes, 1 + failures). We then Monte-Carlo sample from both posteriors and compute: • P(B > A) — probability that B's true rate is higher • expected lift — posterior mean of (rate_B - rate_A) / rate_A • credible interval — 95% HDI on the lift This gives growth teams the answer they actually want in one SQL call instead of a hand-rolled scipy notebook: "did the experiment win, how much, and how confident are we?" Usage (user-side aggregation with ARRAY_AGG gives guaranteed alignment between the two parallel columns): SELECT ab_test( ARRAY_AGG(variant), ARRAY_AGG(converted) ) AS result FROM experiment_log WHERE experiment_id = 'homepage_hero_v3'; With a GROUP BY, you get one posterior per experiment in one query: SELECT experiment_id, ab_test(ARRAY_AGG(variant), ARRAY_AGG(converted)) AS result FROM experiment_log GROUP BY experiment_id;

Examples

Two-arm test with obvious winner

WITH
  log AS (
    SELECT
      *
    FROM
      (
        VALUES
          ('A', TRUE),
          ('A', FALSE),
          ('A', FALSE),
          ('A', FALSE),
          ('A', FALSE),
          ('A', TRUE),
          ('A', FALSE),
          ('A', FALSE),
          ('A', FALSE),
          ('A', TRUE),
          ('B', TRUE),
          ('B', TRUE),
          ('B', FALSE),
          ('B', TRUE),
          ('B', TRUE),
          ('B', TRUE),
          ('B', FALSE),
          ('B', TRUE),
          ('B', TRUE),
          ('B', TRUE)
      ) AS t (variant, converted)
  )
SELECT
  ab_test (ARRAY_AGG(variant), ARRAY_AGG(converted)) AS result
FROM
  log;

Nearby rabbit holes

same domain
scalar

ASSESS_CONFIDENCE

Assess the confidence/quality of a cascade execution result.

analysisllmtext
aggregate

COMPARE

Compare values in a group for similarities, differences, and patterns

analysisllmtext
scalar

COUNTERARGUMENT

Generate the strongest counterargument to a position

analysisllmtext
scalar

DP_COUNT

Differentially-private count (Laplace mechanism, sensitivity=1)

analysisdeterministicjson
scalar

DP_MEAN

Differentially-private mean (Laplace mechanism)

analysisdeterministicjson
scalar

FALLACY

Detect logical fallacies in an argument

analysisllmtext
Climb back to The Looking Glass