surface↓entity resolution↓dedupe_llm
Entity Resolutionaggregate · returns varchar

DEDUPE_LLM

LLM-backed deduplication (escape hatch for DEDUPE)

Per-group — reads the whole group in one call.

entity-resolutionllmllm-escape-hatchscales-largejson

Syntax

DEDUPE_LLM({{ texts }})
DEDUPE_LLM({{ texts }}, '{{ criteria }}')

Arguments

nametypedescription
textsJSON—
criteriaVARCHAR—

About

LLM-backed escape hatch for semantic deduplication. Use when the canonical embedding-threshold path (DEDUPE) is insufficient because the dedup requires world knowledge or nuanced judgment — e.g., knowing that "IBM" and "International Business Machines" are the same entity, or handling abbreviations and acronyms that don't cluster well in the embedding space. For routine fuzzy dedup (typos, variant spellings, name forms), prefer DEDUPE — it is 100-1000x faster and scales to 10K+ values.

Examples

LLM escape hatch dedups abbreviations with expanded forms

WITH
  test_data AS (
    SELECT
      *
    FROM
      (
        VALUES
          ('IBM'),
          ('International Business Machines'),
          ('Apple Inc.'),
          ('Apple Computer')
      ) AS t (name)
  )
SELECT
  DEDUPE_LLM (name)
FROM
  test_data

Nearby rabbit holes

same domain
aggregate

DEDUPE

Deduplicates texts by semantic similarity (embeddings + threshold graph)

entity-resolutionembedding-modelspecialist-zooscales-large+1
aggregate

GOLDEN_RECORD

Merge duplicate records into a composite golden record

entity-resolutionllmtext
aggregate

MERGE_RECORDS

Merge records with an explicit conflict-resolution strategy

entity-resolutionllmtext
scalar

SAME_AS

Check if two values refer to the same entity (fuzzy equality)

entity-resolutionllmtext
Climb back to The Looking Glass