Blocking

Blocking: which record pairs a model looks at.

Without blocking, training, evaluation and prediction take every pair of left x right. A Blocking rule keeps only the candidate pairs that agree on some columns (exact), lie close on others (within) or have similar strings (min_similarity), as rule-based record linkage does, e.g. the same sex, birth parish and birth year within two years.

The same rule serves everywhere, so that a model is trained on the kind of pairs it later scores:

• BaselineTrainingPipe takes its non-matches for fitting from the candidates only and scores only them. A true match that the rule excludes counts as missed (a false negative); the other excluded pairs count as correctly rejected. The metrics therefore describe blocking and model together, on all pairs, and add Pairs, Candidates and BlockingRecall (the share of the true matches among the candidates).

• suggest of the baseline models and FeatureSelector take blocking too, and GenerateID passes it on to suggest.

The rule is passed, not a list of pairs: each split computes its own candidates. Instead of a Blocking, any function f(left, right) that returns the candidate pairs as row positions (columns left and right) can be passed.

class neer_match_utilities.blocking.Blocking(exact=(), within=dict(), min_similarity=dict())

A rule for the candidate pairs of left x right.

A pair is a candidate if it meets every condition given. Columns are named as in a similarity map: "col" for a column of that name on both sides, "left_col~right_col" for differently named ones. A record with a missing value in a column the rule uses has no candidates. Values are compared as they are in the frames passed (e.g. after Prepare.format).

Parameters:

• exact (Sequence[str]) – Columns whose values must be equal.

• within (Mapping[str, float]) – Numeric columns and the largest absolute difference allowed, e.g. {"birth_year": 2}. A difference of 0 is the same as exact.

• min_similarity (Mapping[str, float]) – String columns and the smallest Jaro-Winkler similarity allowed (0 to 1), e.g. {"last_name": 0.7}, compared on str values. Checked only on the pairs that pass exact and within; given alone, it compares every pair.

Examples

>>> Blocking(exact=["sex", "birth_parish"], within={"birth_year": 2})
>>> Blocking(exact=["county"], within={"birth_year": 2}, min_similarity={"last_name": 0.7})

__init__(exact=(), within=<factory>, min_similarity=<factory>)

candidates(left, right)

The candidate pairs as row positions, columns left and right, sorted.

Return type:

DataFrame

columns()

The columns the rule reads in left and in right.

Return type:

tuple[list[str], list[str]]

neer_match_utilities.blocking.blocked_metrics(y_true, y_hat, n_pairs, n_matches)

Metrics of blocking and model together, on all n_pairs pairs.

y_true and y_hat hold the candidates’ labels and predictions. The pairs outside the candidates count as predicted non-matches: the n_matches - y_true.sum() true matches among them as false negatives. Adds Pairs, Candidates and BlockingRecall to the counts and rates of classification_metrics().

Return type:

dict

neer_match_utilities.blocking.blocking_columns(blocking)

The columns a Blocking reads; None for a function, which may read any.

Return type:

tuple[list[str], list[str]] | None

neer_match_utilities.blocking.candidate_pairs(left, right, blocking)

The candidate pairs of left x right under blocking, as row positions.

Returns None for blocking=None (every pair is a candidate). A function passed as blocking must return columns left and right with row positions 0..n-1 of the two frames; the pairs are deduplicated and sorted.

Return type:

DataFrame | None

neer_match_utilities.blocking.tune_threshold_blocked(labels, probabilities, n_pairs, n_matches, metric='mcc', thresholds=None)

The threshold that maximises metric of blocking and model together.

As tune_threshold(), on the candidates’ labels and probabilities, with the pairs outside the candidates counted as in blocked_metrics(). Of equally good thresholds the lowest wins.

Return type:

tuple[float, dict]

• Index

• Module Index