Similarity Features

class neer_match_utilities.similarity_features.SimilarityFeatures(similarity_map)

__init__(similarity_map)

pairs(left, right, matches, left_id_col, right_id_col, match_col='match', matches_id_left='left', matches_id_right='right', matches_are_indices=True)

The rows of pairwise_similarity_dataframe(), built on demand.

Takes the same arguments. Nothing is computed per pair until rows are requested from the returned object, in full, in blocks of left rows, or selected by position.

Return type:

SimilarityPairs

pairwise_similarity_dataframe(left, right, matches, left_id_col, right_id_col, match_col='match', matches_id_left='left', matches_id_right='right', matches_are_indices=True)

Build full cross-join of left × right, compute similarity features specified in SimilarityMap, and attach match indicator.

Parameters:

• left (DataFrame) – Left- and right-hand side entity tables.

• right (DataFrame) – Left- and right-hand side entity tables.

• matches (DataFrame) – DataFrame describing which pairs are true matches. If matches_are_indices=True, matches[matches_id_left] and matches[matches_id_right] are interpreted as row positions in left and right (0..n-1), as neer_match interprets them, whatever the frames’ index labels are. A position outside 0..n-1 raises ValueError. If False, they are interpreted as IDs in the same space as left[left_id_col] / right[right_id_col]. A pair listed more than once is counted once.

• left_id_col (str) – Column names in left / right that contain the entity IDs.

• right_id_col (str) – Column names in left / right that contain the entity IDs.

• match_col (str) – Name of the binary match indicator column in the output.

• matches_id_left (str) – Column names in matches identifying the left/right side.

• matches_id_right (str) – Column names in matches identifying the left/right side.

• matches_are_indices (bool) – If True (default), treat matches_id_left / matches_id_right as row positions in left and right. If False, treat them as IDs.

Return type:

DataFrame

Notes

The frame holds one row per pair, so its memory grows with len(left) * len(right). pairs() builds the same rows in blocks, or only selected ones.

class neer_match_utilities.similarity_features.SimilarityPairs(similarity_map, left, right, matches, left_id_col, right_id_col, match_col, matches_id_left, matches_id_right, matches_are_indices)

The rows of a pairwise similarity frame, built on demand.

Row k pairs left row k // len(right) with right row k % len(right), as in the full cross join. Everything that does not depend on the pair – the ID lookups, each compared column’s values, the match keys – is derived once here, so building the rows in several calls costs no more than building them in one, and gives the same values.

__init__(similarity_map, left, right, matches, left_id_col, right_id_col, match_col, matches_id_left, matches_id_right, matches_are_indices)

__len__()

Return type:

int

chunks(pairs_per_chunk=250000)

All rows, in consecutive blocks of whole left rows.

Each block holds as many left rows as fit in pairs_per_chunk pairs, and at least one. Its index continues the full frame’s, so concatenated, the blocks equal frame().

Return type:

Iterator[DataFrame]

frame()

All rows, as SimilarityFeatures.pairwise_similarity_dataframe() returns them.

Return type:

DataFrame

match_flags(pairs_per_chunk=250000)

The match column of frame(), without computing any similarity.

Return type:

ndarray

rows(positions)

The rows of frame() at positions, in that order and indexed by them.

Return type:

DataFrame

neer_match_utilities.similarity_features.subsample_non_match_positions(match, mismatch_share=1.0, random_state=None, shuffle=True)

Positions of the rows subsample_non_matches() keeps, in its order.

subsample_non_matches(df, ...) equals df.iloc[positions] for positions = subsample_non_match_positions(df[match_col].to_numpy(), ...) and the same remaining arguments. Only the match column is needed, so the similarity features can be computed afterwards for the kept rows alone.

Both draws are made as pandas’ DataFrame.sample makes them: an integer seed starts a fresh numpy.random.RandomState for each, None uses numpy’s global one, and round(mismatch_share * n) non-matches are kept. Calling numpy directly keeps the result independent of pandas’ implementation; numpy holds the RandomState stream fixed across versions (NEP 19).

Parameters:

• match (ndarray) – Binary match indicator per row (1 = match).

• mismatch_share (float) – As for subsample_non_matches().

• random_state (int | None) – As for subsample_non_matches().

• shuffle (bool) – As for subsample_non_matches().

Returns:

Row positions: all matches, then the sampled non-matches, shuffled together if shuffle is True.

Return type:

ndarray

neer_match_utilities.similarity_features.subsample_non_matches(df, match_col='match', mismatch_share=1.0, random_state=None, shuffle=True)

Return a subsample of df where all matches are kept and a fraction of non-matches is sampled.

Parameters:

• df (DataFrame) – DataFrame with a binary match column.

• match_col (str) – Name of the binary target column (1 = match, 0 = non-match).

• mismatch_share (float) – Share of non-matches to keep. Must satisfy 0 < mismatch_share <= 1.0. - 1.0 → keep all non-matches - 0.1 → keep 10% of non-matches

• random_state (int | None) – Random seed for reproducible sampling.

• shuffle (bool) – If True, shuffle the resulting DataFrame.

Returns:

Subsampled DataFrame with all matches and a subset of non-matches.

Return type:

DataFrame

neer_match_utilities.similarity_features.to_X_y(df, match_col='match')

Extract (X, y) from a pairwise similarity DataFrame.

Parameters:

• df (DataFrame) – DataFrame produced by SimilarityFeatures.pairwise_similarity_dataframe().

• match_col (str) – Name of the binary match indicator column.

Returns:

• X (pd.DataFrame) – Feature matrix containing all similarity columns (col_*).

• y (np.ndarray) – Target array (0/1).

• Index

• Module Index