Baseline Models

class neer_match_utilities.baseline_models.GradientBoostingModel(model=<factory>)

Gradient boosting baseline on similarity features using scikit-learn.

Designed as an alternative to the DL/NS models in neer_match, using a tree-based GradientBoostingClassifier on top of similarity features produced by AlternativeModels.

It supports: - evaluation with TP, FP, TN, FN, Accuracy, Recall, Precision, F1, MCC, - a simple summary() reporting feature importances.

Notes

• Unlike Logit/Probit, this model has no statistical inference (SE/p-values).

• Works well with nonlinearities and interactions in similarity features.

__init__(model=<factory>)

best_threshold(df_val, match_col='match', feature_prefix='col_', metric='mcc', thresholds=None, store_treshold=True)

Find the classification threshold that maximizes a metric on validation data.

Parameters:

• df_val (DataFrame) – Validation DataFrame produced by AlternativeModels.pairwise_similarity_dataframe().

• match_col (str) – Target column.

• feature_prefix (str) – Feature column prefix.

• metric (str) – Metric to maximize.

• thresholds (ndarray | None) – Threshold grid. If None, uses np.linspace(0.01, 0.99, 99).

Return type:

tuple[float, dict]

Returns:

• best_t (float) – Threshold achieving the best metric on df_val.

• best_stats (dict) – Evaluation dict (TP/FP/TN/FN/Accuracy/Recall/Precision/F1/MCC) at best_t.

evaluate(df, match_col='match', feature_prefix='col_', threshold=0.5)

Evaluate the model on a pairwise similarity DataFrame.

Returns a dict:

• TP, FP, TN, FN (integers)

• Accuracy, Recall, Precision, F1, MCC (floats)

Return type:

dict

fit(df, match_col='match', feature_prefix='col_', use_class_weight=False)

Fit gradient boosting on a pairwise similarity DataFrame.

Parameters:

• df (DataFrame) – (Possibly subsampled) DataFrame produced by AlternativeModels.

• match_col (str) – Name of the binary target column.

• feature_prefix (str) – Prefix of feature columns (similarity features).

• use_class_weight (bool) – If True, uses inverse-frequency sample weights to upweight matches. Useful if you fit on a very imbalanced dataset.

Return type:

GradientBoostingModel

predict_proba(df, feature_prefix='col_')

Predict match probabilities for a pairwise similarity DataFrame.

Returns the probability for the positive class (match = 1).

Return type:

ndarray

summary(top_k=20)

Return a simple “summary” as a DataFrame of feature importances.

Parameters:

top_k (int) – Number of most important features to return.

Returns:

Columns: feature, importance

Return type:

DataFrame

class neer_match_utilities.baseline_models.LogitMatchingModel

Logistic regression baseline on similarity features using statsmodels.

This class is designed as an alternative to the DL/NS models in neer_match, using statsmodels’ Logit on top of the similarity features produced by AlternativeModels.

It supports: - evaluation with TP, FP, TN, FN, Accuracy, Recall, Precision, F1, MCC, - full inference via summary().

__init__()

fit(df, match_col='match', feature_prefix='col_')

Fit logistic regression on a pairwise similarity DataFrame.

Parameters:

• df (DataFrame) – (Possibly subsampled) DataFrame produced by AlternativeModels.

• match_col (str) – Name of the binary target column.

• feature_prefix (str) – Prefix of feature columns (similarity features).

Return type:

LogitMatchingModel

class neer_match_utilities.baseline_models.ProbitMatchingModel

Probit regression baseline on similarity features using statsmodels.

Same interface as LogitMatchingModel, but using a normal CDF link.

__init__()

fit(df, match_col='match', feature_prefix='col_')

Fit probit regression on a pairwise similarity DataFrame.

Parameters:

• df (DataFrame) – (Possibly subsampled) DataFrame produced by AlternativeModels.

• match_col (str) – Name of the binary target column.

• feature_prefix (str) – Prefix of feature columns (similarity features).

Return type:

ProbitMatchingModel

class neer_match_utilities.baseline_models.StatsmodelsInferenceMixin

Keeps a fitted model usable without its statsmodels Results object.

Two problems motivate this. First, a Results object holds a reference to its Model, and thereby to endog / exog – the full training data. Pickling it stores the entire design matrix, which in entity matching is the complete left x right cross join. Second, a pickled Results contains pandas objects (params is a Series, normalized_cov_params a DataFrame), and pandas pickles are not portable across pandas versions – an archive written under one version can be unreadable under another.

capture_inference() therefore extracts everything needed for later use – the coefficient vector, the parameter names and the regression table – into plain Python and numpy types. neer_match_utilities.baseline_io.ModelBaseline() stores those in meta.pkl, and a model reconstructed from them alone can still predict, score and report its summary. The statsmodels pickle is then an optional extra rather than a dependency.

predict_proba still delegates to the Results object whenever one is present, so behavior is unchanged for freshly fitted models and for archives that load cleanly. The pandas-free path engages only when the result is unavailable.

capture_inference()

Extract the fitted model into plain types, while the training data is still attached.

Stores the coefficient vector, the parameter names and the regression table. These are what ModelBaseline writes to meta.pkl, and what a reloaded model falls back on when the statsmodels pickle is unavailable.

Return type:

None

coefficients()

Coefficients, standard errors, z-statistics, p-values and confidence intervals.

Return type:

DataFrame

predict_proba(df, feature_prefix='col_')

Predict match probabilities.

Delegates to the statsmodels result when one is present. Otherwise applies the model’s link function to the linear predictor built from the stored coefficients, which gives the same answer without requiring the pickled result.

Return type:

ndarray

summary()

The regression table.

Returns the live statsmodels Summary while the result still carries its data, and the summary captured at save time once it does not.

class neer_match_utilities.baseline_models.SuggestMixin

Adds a NeerMatch-like .suggest(left, right, count, verbose) API to baseline models.

Requires:

• self.predict_proba(df_pairs) implemented

• self.similarity_map set to a SimilarityMap (or dict) describing features to compute

suggest(left, right, *, count=10, verbose=0, left_id_col=None, right_id_col=None)

Return top-k candidate matches per left record (like neer_match DL models).

Return type:

DataFrame

Output columns:

• left: integer row index into left (0..len(left)-1)

• right: integer row index into right (0..len(right)-1)

• prediction: match probability

• Index

• Module Index