Split

exception neer_match_utilities.split.SplitError[source]

Custom exception for errors in data splitting.

neer_match_utilities.split.split_test_train(left, right, matches, test_ratio=0.3, validation_ratio=0.1, random_state=None, keep_components=False)[source]

Splits datasets into training, validation, and testing subsets.

This function ensures that only observations from left and right that are referenced in the matches DataFrame are included in the split process.

Parameters:

  • left (DataFrame) – The left dataset to split.

  • right (DataFrame) – The right dataset to split.

  • matches (DataFrame) – A DataFrame containing matching pairs between the left and right datasets. It must include columns ‘left’ and ‘right’, referencing indices in left and right.

  • test_ratio (float) – The proportion of the data to be used for testing (default is 0.3).

  • validation_ratio (float) – The proportion of the data to be used for validation (default is 0.1).

  • random_state (int | None) – Seed for the sampling of the three subsets. Pass an integer to obtain a reproducible split, which is required to re-create a training run. If None (default), the split is drawn from unseeded entropy and differs on every call.

  • keep_components (bool) – If False (default), the match pairs are drawn one by one. If True, pairs that share a left or a right record, directly or through other pairs, stay in the same subset. Use it when one entity has several pairs, for example links along a chain of volumes (1922 -> 1925 -> 1926) completed by SetupData.data_preparation_cs: drawn one by one, the pairs of a chain end up in different subsets, and inside a subset a true match whose pair went to another subset counts as a non-match. Whole groups fill the test subset, then the validation subset, in random order, so these two can exceed their requested size by less than one group; the training subset takes the rest. No left or right record then appears in two subsets.

Returns:

A tuple containing: - left_train : pd.DataFrame - right_train : pd.DataFrame - matches_train : pd.DataFrame - left_validation : pd.DataFrame - right_validation : pd.DataFrame - matches_validation : pd.DataFrame - left_test : pd.DataFrame - right_test : pd.DataFrame - matches_test : pd.DataFrame

Return type:

tuple

Raises:

SplitError – If the total counts of split subsets do not match the original dataset size.