Embedding Similarity
Similarity of field values from a pretrained text encoder (sentence embeddings).
Each distinct value of the chosen columns is encoded once by a pretrained model
(by default BAAI/bge-m3, multilingual); the similarity of two values is the
cosine of their vectors. It is registered like any other similarity, so it can
be named in the similarity map, and then the logit, probit and gradient-boosting
models (through SimilarityFeatures),
the neural network of neer_match (through its SimilarityEncoder) and the
feature selection all use it.
The vectors are kept in a cache folder. Encoding needs the optional packages
sentence-transformers and torch (pip install
"neer-match-utilities[embeddings]"); reading a cache needs only numpy, so the
models can be trained in an environment without torch, and the cache is what a
replication package stores.
Encoding runs in a Python process of its own (_encode_worker.py), which
loads neither this package nor TensorFlow: on Linux with a GPU, TensorFlow and
triton (loaded by sentence-transformers) cannot be loaded into one process in
this order.
- class neer_match_utilities.embeddings.EmbeddingSimilarity(cache_dir, model_name='BAAI/bge-m3', *, revision=None, similarity_name='embedding_cosine', batch_size=64, max_seq_length=512, device=None, fp16=False, cuda_visible_devices=None, encoder=None)[source]
Cosine similarity of sentence embeddings, available to neer_match by name.
- Parameters:
cache_dir (
str|Path) – Folder for the vectors (vectors.npy), the encoded values (values.json) and the model they come from (meta.json). If it holds a cache already, that cache is read, and only values not in it are encoded later. The values are the data themselves: store the folder under the same terms as the data.model_name (
str) – Model on the Hugging Face Hub, or a local path, loaded withsentence_transformers.SentenceTransformer. Default"BAAI/bge-m3". A cache made with another model raises ValueError.revision (
str|None) – Commit of the model to use. Default None: the commit recorded in the cache, or else the current one on the Hub, which is then recorded.similarity_name (
str) – Name under which the similarity is registered and used in the similarity map. Default"embedding_cosine". Use one name per model if several are compared.batch_size (
int) – Values encoded at once. Default 64.max_seq_length (
int) – Tokens per value; longer values are cut. Default 512 (bge-m3 accepts up to 8192, which field values rarely need).device (
str|None) – Device for encoding, e.g."cuda"or"cpu". Default None: the GPU if one is available.fp16 (
bool) – Encode in half precision (GPU only). Default False.cuda_visible_devices (
str|None) – GPUs the encoding process may use, as in the environment variableCUDA_VISIBLE_DEVICES(e.g."1"). Default None: the same GPUs as this process. Lets this process hide the GPUs from TensorFlow (CUDA_VISIBLE_DEVICES=), which otherwise may take a GPU’s memory when the package is imported, while the model encodes on one.encoder (
Optional[Callable[[list],ndarray]]) – Instead of the model: a function that takes a list of strings and returns one vector per string (e.g. for tests or another library). Default None.
Notes
A value is looked up exactly as it appears in the data (as
str(value)), so encode the columns in the form they have when the similarities are computed, i.e. afterPrepare.format, and encode again after adding rows, e.g. withsynth_matchesorsynth_mismatches: only the new values are encoded. A value that was not encoded raises KeyError.Missing values, and strings that are empty, have similarity 0, as in
SimilarityFeatures.The cosine is cut to [0, 1], the range of the other similarities.
The vectors are normalized and kept as float32: 4 bytes times the dimension (1024 for bge-m3) per distinct value.
To use a saved model later, create the object again from the cache before predicting, so that the similarity is registered.
Each call of
encode()that finds new values starts a Python process that loads the model, encodes them and ends (see the module notes); encode all frames in one call where possible.
- __init__(cache_dir, model_name='BAAI/bge-m3', *, revision=None, similarity_name='embedding_cosine', batch_size=64, max_seq_length=512, device=None, fp16=False, cuda_visible_devices=None, encoder=None)[source]
- property dim: int
Length of the vectors (0 before anything is encoded).
- encode(*frames, columns)[source]
Encode the values of columns in frames that are not in the cache yet.
- Parameters:
*frames (
DataFrame) – Data whose values are encoded, e.g.left, right. Each must have all columns; if the left and right names differ, call once per frame.columns (
Sequence[str]) – Columns to encode.
- Returns:
The object itself; the cache folder is updated if anything was new.
- Return type:
- similarity(x, y)[source]
Cosine of the vectors of x and y, cut to [0, 1]; 0 if either is missing.
- Return type:
float
- property vectors: ndarray
The normalized vectors, one row per encoded value.