Travis-CI allows for the creation of aarch64 wheels.

Build: https://travis-ci.com/github/janaknat/nmslib/builds/205780637

There are 8-9 failures when testing hnsw. Any suggestions on how to fix these? A majority of the failures are due to expected=0.99 and calculated=~0.98.

Tagging @jmazanec15 since he added ARM compatibility.

Currently pip installing is slow, since there is a compile step. Is there any way to speed it up? On my macbook:

time pip install --no-cache nmslib
Collecting nmslib
  Downloading https://files.pythonhosted.org/packages/e1/95/1f7c90d682b79398c5ee3f9296be8d2640fa41de24226bcf5473c801ada6/nmslib-1.7.3.6.tar.gz (255kB)
    100% |████████████████████████████████| 256kB 8.8MB/s 
Requirement already satisfied: pybind11>=2.0 in .../virtualenv/python3.6/lib/python3.6/site-packages (from nmslib) (2.2.4)
Requirement already satisfied: numpy in .../virtualenv/python3.6/lib/python3.6/site-packages (from nmslib) (1.15.4)
Installing collected packages: nmslib
  Running setup.py install for nmslib ... -
done
Successfully installed nmslib-1.7.3.6

real	3m11.091s

would it be a good idea to provide pre-compiled wheels over pip? That would also simplify the process of finding the pybind11 headers (I had to do something special to copy them in for pip when running with a --target dir)

Hi, this might me more of a question than problem in the library. I have created an index with NAPP and saved it using saveIndex. However when I load it with loadIndex I get the following error:

Check failed: A previously saved index is apparently used with a different data set, a different data set split, and/or a different gold standard file! (detected an object index >= #of data points

Am I doing something wrong?

Thanks for the help.

EDIT: The message doesn't make sense to me because I'm not "using the index with a data set", I'm just loading it.

EDIT2: I'm using the Python interface.

Hello,

I wanted to perform NN search on a dataset of genomes. For this task, the distance between 2 datapoints is calculated by a custom script? Is there I can incorporate this without having to create the entire NN search algorithm myself and only modify some parts of your code?

nmslib is the only lib in our project that relies on pybind11 and we could narrow it down to the Dask nodes that use nmslib. When we disable the nodes that use nmslib it doesn't crash.

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  TypeError: '>=' not supported between instances of 'int' and 'NoneType'

At:
  /opt/conda/envs/jobnet-env/lib/python3.6/logging/__init__.py(1546): isEnabledFor
  /opt/conda/envs/jobnet-env/lib/python3.6/logging/__init__.py(1293): debug

/usr/local/bin/entrypoint.sh: line 46:    21 Aborted                 (core dumped) python scripts/cli.py "${@:2}"```

Version:

- nmslib~=1.7.2
- pybind11=2.2

Hello,

I am facing an error in this step:

[ 75%] Linking CXX executable ../release/experiment

All of errors liked that:

undefined reference to `boost::program_options:

I install latest libraries version and checked that libboost 1.58 is compatible with g++ 4.9. I think maybe it related with C++11, however It returns error in both g++ 4.9 and 4.7.

This is my system information:

-- The C compiler identification is GNU 4.9.3 -- The CXX compiler identification is GNU 4.9.3 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Build type: Release -- GSL using gsl-config /usr/bin/gsl-config -- Using GSL from /usr -- Found GSL. -- Found Eigen3: /usr/include/eigen3 (Required is at least version "3") -- Found Eigen3. -- Boost version: 1.58.0 -- Found the following Boost libraries: -- system -- filesystem -- program_options -- Found BOOST.

I also install Clang and LLDB 3.6. I tried search many possible solution but can not fix that :(.

Hi, I am working on a problem where I need to retrieve ~500 nearest neighbors out of a million points. I am using the python wrapper for HNSW method. The code works perfectly well if I set the value of parameter M <=100 but setting it greater than 100, the code crashes during retrieving nearest neighbors (no issues while building the model) with an "invalid next size" error. Any idea why this might be happening? Thanks Himanshu

An all-zero query vector will result in NMSLib incorrectly reporting a distance of zero for its nearest neighbours (see example below). Is this related to #187? Is there a suggested workaround?

# Training set (CSR sparse matrix)
X.todense()
# Out:
# matrix([[4., 2., 3., 1., 0., 0., 0., 0., 0.],
#         [2., 1., 0., 0., 3., 0., 1., 2., 1.],
#         [4., 2., 0., 0., 3., 1., 0., 0., 0.]], dtype=float32)

# Query vector (CSR sparse matrix)
r.todense()
# Out:
# matrix([[0., 0., 0., 0., 0., 0., 0., 0., 0.]], dtype=float32)

# Train and query
import nmslib
index = nmslib.init(
    method='hnsw',
    space='cosinesimil_sparse_fast',
    data_type=nmslib.DataType.SPARSE_VECTOR,
    dtype=nmslib.DistType.FLOAT)
index.addDataPointBatch(X)
index.createIndex()
index.knnQueryBatch(r, k=3)
# Out:
# [(array([2, 1, 0], dtype=int32), array([0., 0., 0.], dtype=float32))]

# Note that distances are all 0, which is incorrect!
# Same result for dense training & query vectors.

Hi,

I want to know if HSNW provides Jaccard (similarity or distance, does not matter), besides cosine, for sparse features. There are scenarios in which Jaccard outperforms.

Python notebooks provided show the following metrices: l2, l2sqr_sift, cosinesimil_sparse.

According to space_sparse_scalar.h, the following metrices seem to be implemented, or in preparation, to sparse features: #define SPACE_SPARSE_COSINE_SIMILARITY "cosinesimil_sparse" #define SPACE_SPARSE_ANGULAR_DISTANCE "angulardist_sparse" #define SPACE_SPARSE_NEGATIVE_SCALAR "negdotprod_sparse" #define SPACE_SPARSE_QUERY_NORM_NEGATIVE_SCALAR "querynorm_negdotprod_sparse"

What does each of these metrices mean? I also saw cosinesimil_sparse_fast in a few files. What is it, and how is it compared to cosinesimil_sparse? Is it ready for use?

I can provide a Jaccard implementation for sparse vectors, given 2 vectors implemented as hash tables, but I haven't found out how to integrate it to the code. It would also be preferable to check which metrices are already available. The closest clue I got was to expand the following files: distcomp_scalar.cc, hnsw.cc and hnsw_distfunc_opt.cc, but I am not sure which steps to make. I saw some mentions to Jaccard in space_sparse_jaccard.cc and distcomp.h. But no examples are given.

Thanks in advance.

I'm trying to install python bindings on Ubuntu 16.04 machine:

$ pip3 install pybind11 nmslib
Collecting nmslib
  Using cached https://files.pythonhosted.org/packages/de/eb/28b2060bb1750426c5618e3ad6ce830ac3cfd56cb3eccfb799e52d6064db/nmslib-1.7.2.tar.gz
Requirement already satisfied: pybind11>=2.0 in /homes/alexandrov/.virtualenvs/pytorch/lib/python3.5/site-packages (from nmslib) (2.2.2)
Requirement already satisfied: numpy in /homes/alexandrov/.virtualenvs/pytorch/lib/python3.5/site-packages (from nmslib) (1.14.2)
Building wheels for collected packages: nmslib
  Running setup.py bdist_wheel for nmslib ... error
  Complete output from command /homes/alexandrov/.virtualenvs/pytorch/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-0y71oxa4/nmslib/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-916r1rr9 --python-tag cp35:
  running bdist_wheel
  running build
  running build_ext
  creating tmp
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/homes/alexandrov/.virtualenvs/pytorch/include/python3.5m -c /tmp/tmpwekdswov.cpp -o tmp/tmpwekdswov.o -std=c++14
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/homes/alexandrov/.virtualenvs/pytorch/include/python3.5m -c /tmp/tmpyyphh022.cpp -o tmp/tmpyyphh022.o -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  building 'nmslib' extension
  creating build
  creating build/temp.linux-x86_64-3.5
  creating build/temp.linux-x86_64-3.5/nmslib
  creating build/temp.linux-x86_64-3.5/nmslib/similarity_search
  creating build/temp.linux-x86_64-3.5/nmslib/similarity_search/src
  creating build/temp.linux-x86_64-3.5/nmslib/similarity_search/src/method
  creating build/temp.linux-x86_64-3.5/nmslib/similarity_search/src/space
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I./nmslib/similarity_search/include -Iinclude -Iinclude -I/homes/alexandrov/.virtualenvs/pytorch/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/homes/alexandrov/.virtualenvs/pytorch/include/python3.5m -c nmslib.cc -o build/temp.linux-x86_64-3.5/nmslib.o -O3 -march=native -fopenmp -DVERSION_INFO="1.7.2" -std=c++14 -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  nmslib.cc:16:31: fatal error: pybind11/pybind11.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Clearly, pybind11 headers were not installed on my machine. This library is not packaged for apt-get (at least not for Ubuntu 16.04), so I needed to manually install from source.

Would be nice if nmslib install script took care of this.

Basically, this is what I am trying to do

import nmslib

space = 'negdotprod'

vectors = [[1, 2], [3, 4], [5, 6]]

index = nmslib.init(space=space, method='hnsw')
index.addDataPointBatch(vectors)
index.createIndex(
    {'M': 15, 'efConstruction': 200, 'skip_optimized_index': 0, 'post': 0}
)
index.saveIndex('test.index')

new_index = nmslib.init(space=space, method='hnsw')
new_index.loadIndex('test.index')

and it raises

Check failed: totalElementsStored_ == this->data_.size() The number of stored elements 3 doesn't match the number of data points ! Did you forget to re-load data?
Traceback (most recent call last):
  File "8.py", line 15, in <module>
    new_index.loadIndex('test.index')
RuntimeError: Check failed: The number of stored elements 3 doesn't match the number of data points ! Did you forget to re-load data?

If I change space variable to cosinesimil, it works just fine. It seems that data points are not stored, even though hnsw method with skip_optimized_index=0 is used.

Hey sorry to bother you,

I've been trying to download scispacy via pip on windows 10 using python 3.10.0 today and it keeps failing due to errors about nmslib I've tried pip installing nmslib versions: 1.7.3.6 1.8 2.1.1

None of them have worked though, curiously. I've had a long look around scispacys github and yours but nothing I've read has given me any solutions.

I've also flagged it with scispacy on their github. Anyway I have no idea what's going on but just thought I'd let you know. Cheers Kind regards, Chris

Hey, I'm trying to use nmslib's HNSW with a csr_matrix containing sparse vectors.

Creating the index works fine, adding the data and setting query time params too:

    items = ["foo is a kind of thing", "bar is another one", "this bar is a real one!", "I prefer to use a foo"] # etc, len=3000
    similar_items_index = nmslib.init(
        space="cosinesimil_sparse",
        method="hnsw",
        data_type=nmslib.DataType.SPARSE_VECTOR,
        dtype=nmslib.DistType.FLOAT,
    )
    vectorizer = TfidfVectorizer(dtype=np.float32, token_pattern=r"\S+")
    embeddings: csr_matrix = vectorizer.fit_transform(items)
    similar_items_index.addDataPointBatch(embeddings)
    similar_items_index.createIndex({"M": 128, "efConstruction": 32, "post": 2}, print_progress=False)
    similar_items_index.setQueryTimeParams({"ef": 512})

But when I search with knnQueryBatch, all the returned distances are equal to 1:

similar_items_index.knnQueryBatch([query_embedding], 5)[0]

->

Am I missing something in the proper usage of HNSW with sparse vector data?

import csv
from typing import Dict

import nmslib
import numpy as np
from implicit.evaluation import csr_matrix
from sklearn.feature_extraction.text import TfidfVectorizer

CSV_PATH = "/tmp/data/"


def main():
    similar_items_index = nmslib.init(
        space="cosinesimil_sparse",
        method="hnsw",
        data_type=nmslib.DataType.SPARSE_VECTOR,
        dtype=nmslib.DistType.FLOAT,
    )
    items = set()
    ids: Dict[str, int] = {}
    rids: Dict[int, str] = {}
    similarities = {}
    for file in [
        f"{CSV_PATH}/similarity-test.csv",
        f"{CSV_PATH}/similarity-train.csv",
    ]:
        with open(file) as f:
            reader = csv.reader(f, delimiter=",", quotechar="|")
            header = next(reader)
            for i, l in enumerate(reader):
                desc_x = l[header.index("description_x")]
                desc_y = l[header.index("description_y")]
                similar = bool(l[header.index("same_security")])
                id = len(items)
                if desc_x not in items:
                    items.add(desc_x)
                    ids[desc_x] = id
                    rids[id] = desc_x
                    id_x = id
                    id += 1
                else:
                    id_x = ids[desc_x]
                if desc_y not in items:
                    items.add(desc_y)
                    ids[desc_y] = id
                    rids[id] = desc_y
                    id_y = id
                    id += 1
                else:
                    id_y = ids[desc_y]
                if similar:
                    similarities[id_x] = id_y
                    similarities[id_y] = id_x
         print(f"Loaded {len(items)}, total {len(similarities)/2} pairs of similar queries.")
         vectorizer = TfidfVectorizer(dtype=np.float32, token_pattern=r"\S+")
    embeddings: csr_matrix = vectorizer.fit_transform(items)
    print("Embedded items, adding datapoints..")
    similar_items_index.addDataPointBatch(embeddings)
    print("Creating index..")
    similar_items_index.createIndex({"M": 128, "efConstruction": 32, "post": 2}, print_progress=False)
    print("Setting index query params..")
    similar_items_index.setQueryTimeParams({"ef": 512})
    print("Searching...")
    score = 0
    total_similar = 0
    for item_id, item in enumerate(items):
        query_embedding = vectorizer.transform([item]).getrow(0).toarray()
        top_50, distances = similar_items_index.knnQueryBatch([query_embedding], 50)[0]
        top_50_texts = [rids[t] for t in top_50]
        try:
            expected = similarities[item_id]
            expected_text = rids[expected]
            if expected:
                score += 1 if expected in top_50 else 0
        except KeyError:
            continue  # No similar noted on this item.
        total_similar += 1
    print(
        f"After querying {len(items)} of which {total_similar}, we found the similar item in the top50 {score} times."
    )


if __name__ == "__main__":
    main()

On a Mac architecture, platform.processor may return i386 even when on a Mac M1. The code below should be more accurate. See stack overflow comment, another stack overflow comment and stack overflow post for some more information / validation that the uname approach is more all encompassing.

I was personally running into this problem and the following fix solved it for me.

This PR is a slightly edited solution to what is contained in https://github.com/nmslib/nmslib/pull/485 with many thanks to @netj for getting this started.

Hi there! Trying to perform knnQuery on an indexed csr_matrix, I got the issue reported in #480 from this code:

        model = TfidfVectorizer(dtype=np.float32, token_pattern=r"\S+")
        embeddings = model.fit_transform(corpus_tfidf)
        logger.info(f"Creating vector index from a {len(corpus_tfidf)} corpus embedded as {embeddings.shape}...")
        index = nmslib.init(method="hnsw", space="cosinesimil_sparse", data_type=nmslib.DataType.SPARSE_VECTOR, dtype=nmslib.DistType.FLOAT)
        logger.info("Adding datapoints to index...")
        index.addDataPointBatch(embeddings)
        logger.info("Creating final index...")
        index.createIndex()

        logger.info(f"Search neightbors for first embedding {embeddings[0]})
        index.knnQuery(embeddings[0])

As described in #480, this results in an IndexError: tuple index out of range.

When trying to apply the index.setQueryTimeParams({'efSearch': efS, 'algoType': 'old'}) workaround mentioned in another issue , it results in a segmentation fault.

I can reproduce it with the following minimal example, looks like even without arguments the call errors:

index = nmslib.init(method="hnsw", space="cosinesimil_sparse", data_type=nmslib.DataType.SPARSE_VECTOR, dtype=nmslib.DistType.FLOAT)
print("Setting index queryParams...")
index.setQueryTimeParams()
print("Adding datapoints to index...")

->

Setting index queryParams...
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Env info

• python -V -> Python 3.7.11
• pip freeze | grep nmslib -> nmslib==2.1.1

Hello,

We use nmslib as default engine for TensorFlow Similarity due to it's broad compatibility with various OSes. We got multiple reports, and I was able to confirm it, that nmslib don't install on Windows 11, potentially related to the issue #498.

Do you have any idea if/when you will be able to take a look at this? With the increased adoption of Win11 it become problematic for us.

Thanks :)