Stell embeddings doesn't match with the official example (w/ sentence transformers)

[girishponkiya]

System Info

I started a service with the following command:

sudo docker run -d --gpus all \
    -p 8087:80 \
    --memory 8g \
    --cpus 1 \
    -v embeddings-test:/data  \
    -e TRUST_REMOTE_CODE="true"\
    --name embeddings-stella-1.5b \
    --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.8.3 \
    --model-id NovaSearch/stella_en_1.5B_v5 \
    --hf-token [MY_TOKEN]

..and computed similarity for examples queries and documents - as listed in the model's README. However, the similarity values are different than what I got thought the SentenceTransformer library.

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

1. Start an embedding service with:

sudo docker run -d --gpus all \
    -p 8087:80 \
    --memory 8g \
    --cpus 1 \
    -v embeddings-test:/data  \
    -e TRUST_REMOTE_CODE="true"\
    --name embeddings-stella-1.5b \
    --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.8.3 \
    --model-id NovaSearch/stella_en_1.5B_v5 \
    --hf-token [MY_TOKEN]

2. Compute similarity for the following queries and documents:

query_prompt = "Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: "
queries = [
    "What are some ways to reduce stress?",
    "What are the benefits of drinking green tea?",
]

# docs do not need any prompts
docs = [
    "There are many effective ways to reduce stress. Some common techniques include deep breathing, meditation, and physical activity. Engaging in hobbies, spending time in nature, and connecting with loved ones can also help alleviate stress. Additionally, setting boundaries, practicing self-care, and learning to say no can prevent stress from building up.",
    "Green tea has been consumed for centuries and is known for its potential health benefits. It contains antioxidants that may help protect the body against damage caused by free radicals. Regular consumption of green tea has been associated with improved heart health, enhanced cognitive function, and a reduced risk of certain types of cancer. The polyphenols in green tea may also have anti-inflammatory and weight loss properties.",
]

Script to compute the similarity:
sanity-check.py

Expected behavior

Expected similarity:

# [[0.8178789  0.2958377 ]
#  [0.31938642 0.7853526 ]]

[ucas010]

sir ,same with u
see my issue
#774
thx

[alvarobartt]

Hey @girishponkiya, I'll be reviewing this today, thanks for the reports and apologies for the inconvenience if any!

This is most likely related to the current Qwen2 implementation as that's leveraging https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct i.e. not the standard Qwen2 architecture so there might be differences...

As per your report @ucas010, I'll reply under that issue as it's not quite related given that you're talking about Qwen3-Embedding which is a different architecture 🤗