Skip to content

Commit

Permalink
align readme with current mteb
Browse files Browse the repository at this point in the history
  • Loading branch information
Samoed committed Nov 24, 2024
1 parent 3ff38ec commit 555c47f
Showing 1 changed file with 8 additions and 2 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ model_name = "average_word_embeddings_komninos"
# model_name = "sentence-transformers/all-MiniLM-L6-v2"

model = SentenceTransformer(model_name)
# or directly from mteb:
model = mteb.get_model(model_name)
tasks = mteb.get_tasks(tasks=["Banking77Classification"])
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model, output_folder=f"results/{model_name}")
Expand Down Expand Up @@ -220,9 +222,13 @@ Note that the public leaderboard uses the test splits for all datasets except MS
Models should implement the following interface, implementing an `encode` function taking as inputs a list of sentences, and returning a list of embeddings (embeddings can be `np.array`, `torch.tensor`, etc.). For inspiration, you can look at the [mteb/mtebscripts repo](https://github.com/embeddings-benchmark/mtebscripts) used for running diverse models via SLURM scripts for the paper.

```python
import mteb
from mteb.encoder_interface import PromptType
from mteb.models.wrapper import Wrapper
import numpy as np


class CustomModel:
class CustomModel(Wrapper):
def encode(
self,
sentences: list[str],
Expand All @@ -244,7 +250,7 @@ class CustomModel:
pass

model = CustomModel()
tasks = mteb.get_task("Banking77Classification")
tasks = mteb.get_tasks(tasks=["Banking77Classification"])
evaluation = MTEB(tasks=tasks)
evaluation.run(model)
```
Expand Down

0 comments on commit 555c47f

Please sign in to comment.