fix: Enforce revision ID and model names for future contributions #56
Conversation
Added this change mainly due to duplicates models results and due to recent addition not including a model revision (which I we should probably discourage). This fixes models results for: - voyage
| @@ -4,6 +4,10 @@ type: evaluation | |||
| submission_name: MTEB | |||
| --- | |||
|
|
|||
| > [!NOTE] | |||
| > Previously it was possible to submit models results to MTEB by adding the results to the model metadata. This is no longer an option as we want to ensure high quality metadata. | |||
There was a problem hiding this comment.
Question: is that currently enforced? Or will only be in effect once the new leaderboard goes public?
There was a problem hiding this comment.
The new leaderboard does not fetch it automatically (only fetches from here)
However, we had a PR that fetches everything from HF (once, but probably not something we will run again).
E.g. the models:
'vectoriseai/bge-base-en-v1.5',
'vectoriseai/bge-large-en-v1.5',
'vectoriseai/bge-small-en',
'vectoriseai/bge-small-en-v1.5',
'vectoriseai/e5-base',
'vectoriseai/e5-base-v2',
'vectoriseai/e5-large',
'vectoriseai/e5-large-v2',
'vectoriseai/e5-small-v2',
'vectoriseai/gte-base',
'vectoriseai/gte-large',
'vectoriseai/gte-small',
'vectoriseai/multilingual-e5-large',
are all copies and then creates duplicates on the leaderboard.
There was a problem hiding this comment.
pr. #57 removes these
| @pytest.mark.parametrize("model_rev_pair", model_rev_pairs) | ||
| def test_organization_is_specified_for_new_additions(model_rev_pair): | ||
| """ | ||
| Models added after 26th of November should include a organization ID within their name, e.g. "myorg/my_embedding_model". |
There was a problem hiding this comment.
| Models added after 26th of November should include a organization ID within their name, e.g. "myorg/my_embedding_model". | |
| Models added after November 26, 2024 should include a organization ID within their name, e.g. "myorg/my_embedding_model". |
| @pytest.mark.parametrize("model_rev_pair", model_rev_pairs) | ||
| def test_model_meta_in_folders(model_rev_pair): | ||
| """ | ||
| Models added after the 26th of November should contain a model_meta.json file |
There was a problem hiding this comment.
| Models added after the 26th of November should contain a model_meta.json file | |
| Models added after November 26, 2024 should contain a model_meta.json file |
- Added minor fixes from #56 - removed all vectoriseai/* models (e.g. vectoriseai/gte-large) They seems to be duplicates of the original model.
- Added minor fixes from #56 - removed all vectoriseai/* models (e.g. vectoriseai/gte-large) They seems to be duplicates of the original model.
Added tests to:
Added this change mainly due to duplicates models results and due to recent addition not including a model revision (which I we should probably discourage).
This fixes model results for:
(needed for the leaderboard)