Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable codecarbon by default #1428

Merged

Conversation

Samoed
Copy link
Collaborator

@Samoed Samoed commented Nov 10, 2024

This PR enables code carbon by default. Closes #1427

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks! Let's get the tests passing as well.

mteb/cli.py Outdated Show resolved Hide resolved
@Samoed
Copy link
Collaborator Author

Samoed commented Nov 10, 2024

It seems that the emission tracker fails due to multiple workers, as it works fine when I run it without -n auto.

@KennethEnevoldsen
Copy link
Contributor

Since this is technically breaking I will move it to v2.0.0

@isaac-chung
Copy link
Collaborator

Based one this issue, it seems like we can* allow multiple runs, but the tracking will be somewhat skewed. It won't matter in the tests, but will for sure affect real runs if multiple runs are done on the same machine. I think we would also add a line in README and/or a warning to let users know.

@KennethEnevoldsen
Copy link
Contributor

Sounds like this can be merged into v2.0.0?

@isaac-chung
Copy link
Collaborator

Yes, just missing a line in README and/or a warning in the comment above.

@Samoed
Copy link
Collaborator Author

Samoed commented Nov 15, 2024

@isaac-chung I added logging message

@isaac-chung isaac-chung changed the title Feat: enable codecarbon by default feat: enable codecarbon by default Nov 15, 2024
Copy link
Collaborator

@isaac-chung isaac-chung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
Also added some clarity around the variable as we flip enable/disable, true/false.

@isaac-chung isaac-chung merged commit 70a3ff2 into embeddings-benchmark:v2.0.0 Nov 15, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable Code Carbon by default in evaluation runs
3 participants