Hydra Sweeper plugin for Orion

Provides a mechanism for Hydra applications to use Orion algorithms for the optimization of the parameters of any experiment.

See website for more information

Install

pip install hydra-orion-sweeper

Search Space

Orion defines 5 different dimensions that can be used to define your search space.

• uniform(low, high, [discrete=False, precision=4, shape=None, default_value=None])
• loguniform(low, high, [discrete=False, precision=4, shape=None, default_value=None])
• normal(loc, scale, [discrete=False, precision=4, shape=None, default_value=None])
• choices(*options)
• fidelity(low, high, base=2)

Fidelity is a special dimension that is used to represent the training time, you can think of it as the epoch dimension.

Documentation

For in-depth documentation about the plugin and its configuration options you should refer to Orion as the plugin configurations are simply passed through.

• algorithm
• worker
• storage
• parametrization

Example

Configuration

defaults:
- override hydra/sweeper: orion

hydra:
    sweeper:
       params:
          a: "uniform(0, 1)"
          b: "uniform(0, 1)"

       experiment:
          name: 'experiment'
          version: '1'

       algorithm:
          type: random
          config:
             seed: 1

       worker:
          n_workers: -1
          max_broken: 3
          max_trials: 100

       storage:
          type: legacy
          database:
             type: pickleddb
             host: 'database.pkl'

# Default values
a: 0
b: 0

Note

If the orion database path is relative; orion will create one database per multirun, this is because hydra is in charge of the working directory, and hydra creates a new directory per run.

To share a database between multiruns you can use an absolute path.

This also means, that if the database path is relative the HPO runs are never resumed and always start from scratch.

You can use an absolute path to resume a previous run.

Code

import hydra
from omegaconf import DictConfig

@hydra.main(config_path=".", config_name="config")
def main(cfg: DictConfig) -> float:
   """Simple main function"""
   a = cfg.a
   b = cfg.b

   return float(a + b)

if __name__ == "__main__":
   main()

Running

To run the hyper parameter optimization process you need to specify the --multirun argument.

python my_app.py --multirun

The search space can also be tweaked from the command line

python my_app.py --multirun batch_size=4,8,12,16 optimizer.name=Adam,SGD 'optimizer.lr="loguniform(0.001, 1.0)"'

Note

When specifying overrides you need to be careful with your bash/zsh/fish environment and escape the arguments correctly.