Use multi-thread for cloud experiments and mutli-process for local ones

[DonggeLiu]

With clould experiments enabled, local program is left with no computationally heavy tasks. Most of the time, it only waits for cloud build results and writes them to a file.
using multi-thread can achieve higher parallelism (e.g., avoid overhead in creation, management, context switching) and require less resource (e.g., reduce memory usage).

[oliverchang]

Is there a way we can consolidate the two code paths more?

e.g. something like:

if ...:
  pool =  ThreadPool
else:
  pool = Pool

Not sure what the difference is exactly between ThreadPoolExecutor and ThreadPool

[oliverchang]

See https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.ThreadPool

[DonggeLiu]

See https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.ThreadPool

I reckon the doc implies ThreadPoolExecutor is more modern and supports thread-level parallelism better?
I can definitely try to consolidate the two code parts to make them look better.

[DonggeLiu]

Made an attempt to clean up the related code in run_all_experiments.py a bit, will do the same in run_one_experiment.py.
The code should be more readable and have less repetition, but I did not find a perfect way to consolidate them further without over-engineering this.

Please let me know if there are other options.

[oliverchang]

Can we use https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor instead to make the code paths more consistent for the process case?

[DonggeLiu]

Can we use https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor instead to make the code paths more consistent for the process case?

Yep, sure.
Thanks!

[DonggeLiu]

Not high priority, converting it to a draft for now and will come back to this later.