X-Git-Url: https://wannabe.guru.org/gitweb/?a=blobdiff_plain;f=src%2Fpyutils%2Fparallelize%2Fparallelize.py;h=6d31174424c2b415866a0b539a6cb4d64579a7fc;hb=eb9336c0c36876cd3da97a8f8ff69fd8d75495c5;hp=9824e8ac6d242405e2b1f0ea801266569d69ebae;hpb=69566c003b4f1c3a4905f37d3735d7921502d14a;p=pyutils.git diff --git a/src/pyutils/parallelize/parallelize.py b/src/pyutils/parallelize/parallelize.py index 9824e8a..6d31174 100644 --- a/src/pyutils/parallelize/parallelize.py +++ b/src/pyutils/parallelize/parallelize.py @@ -2,7 +2,19 @@ # © Copyright 2021-2022, Scott Gasch -"""A decorator to help with dead simple parallelization.""" +"""A decorator to help with dead simple parallelization. See usage +below. + +This will just work with `Method.THREAD` and `Method.PROCESS` but to +use `Method.REMOTE` you need to do some setup work. You need to +configure a pool of workers. Each worker should run the same version +of Python, ideally in identically configured virtual environments. +And you need to be able to ssh into each machine using key-based +authentication (i.e. non-iteractively) and run python. List machines +in the location set by `--remote_worker_records_file` (see +:file:executors.h for flag and an example JSON file under examples). + +""" import atexit