More documentation.

diff --git a/docs/pyutils.rst b/docs/pyutils.rst
index f548756452a5cf18f4af50a4eb8464de92431556..8fc96f7932087b796fad4fb06d818670abe089d9 100644 (file)
--- a/docs/pyutils.rst
+++ b/docs/pyutils.rst
@@ -2137,6 +2137,8 @@ Here are the predefined color names it knows:
    You can also use raw RGB values with this module so you do not have to use
    the predefined color names.
 
+---
+
 .. automodule:: pyutils.ansi
    :members:
    :undoc-members:
@@ -2155,6 +2157,8 @@ fill those gaps.  It include stuff like:
       addresses, filenames, percentages, dates, datetimes, and
       durations passed as flags.
 
+---
+
 .. automodule:: pyutils.argparse_utils
    :members:
    :undoc-members:
@@ -2174,6 +2178,8 @@ This decorator will do several things for you:
     - It initializes logging for your program (see :file:`logging.py`).
     - It can optionally run a code and/or memory profiler on your code.
 
+---
+
 .. automodule:: pyutils.bootstrap
    :members:
    :undoc-members:
@@ -2199,6 +2205,8 @@ the local disk or on Zookeeper.
 
 All of my examples use this as does the pyutils library itself.
 
+---
+
 .. automodule:: pyutils.config
    :members:
    :undoc-members:
@@ -2209,6 +2217,8 @@ pyutils.decorator\_utils module
 
 This is a grab bag of decorators.
 
+---
+
 .. automodule:: pyutils.decorator_utils
    :members:
    :undoc-members:
@@ -2219,6 +2229,8 @@ pyutils.dict\_utils module
 
 A bunch of helpers for dealing with Python dicts.
 
+---
+
 .. automodule:: pyutils.dict_utils
    :members:
    :undoc-members:
@@ -2229,6 +2241,8 @@ pyutils.exec\_utils module
 
 Helper code for dealing with subprocesses.
 
+---
+
 .. automodule:: pyutils.exec_utils
    :members:
    :undoc-members:
@@ -2239,6 +2253,8 @@ pyutils.function\_utils module
 
 Helper util for dealing with functions.
 
+---
+
 .. automodule:: pyutils.function_utils
    :members:
    :undoc-members:
@@ -2249,6 +2265,8 @@ pyutils.id\_generator module
 
 Generate unique identifiers.
 
+---
+
 .. automodule:: pyutils.id_generator
    :members:
    :undoc-members:
@@ -2260,6 +2278,8 @@ pyutils.iter\_utils module
 Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`,
 and :py:class:`SamplingIterator`.
 
+---
+
 .. automodule:: pyutils.iter_utils
    :members:
    :undoc-members:
@@ -2270,6 +2290,8 @@ pyutils.list\_utils module
 
 Utilities for dealing with Python lists.
 
+---
+
 .. automodule:: pyutils.list_utils
    :members:
    :undoc-members:
@@ -2292,6 +2314,8 @@ logging should be initialized and controlled.  It uses standard Python
     - Log probalistically,
     - Clear rogue logging handlers added by other imports.
 
+---
+
 .. automodule:: pyutils.logging_utils
    :members:
    :undoc-members:
@@ -2305,6 +2329,8 @@ makes population summary statistics available to your code quickly, GCD
 computation, literate float truncation, percentage <-> multiplier, prime
 number determination, etc...
 
+---
+
 .. automodule:: pyutils.math_utils
    :members:
    :undoc-members:
@@ -2315,6 +2341,8 @@ pyutils.misc\_utils module
 
 Miscellaneous utilities: are we running as root, and is a debugger attached?
 
+---
+
 .. automodule:: pyutils.misc_utils
    :members:
    :undoc-members:
@@ -2328,6 +2356,8 @@ singleton classes that (optionally / conditionally) load their state
 from some external location and (optionally / conditionally) save their
 state to an external location at shutdown.
 
+---
+
 .. automodule:: pyutils.persistent
    :members:
    :undoc-members:
@@ -2340,6 +2370,8 @@ This module defines a helper that is invoked by the remote executor to
 run pickled code on a remote machine.  It is used by code marked with
 `@parallelize(method=Method.REMOTE)` in the parallelize framework.
 
+---
+
 .. automodule:: pyutils.remote_worker
    :members:
    :undoc-members:
@@ -2356,6 +2388,8 @@ state that dynamically changes.  These classes update their state
 (either automatically or when invoked to poll) and allow their callers
 to wait on state changes.
 
+---
+
 .. automodule:: pyutils.state_tracker
    :members:
    :undoc-members:
@@ -2367,6 +2401,8 @@ pyutils.stopwatch module
 This is a stopwatch context that just times how long something took to
 execute.
 
+---
+
 .. automodule:: pyutils.stopwatch
    :members:
    :undoc-members:
@@ -2379,6 +2415,8 @@ A bunch of utilities for dealing with strings.  Based on a really great
 starting library from Davide Zanotti, I've added a pile of other string
 functions so hopefully it will handle all of your string-needs.
 
+---
+
 .. automodule:: pyutils.string_utils
    :members:
    :undoc-members:
@@ -2397,6 +2435,8 @@ Utilities for dealing with and creating text chunks.  For example:
     - create a header line,
     - draw a box around some text.
 
+---
+
 .. automodule:: pyutils.text_utils
    :members:
    :undoc-members:
@@ -2407,6 +2447,8 @@ pyutils.unittest\_utils module
 
 Utilities to support smarter unit tests.
 
+---
+
 .. automodule:: pyutils.unittest_utils
    :members:
    :undoc-members:
@@ -2417,6 +2459,8 @@ pyutils.unscrambler module
 
 Unscramble scrambled English words quickly.
 
+---
+
 .. automodule:: pyutils.unscrambler
    :members:
    :undoc-members:
@@ -2427,6 +2471,8 @@ pyutils.zookeeper module
 
 A helper module for dealing with Zookeeper that adds some functionality.
 
+---
+
 .. automodule:: pyutils.zookeeper
    :members:
    :undoc-members:
@@ -2435,6 +2481,8 @@ A helper module for dealing with Zookeeper that adds some functionality.
 Module contents
 ---------------
 
+---
+
 .. automodule:: pyutils
    :members:
    :undoc-members:

diff --git a/src/pyutils/parallelize/deferred_operand.py b/src/pyutils/parallelize/deferred_operand.py
index 9edbb9e8072b451ffc13f447235e3da5505bcff6..884967091607ee5a57376c83548fd324b349def6 100644 (file)
--- a/src/pyutils/parallelize/deferred_operand.py
+++ b/src/pyutils/parallelize/deferred_operand.py
@@ -2,9 +2,11 @@
 
 # © Copyright 2021-2022, Scott Gasch
 
-"""This is a helper class that tries to define every __dunder__ method
-so as to defer that evaluation of an object as long as possible.  It
-is used by smart_future.py as a base class.
+"""This is the base class of :class:`SmartFuture`.  It is essentially
+a class that tries to have every Python __dunder__ method defined
+reasonably for it such that, when it is used in a manner that requires
+its value to be known, it calls a `resolve` method to wait for the
+data it represents.
 
 """
 

diff --git a/src/pyutils/parallelize/parallelize.py b/src/pyutils/parallelize/parallelize.py
index 9824e8ac6d242405e2b1f0ea801266569d69ebae..6d31174424c2b415866a0b539a6cb4d64579a7fc 100644 (file)
--- a/src/pyutils/parallelize/parallelize.py
+++ b/src/pyutils/parallelize/parallelize.py
@@ -2,7 +2,19 @@
 
 # © Copyright 2021-2022, Scott Gasch
 
-"""A decorator to help with dead simple parallelization."""
+"""A decorator to help with dead simple parallelization.  See usage
+below.
+
+This will just work with `Method.THREAD` and `Method.PROCESS` but to
+use `Method.REMOTE` you need to do some setup work.  You need to
+configure a pool of workers.  Each worker should run the same version
+of Python, ideally in identically configured virtual environments.
+And you need to be able to ssh into each machine using key-based
+authentication (i.e. non-iteractively) and run python.  List machines
+in the location set by `--remote_worker_records_file` (see
+:file:executors.h for flag and an example JSON file under examples).
+
+"""
 
 
 import atexit

diff --git a/src/pyutils/remote_worker.py b/src/pyutils/remote_worker.py
index 54f562da181d8d21b9891c8b99cfcc8e2dcd43fe..630d7e035fd414b166a1dda358ffd6aa430a70cd 100755 (executable)
--- a/src/pyutils/remote_worker.py
+++ b/src/pyutils/remote_worker.py
@@ -4,6 +4,11 @@
 
 """A simple utility to unpickle some code, run it, and pickle the
 results.  Please don't unpickle (or run!) code you do not know.
+
+This script is used by code in parallelize, namely the
+:class:`RemoteExecutor`, to schedule work on a remote machine.
+The code in :file:`parallelize.py` uses a user-defined configuration
+to schedule work this way.  See that file for setup instructions.
 """
 
 import logging