More documentation.

diff --git a/docs/pyutils.rst b/docs/pyutils.rst
index f548756452a5cf18f4af50a4eb8464de92431556..8fc96f7932087b796fad4fb06d818670abe089d9 100644 (file)
--- a/docs/pyutils.rst
+++ b/docs/pyutils.rst
@@ -2137,6 +2137,8 @@ Here are the predefined color names it knows:
    You can also use raw RGB values with this module so you do not have to use
    the predefined color names.
 
    You can also use raw RGB values with this module so you do not have to use
    the predefined color names.
 

+---
+

 .. automodule:: pyutils.ansi
    :members:
    :undoc-members:
 .. automodule:: pyutils.ansi
    :members:
    :undoc-members:


@@ -2155,6 +2157,8 @@ fill those gaps.  It include stuff like:
       addresses, filenames, percentages, dates, datetimes, and
       durations passed as flags.
 
       addresses, filenames, percentages, dates, datetimes, and
       durations passed as flags.
 

+---
+

 .. automodule:: pyutils.argparse_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.argparse_utils
    :members:
    :undoc-members:


@@ -2174,6 +2178,8 @@ This decorator will do several things for you:
     - It initializes logging for your program (see :file:`logging.py`).
     - It can optionally run a code and/or memory profiler on your code.
 
     - It initializes logging for your program (see :file:`logging.py`).
     - It can optionally run a code and/or memory profiler on your code.
 

+---
+

 .. automodule:: pyutils.bootstrap
    :members:
    :undoc-members:
 .. automodule:: pyutils.bootstrap
    :members:
    :undoc-members:


@@ -2199,6 +2205,8 @@ the local disk or on Zookeeper.
 
 All of my examples use this as does the pyutils library itself.
 
 
 All of my examples use this as does the pyutils library itself.
 

+---
+

 .. automodule:: pyutils.config
    :members:
    :undoc-members:
 .. automodule:: pyutils.config
    :members:
    :undoc-members:


@@ -2209,6 +2217,8 @@ pyutils.decorator\_utils module
 
 This is a grab bag of decorators.
 
 
 This is a grab bag of decorators.
 

+---
+

 .. automodule:: pyutils.decorator_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.decorator_utils
    :members:
    :undoc-members:


@@ -2219,6 +2229,8 @@ pyutils.dict\_utils module
 
 A bunch of helpers for dealing with Python dicts.
 
 
 A bunch of helpers for dealing with Python dicts.
 

+---
+

 .. automodule:: pyutils.dict_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.dict_utils
    :members:
    :undoc-members:


@@ -2229,6 +2241,8 @@ pyutils.exec\_utils module
 
 Helper code for dealing with subprocesses.
 
 
 Helper code for dealing with subprocesses.
 

+---
+

 .. automodule:: pyutils.exec_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.exec_utils
    :members:
    :undoc-members:


@@ -2239,6 +2253,8 @@ pyutils.function\_utils module
 
 Helper util for dealing with functions.
 
 
 Helper util for dealing with functions.
 

+---
+

 .. automodule:: pyutils.function_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.function_utils
    :members:
    :undoc-members:


@@ -2249,6 +2265,8 @@ pyutils.id\_generator module
 
 Generate unique identifiers.
 
 
 Generate unique identifiers.
 

+---
+

 .. automodule:: pyutils.id_generator
    :members:
    :undoc-members:
 .. automodule:: pyutils.id_generator
    :members:
    :undoc-members:


@@ -2260,6 +2278,8 @@ pyutils.iter\_utils module
 Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`,
 and :py:class:`SamplingIterator`.
 
 Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`,
 and :py:class:`SamplingIterator`.
 

+---
+

 .. automodule:: pyutils.iter_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.iter_utils
    :members:
    :undoc-members:


@@ -2270,6 +2290,8 @@ pyutils.list\_utils module
 
 Utilities for dealing with Python lists.
 
 
 Utilities for dealing with Python lists.
 

+---
+

 .. automodule:: pyutils.list_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.list_utils
    :members:
    :undoc-members:


@@ -2292,6 +2314,8 @@ logging should be initialized and controlled.  It uses standard Python
     - Log probalistically,
     - Clear rogue logging handlers added by other imports.
 
     - Log probalistically,
     - Clear rogue logging handlers added by other imports.
 

+---
+

 .. automodule:: pyutils.logging_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.logging_utils
    :members:
    :undoc-members:


@@ -2305,6 +2329,8 @@ makes population summary statistics available to your code quickly, GCD
 computation, literate float truncation, percentage <-> multiplier, prime
 number determination, etc...
 
 computation, literate float truncation, percentage <-> multiplier, prime
 number determination, etc...
 

+---
+

 .. automodule:: pyutils.math_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.math_utils
    :members:
    :undoc-members:


@@ -2315,6 +2341,8 @@ pyutils.misc\_utils module
 
 Miscellaneous utilities: are we running as root, and is a debugger attached?
 
 
 Miscellaneous utilities: are we running as root, and is a debugger attached?
 

+---
+

 .. automodule:: pyutils.misc_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.misc_utils
    :members:
    :undoc-members:


@@ -2328,6 +2356,8 @@ singleton classes that (optionally / conditionally) load their state
 from some external location and (optionally / conditionally) save their
 state to an external location at shutdown.
 
 from some external location and (optionally / conditionally) save their
 state to an external location at shutdown.
 

+---
+

 .. automodule:: pyutils.persistent
    :members:
    :undoc-members:
 .. automodule:: pyutils.persistent
    :members:
    :undoc-members:


@@ -2340,6 +2370,8 @@ This module defines a helper that is invoked by the remote executor to
 run pickled code on a remote machine.  It is used by code marked with
 `@parallelize(method=Method.REMOTE)` in the parallelize framework.
 
 run pickled code on a remote machine.  It is used by code marked with
 `@parallelize(method=Method.REMOTE)` in the parallelize framework.
 

+---
+

 .. automodule:: pyutils.remote_worker
    :members:
    :undoc-members:
 .. automodule:: pyutils.remote_worker
    :members:
    :undoc-members:


@@ -2356,6 +2388,8 @@ state that dynamically changes.  These classes update their state
 (either automatically or when invoked to poll) and allow their callers
 to wait on state changes.
 
 (either automatically or when invoked to poll) and allow their callers
 to wait on state changes.
 

+---
+

 .. automodule:: pyutils.state_tracker
    :members:
    :undoc-members:
 .. automodule:: pyutils.state_tracker
    :members:
    :undoc-members:


@@ -2367,6 +2401,8 @@ pyutils.stopwatch module
 This is a stopwatch context that just times how long something took to
 execute.
 
 This is a stopwatch context that just times how long something took to
 execute.
 

+---
+

 .. automodule:: pyutils.stopwatch
    :members:
    :undoc-members:
 .. automodule:: pyutils.stopwatch
    :members:
    :undoc-members:


@@ -2379,6 +2415,8 @@ A bunch of utilities for dealing with strings.  Based on a really great
 starting library from Davide Zanotti, I've added a pile of other string
 functions so hopefully it will handle all of your string-needs.
 
 starting library from Davide Zanotti, I've added a pile of other string
 functions so hopefully it will handle all of your string-needs.
 

+---
+

 .. automodule:: pyutils.string_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.string_utils
    :members:
    :undoc-members:


@@ -2397,6 +2435,8 @@ Utilities for dealing with and creating text chunks.  For example:
     - create a header line,
     - draw a box around some text.
 
     - create a header line,
     - draw a box around some text.
 

+---
+

 .. automodule:: pyutils.text_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.text_utils
    :members:
    :undoc-members:


@@ -2407,6 +2447,8 @@ pyutils.unittest\_utils module
 
 Utilities to support smarter unit tests.
 
 
 Utilities to support smarter unit tests.
 

+---
+

 .. automodule:: pyutils.unittest_utils
    :members:
    :undoc-members:
 .. automodule:: pyutils.unittest_utils
    :members:
    :undoc-members:


@@ -2417,6 +2459,8 @@ pyutils.unscrambler module
 
 Unscramble scrambled English words quickly.
 
 
 Unscramble scrambled English words quickly.
 

+---
+

 .. automodule:: pyutils.unscrambler
    :members:
    :undoc-members:
 .. automodule:: pyutils.unscrambler
    :members:
    :undoc-members:


@@ -2427,6 +2471,8 @@ pyutils.zookeeper module
 
 A helper module for dealing with Zookeeper that adds some functionality.
 
 
 A helper module for dealing with Zookeeper that adds some functionality.
 

+---
+

 .. automodule:: pyutils.zookeeper
    :members:
    :undoc-members:
 .. automodule:: pyutils.zookeeper
    :members:
    :undoc-members:


@@ -2435,6 +2481,8 @@ A helper module for dealing with Zookeeper that adds some functionality.
 Module contents
 ---------------
 
 Module contents
 ---------------
 

+---
+

 .. automodule:: pyutils
    :members:
    :undoc-members:
 .. automodule:: pyutils
    :members:
    :undoc-members:

diff --git a/src/pyutils/parallelize/deferred_operand.py b/src/pyutils/parallelize/deferred_operand.py
index 9edbb9e8072b451ffc13f447235e3da5505bcff6..884967091607ee5a57376c83548fd324b349def6 100644 (file)
--- a/src/pyutils/parallelize/deferred_operand.py
+++ b/src/pyutils/parallelize/deferred_operand.py
@@ -2,9 +2,11 @@
 
 # © Copyright 2021-2022, Scott Gasch
 
 
 # © Copyright 2021-2022, Scott Gasch
 

-"""This is a helper class that tries to define every __dunder__ method
-so as to defer that evaluation of an object as long as possible.  It
-is used by smart_future.py as a base class.
+"""This is the base class of :class:`SmartFuture`.  It is essentially
+a class that tries to have every Python __dunder__ method defined
+reasonably for it such that, when it is used in a manner that requires
+its value to be known, it calls a `resolve` method to wait for the
+data it represents.

 
 """
 
 
 """
 

diff --git a/src/pyutils/parallelize/parallelize.py b/src/pyutils/parallelize/parallelize.py
index 9824e8ac6d242405e2b1f0ea801266569d69ebae..6d31174424c2b415866a0b539a6cb4d64579a7fc 100644 (file)
--- a/src/pyutils/parallelize/parallelize.py
+++ b/src/pyutils/parallelize/parallelize.py
@@ -2,7 +2,19 @@
 
 # © Copyright 2021-2022, Scott Gasch
 
 
 # © Copyright 2021-2022, Scott Gasch
 

-"""A decorator to help with dead simple parallelization."""
+"""A decorator to help with dead simple parallelization.  See usage
+below.
+
+This will just work with `Method.THREAD` and `Method.PROCESS` but to
+use `Method.REMOTE` you need to do some setup work.  You need to
+configure a pool of workers.  Each worker should run the same version
+of Python, ideally in identically configured virtual environments.
+And you need to be able to ssh into each machine using key-based
+authentication (i.e. non-iteractively) and run python.  List machines
+in the location set by `--remote_worker_records_file` (see
+:file:executors.h for flag and an example JSON file under examples).
+
+"""

 
 
 import atexit
 
 
 import atexit

diff --git a/src/pyutils/remote_worker.py b/src/pyutils/remote_worker.py
index 54f562da181d8d21b9891c8b99cfcc8e2dcd43fe..630d7e035fd414b166a1dda358ffd6aa430a70cd 100755 (executable)
--- a/src/pyutils/remote_worker.py
+++ b/src/pyutils/remote_worker.py
@@ -4,6 +4,11 @@
 
 """A simple utility to unpickle some code, run it, and pickle the
 results.  Please don't unpickle (or run!) code you do not know.
 
 """A simple utility to unpickle some code, run it, and pickle the
 results.  Please don't unpickle (or run!) code you do not know.

+
+This script is used by code in parallelize, namely the
+:class:`RemoteExecutor`, to schedule work on a remote machine.
+The code in :file:`parallelize.py` uses a user-defined configuration
+to schedule work this way.  See that file for setup instructions.

 """
 
 import logging
 """
 
 import logging