From: Scott Gasch Date: Fri, 14 Oct 2022 16:26:58 +0000 (-0700) Subject: More documentation. X-Git-Url: https://wannabe.guru.org/gitweb/?a=commitdiff_plain;ds=inline;h=eb9336c0c36876cd3da97a8f8ff69fd8d75495c5;p=pyutils.git More documentation. --- diff --git a/docs/pyutils.rst b/docs/pyutils.rst index f548756..8fc96f7 100644 --- a/docs/pyutils.rst +++ b/docs/pyutils.rst @@ -2137,6 +2137,8 @@ Here are the predefined color names it knows: You can also use raw RGB values with this module so you do not have to use the predefined color names. +--- + .. automodule:: pyutils.ansi :members: :undoc-members: @@ -2155,6 +2157,8 @@ fill those gaps. It include stuff like: addresses, filenames, percentages, dates, datetimes, and durations passed as flags. +--- + .. automodule:: pyutils.argparse_utils :members: :undoc-members: @@ -2174,6 +2178,8 @@ This decorator will do several things for you: - It initializes logging for your program (see :file:`logging.py`). - It can optionally run a code and/or memory profiler on your code. +--- + .. automodule:: pyutils.bootstrap :members: :undoc-members: @@ -2199,6 +2205,8 @@ the local disk or on Zookeeper. All of my examples use this as does the pyutils library itself. +--- + .. automodule:: pyutils.config :members: :undoc-members: @@ -2209,6 +2217,8 @@ pyutils.decorator\_utils module This is a grab bag of decorators. +--- + .. automodule:: pyutils.decorator_utils :members: :undoc-members: @@ -2219,6 +2229,8 @@ pyutils.dict\_utils module A bunch of helpers for dealing with Python dicts. +--- + .. automodule:: pyutils.dict_utils :members: :undoc-members: @@ -2229,6 +2241,8 @@ pyutils.exec\_utils module Helper code for dealing with subprocesses. +--- + .. automodule:: pyutils.exec_utils :members: :undoc-members: @@ -2239,6 +2253,8 @@ pyutils.function\_utils module Helper util for dealing with functions. +--- + .. automodule:: pyutils.function_utils :members: :undoc-members: @@ -2249,6 +2265,8 @@ pyutils.id\_generator module Generate unique identifiers. +--- + .. automodule:: pyutils.id_generator :members: :undoc-members: @@ -2260,6 +2278,8 @@ pyutils.iter\_utils module Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`, and :py:class:`SamplingIterator`. +--- + .. automodule:: pyutils.iter_utils :members: :undoc-members: @@ -2270,6 +2290,8 @@ pyutils.list\_utils module Utilities for dealing with Python lists. +--- + .. automodule:: pyutils.list_utils :members: :undoc-members: @@ -2292,6 +2314,8 @@ logging should be initialized and controlled. It uses standard Python - Log probalistically, - Clear rogue logging handlers added by other imports. +--- + .. automodule:: pyutils.logging_utils :members: :undoc-members: @@ -2305,6 +2329,8 @@ makes population summary statistics available to your code quickly, GCD computation, literate float truncation, percentage <-> multiplier, prime number determination, etc... +--- + .. automodule:: pyutils.math_utils :members: :undoc-members: @@ -2315,6 +2341,8 @@ pyutils.misc\_utils module Miscellaneous utilities: are we running as root, and is a debugger attached? +--- + .. automodule:: pyutils.misc_utils :members: :undoc-members: @@ -2328,6 +2356,8 @@ singleton classes that (optionally / conditionally) load their state from some external location and (optionally / conditionally) save their state to an external location at shutdown. +--- + .. automodule:: pyutils.persistent :members: :undoc-members: @@ -2340,6 +2370,8 @@ This module defines a helper that is invoked by the remote executor to run pickled code on a remote machine. It is used by code marked with `@parallelize(method=Method.REMOTE)` in the parallelize framework. +--- + .. automodule:: pyutils.remote_worker :members: :undoc-members: @@ -2356,6 +2388,8 @@ state that dynamically changes. These classes update their state (either automatically or when invoked to poll) and allow their callers to wait on state changes. +--- + .. automodule:: pyutils.state_tracker :members: :undoc-members: @@ -2367,6 +2401,8 @@ pyutils.stopwatch module This is a stopwatch context that just times how long something took to execute. +--- + .. automodule:: pyutils.stopwatch :members: :undoc-members: @@ -2379,6 +2415,8 @@ A bunch of utilities for dealing with strings. Based on a really great starting library from Davide Zanotti, I've added a pile of other string functions so hopefully it will handle all of your string-needs. +--- + .. automodule:: pyutils.string_utils :members: :undoc-members: @@ -2397,6 +2435,8 @@ Utilities for dealing with and creating text chunks. For example: - create a header line, - draw a box around some text. +--- + .. automodule:: pyutils.text_utils :members: :undoc-members: @@ -2407,6 +2447,8 @@ pyutils.unittest\_utils module Utilities to support smarter unit tests. +--- + .. automodule:: pyutils.unittest_utils :members: :undoc-members: @@ -2417,6 +2459,8 @@ pyutils.unscrambler module Unscramble scrambled English words quickly. +--- + .. automodule:: pyutils.unscrambler :members: :undoc-members: @@ -2427,6 +2471,8 @@ pyutils.zookeeper module A helper module for dealing with Zookeeper that adds some functionality. +--- + .. automodule:: pyutils.zookeeper :members: :undoc-members: @@ -2435,6 +2481,8 @@ A helper module for dealing with Zookeeper that adds some functionality. Module contents --------------- +--- + .. automodule:: pyutils :members: :undoc-members: diff --git a/src/pyutils/parallelize/deferred_operand.py b/src/pyutils/parallelize/deferred_operand.py index 9edbb9e..8849670 100644 --- a/src/pyutils/parallelize/deferred_operand.py +++ b/src/pyutils/parallelize/deferred_operand.py @@ -2,9 +2,11 @@ # © Copyright 2021-2022, Scott Gasch -"""This is a helper class that tries to define every __dunder__ method -so as to defer that evaluation of an object as long as possible. It -is used by smart_future.py as a base class. +"""This is the base class of :class:`SmartFuture`. It is essentially +a class that tries to have every Python __dunder__ method defined +reasonably for it such that, when it is used in a manner that requires +its value to be known, it calls a `resolve` method to wait for the +data it represents. """ diff --git a/src/pyutils/parallelize/parallelize.py b/src/pyutils/parallelize/parallelize.py index 9824e8a..6d31174 100644 --- a/src/pyutils/parallelize/parallelize.py +++ b/src/pyutils/parallelize/parallelize.py @@ -2,7 +2,19 @@ # © Copyright 2021-2022, Scott Gasch -"""A decorator to help with dead simple parallelization.""" +"""A decorator to help with dead simple parallelization. See usage +below. + +This will just work with `Method.THREAD` and `Method.PROCESS` but to +use `Method.REMOTE` you need to do some setup work. You need to +configure a pool of workers. Each worker should run the same version +of Python, ideally in identically configured virtual environments. +And you need to be able to ssh into each machine using key-based +authentication (i.e. non-iteractively) and run python. List machines +in the location set by `--remote_worker_records_file` (see +:file:executors.h for flag and an example JSON file under examples). + +""" import atexit diff --git a/src/pyutils/remote_worker.py b/src/pyutils/remote_worker.py index 54f562d..630d7e0 100755 --- a/src/pyutils/remote_worker.py +++ b/src/pyutils/remote_worker.py @@ -4,6 +4,11 @@ """A simple utility to unpickle some code, run it, and pickle the results. Please don't unpickle (or run!) code you do not know. + +This script is used by code in parallelize, namely the +:class:`RemoteExecutor`, to schedule work on a remote machine. +The code in :file:`parallelize.py` uses a user-defined configuration +to schedule work this way. See that file for setup instructions. """ import logging