X-Git-Url: https://wannabe.guru.org/gitweb/?a=blobdiff_plain;f=docs%2Fpyutils.rst;h=57a76f0a973c486c64631b936f0ef207ee0cb143;hb=993b0992473c12294ed659e52b532e1c8cf9cd1e;hp=8fc96f7932087b796fad4fb06d818670abe089d9;hpb=eb9336c0c36876cd3da97a8f8ff69fd8d75495c5;p=pyutils.git diff --git a/docs/pyutils.rst b/docs/pyutils.rst index 8fc96f7..57a76f0 100644 --- a/docs/pyutils.rst +++ b/docs/pyutils.rst @@ -6,36 +6,50 @@ a generally useful pattern I stuffed it into a local library. That library grew into pyutils: a set of collections, helpers and utilities that I find useful and hope you will too. -Code is under `src/pyutils/`. Most code includes documentation and inline -doctests. +Code is under `src/pyutils/ +`_. +Most code includes inlint documentation and doctests. I've tried to +organize it into logical packages based on the code's functionality. +Note that when words would collide with a Python library or reserved +word I've used a 'z' at the end, e.g. 'collectionz' instead of +'collections', 'typez' instead of 'type', etc... -Unit and integration tests are under `tests/*`. To run all tests:: +There's some example code that uses various features of this project checked +in under `examples/ `_ that you can check out. + +Unit and integration tests are under `tests/ +`_. +To run all tests:: cd tests/ ./run_tests.py --all [--coverage] -See the README under `tests/` and the code of `run_tests.py` for more -options / information. +See the `README `_ under `tests/` and the code of `run_tests.py` for more +options / information about running tests. This package generates Sphinx docs which are available at: https://wannabe.guru.org/pydocs/pyutils/pyutils.html Package code is checked into a local git server and available to clone -from git at https://wannabe.guru.org/git/pyutils.git or from a web browser -at: +from git at https://wannabe.guru.org/git/pyutils.git or to view in a +web browser at: https://wannabe.guru.org/gitweb/?p=pyutils.git;a=summary For a long time this was just a local library on my machine that my -tools imported but I've now decided to release it on PyPi. Early +tools imported but I've now decided to release it on PyPi. Earlier development happened in a different git repo: https://wannabe.guru.org/gitweb/?p=python_utils.git;a=summary -I hope you find this useful. LICENSE and NOTICE describe reusing this -code and where everything came from. Drop me a line if you are using -this, find a bug, have a question, or have a suggestion: +The `LICENSE +`_ +and `NOTICE +<[https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=NOTICE;hb=HEAD>`_ +files at the root of the project describe reusing this code and where +everything came from. Drop me a line if you are using this, find a +bug, have a question, or have a suggestion: --Scott Gasch (scott.gasch@gmail.com) @@ -44,7 +58,8 @@ Subpackages ----------- .. toctree:: - :maxdepth: 4 + :maxdepth: 5 + :name: mastertoc pyutils.collectionz pyutils.compress @@ -62,14 +77,21 @@ pyutils.ansi module ------------------- This file mainly contains code for changing the nature of text printed -to the console via ANSI escape sequences. e.g. it can be used to emit -text that is bolded, underlined, italicised, colorized, etc... +to the console via ANSI escape sequences. For example, it can be used +to emit text that is bolded, underlined, italicised, colorized, etc... + +It does *not* contain ANSI escape sequences that do things like move +the cursor or record/restore its position. It is focused on text style +only. -It also contains a colorizing context that will apply color patterns -based on regular expressions to any data emitted to stdout that may be -useful in adding color to other programs' outputs, for instance. +The file includes a colorizing context that will apply color patterns +based on regular expressions / callables to any data emitted to stdout +that may be useful in adding color to other programs' outputs, for +instance. -Here are the predefined color names it knows: +In addition, it contains a mapping from color name to RGB value that it +uses to enable friendlier color names in most of its functions. Here +is the list of predefined color names it knows: .. raw:: html @@ -2135,7 +2157,7 @@ Here are the predefined color names it knows: .. note:: You can also use raw RGB values with this module so you do not have to use - the predefined color names. + these predefined color names unless you want to. --- @@ -2147,15 +2169,9 @@ Here are the predefined color names it knows: pyutils.argparse\_utils module ------------------------------ -I use the Python internal :py:mod:`argparse` module for commandline argument -parsing but found it lacking in some ways. This module contains code to -fill those gaps. It include stuff like: - - - An :class:`argparse.Action` to create pairs of flags such as - `--feature` and `--no_feature`. - - A helper to parse and validate bools, IP addresses, MAC - addresses, filenames, percentages, dates, datetimes, and - durations passed as flags. +I use and love the Python internal :py:mod:`argparse` module for +commandline argument parsing but found it lacking in some ways. This +module contains code to fill those gaps. See also :py:mod:`pyutils.config`. --- @@ -2167,16 +2183,10 @@ fill those gaps. It include stuff like: pyutils.bootstrap module ------------------------ -Bootstrap module defines a decorator meant to wrap your main function. -This decorator will do several things for you: - - - If your code uses the :file:`config.py` module (see below), it invokes - :py:meth:`parse` automatically to initialize `config.config` from commandline - flags, environment variables, or other sources. - - It will optionally break into pdb in response to an unhandled - Exception at the top level of your code. - - It initializes logging for your program (see :file:`logging.py`). - - It can optionally run a code and/or memory profiler on your code. +The bootstrap module defines a decorator meant to wrap your main function. +This is optional, of course: you can use this library without using the +bootstrap decorator on your main. If you choose to use it, though, it will +do some work for you automatically. --- @@ -2188,22 +2198,18 @@ This decorator will do several things for you: pyutils.config module --------------------- -This module reads the program's configuration parameters from: the -commandline (using :py:mod:`argparse`), environment variables and/or a -shared zookeeper-based configuration. It stores this configuration -state in a dict-like structure that can be queried by your code at -runtime. +The config module is an opinionated way to set up input parameters to +your program. It is enabled by using the :py:mod:`pyutils.bootstrap` +decorator around your main entry point or by simply calling +:py:meth:`pyutils.config.parse` early in main (which is what +:py:meth:`pyutils.bootstrap.initialize` does for you). -It handles creating a nice `--help` message for your code. - -It can optionally react to dynamic config changes and change state at -runtime (if and only if you name your flag with the *dynamic_* prefix -and are using zookeeper-based configs). - -It can optionally save and retrieve sets of arguments from files on -the local disk or on Zookeeper. - -All of my examples use this as does the pyutils library itself. +If you use this module, input parameters to your program come from +the commandline (and are configured using Python's :py:mod:`argparse`). +But they can also be be augmented or replaced using saved configuration +files stored either on the local filesystem or on Apache Zookeeper. +There is a provision for enabling dynamic arguments (i.e. that can change +during runtime) via Zookeeper (which is disabled by default). --- @@ -2215,10 +2221,6 @@ All of my examples use this as does the pyutils library itself. pyutils.decorator\_utils module ------------------------------- -This is a grab bag of decorators. - ---- - .. automodule:: pyutils.decorator_utils :members: :undoc-members: @@ -2227,10 +2229,6 @@ This is a grab bag of decorators. pyutils.dict\_utils module -------------------------- -A bunch of helpers for dealing with Python dicts. - ---- - .. automodule:: pyutils.dict_utils :members: :undoc-members: @@ -2239,10 +2237,6 @@ A bunch of helpers for dealing with Python dicts. pyutils.exec\_utils module -------------------------- -Helper code for dealing with subprocesses. - ---- - .. automodule:: pyutils.exec_utils :members: :undoc-members: @@ -2251,10 +2245,6 @@ Helper code for dealing with subprocesses. pyutils.function\_utils module ------------------------------ -Helper util for dealing with functions. - ---- - .. automodule:: pyutils.function_utils :members: :undoc-members: @@ -2263,10 +2253,6 @@ Helper util for dealing with functions. pyutils.id\_generator module ---------------------------- -Generate unique identifiers. - ---- - .. automodule:: pyutils.id_generator :members: :undoc-members: @@ -2275,11 +2261,6 @@ Generate unique identifiers. pyutils.iter\_utils module -------------------------- -Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`, -and :py:class:`SamplingIterator`. - ---- - .. automodule:: pyutils.iter_utils :members: :undoc-members: @@ -2288,10 +2269,6 @@ and :py:class:`SamplingIterator`. pyutils.list\_utils module -------------------------- -Utilities for dealing with Python lists. - ---- - .. automodule:: pyutils.list_utils :members: :undoc-members: @@ -2300,22 +2277,6 @@ Utilities for dealing with Python lists. pyutils.logging\_utils module ----------------------------- -This is a module that offers an opinionated take on how whole program -logging should be initialized and controlled. It uses standard Python -:py:mod:`logging` but gives you control, via commandline config, to: - - - Set the logging level of the program including overriding the - logging level for individual modules, - - Define the logging message format including easily adding a - PID/TID marker on all messages to help with multithreaded debugging, - - Control the destination (file, `sys.stderr`, syslog) of messages, - - Control the facility and logging level used with syslog, - - Squelch repeated messages, - - Log probalistically, - - Clear rogue logging handlers added by other imports. - ---- - .. automodule:: pyutils.logging_utils :members: :undoc-members: @@ -2324,13 +2285,6 @@ logging should be initialized and controlled. It uses standard Python pyutils.math\_utils module -------------------------- -Helper utilities that are "mathy" such as a :py:class:`NumericPopulation` that -makes population summary statistics available to your code quickly, GCD -computation, literate float truncation, percentage <-> multiplier, prime -number determination, etc... - ---- - .. automodule:: pyutils.math_utils :members: :undoc-members: @@ -2339,10 +2293,6 @@ number determination, etc... pyutils.misc\_utils module -------------------------- -Miscellaneous utilities: are we running as root, and is a debugger attached? - ---- - .. automodule:: pyutils.misc_utils :members: :undoc-members: @@ -2351,13 +2301,6 @@ Miscellaneous utilities: are we running as root, and is a debugger attached? pyutils.persistent module ------------------------- -Persistent defines a class hierarchy and decorator for creating -singleton classes that (optionally / conditionally) load their state -from some external location and (optionally / conditionally) save their -state to an external location at shutdown. - ---- - .. automodule:: pyutils.persistent :members: :undoc-members: @@ -2366,12 +2309,6 @@ state to an external location at shutdown. pyutils.remote\_worker module ----------------------------- -This module defines a helper that is invoked by the remote executor to -run pickled code on a remote machine. It is used by code marked with -`@parallelize(method=Method.REMOTE)` in the parallelize framework. - ---- - .. automodule:: pyutils.remote_worker :members: :undoc-members: @@ -2380,16 +2317,6 @@ run pickled code on a remote machine. It is used by code marked with pyutils.state\_tracker module ----------------------------- -This module defines several classes (:py:class:`StateTracker`, -:py:class:`AutomaticStateTracker`, and -:py:class:`WaitableAutomaticStateTracker`) that can be used as base -classes. These class patterns are meant to encapsulate and represent -state that dynamically changes. These classes update their state -(either automatically or when invoked to poll) and allow their callers -to wait on state changes. - ---- - .. automodule:: pyutils.state_tracker :members: :undoc-members: @@ -2411,9 +2338,14 @@ execute. pyutils.string\_utils module ---------------------------- -A bunch of utilities for dealing with strings. Based on a really great -starting library from Davide Zanotti, I've added a pile of other string -functions so hopefully it will handle all of your string-needs. +A bunch of utilities for dealing with strings. Based on a really +great starting library from Davide Zanotti (forked from +https://github.com/daveoncode/python-string-utils/tree/master/string_utils), +I've added a pile of other string functions (see `NOTICE +<[https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=NOTICE;hb=HEAD>`_ +file in the root of this project for a detailed account of what was +added and changed) so hopefully it will handle all of your +string-needs. --- @@ -2425,18 +2357,6 @@ functions so hopefully it will handle all of your string-needs. pyutils.text\_utils module -------------------------- -Utilities for dealing with and creating text chunks. For example: - - - Make a bar graph, - - make a spark line, - - left, right, center, justify text, - - word wrap text, - - indent text, - - create a header line, - - draw a box around some text. - ---- - .. automodule:: pyutils.text_utils :members: :undoc-members: @@ -2445,10 +2365,6 @@ Utilities for dealing with and creating text chunks. For example: pyutils.unittest\_utils module ------------------------------ -Utilities to support smarter unit tests. - ---- - .. automodule:: pyutils.unittest_utils :members: :undoc-members: @@ -2457,10 +2373,6 @@ Utilities to support smarter unit tests. pyutils.unscrambler module -------------------------- -Unscramble scrambled English words quickly. - ---- - .. automodule:: pyutils.unscrambler :members: :undoc-members: @@ -2469,10 +2381,6 @@ Unscramble scrambled English words quickly. pyutils.zookeeper module ------------------------ -A helper module for dealing with Zookeeper that adds some functionality. - ---- - .. automodule:: pyutils.zookeeper :members: :undoc-members: @@ -2481,8 +2389,6 @@ A helper module for dealing with Zookeeper that adds some functionality. Module contents --------------- ---- - .. automodule:: pyutils :members: :undoc-members: