X-Git-Url: https://wannabe.guru.org/gitweb/?a=blobdiff_plain;f=docs%2Fpyutils.rst;h=b7e07836efc09fd501f8748b37698422d4fcaf65;hb=84f53cd94d18a8d239216704a6a33c7dbf2fc6b9;hp=d8ba243646941be641eab803986f4fe38ec43fb8;hpb=1c60a8d9ae5a162f8d309b48f8a07ee6946f3e25;p=pyutils.git diff --git a/docs/pyutils.rst b/docs/pyutils.rst index d8ba243..b7e0783 100644 --- a/docs/pyutils.rst +++ b/docs/pyutils.rst @@ -1,6 +1,45 @@ pyutils package =============== +When I was writing little tools in Python and found myself implementing +a generally useful pattern I stuffed it into a local library. That +library grew into pyutils: a set of collections, helpers and utilities +that I find useful and hope you will too. + +Code is under `src/pyutils/`. Most code includes documentation and inline +doctests. + +Unit and integration tests are under `tests/*`. To run all tests:: + + cd tests/ + ./run_tests.py --all [--coverage] + +See the README under `tests/` and the code of `run_tests.py` for more +options / information. + +This package generates Sphinx docs which are available at: + + https://wannabe.guru.org/pydocs/pyutils/pyutils.html + +Package code is checked into a local git server and available to clone +from git at https://wannabe.guru.org/git/pyutils.git or from a web browser +at: + + https://wannabe.guru.org/gitweb/?p=pyutils.git;a=summary + +For a long time this was just a local library on my machine that my +tools imported but I've now decided to release it on PyPi. Early +development happened in a different git repo: + + https://wannabe.guru.org/gitweb/?p=python_utils.git;a=summary + +I hope you find this useful. LICENSE and NOTICE describe reusing it +and where everything came from. Drop me a line if you are using this, +find a bug, or have a question: + + --Scott Gasch (scott.gasch@gmail.com) + + Subpackages ----------- @@ -22,6 +61,14 @@ Submodules pyutils.ansi module ------------------- +This file mainly contains code for changing the nature of text printed +to the console via ANSI escape sequences. e.g. it can be used to emit +text that is bolded, underlined, italicised, colorized, etc... + +It also contains a colorizing context that will apply color patterns +based on regular expressions to any data emitted to stdout that may be +useful in adding color to other programs' outputs, for instance. + .. automodule:: pyutils.ansi :members: :undoc-members: @@ -30,6 +77,16 @@ pyutils.ansi module pyutils.argparse\_utils module ------------------------------ +I use the Python internal `argparse` module for commandline argument +parsing but found it lacking in some ways. This module contains code to +fill those gaps. It include stuff like: + + - An `argparse.Action` to create pairs of flags such as + `--feature` and `--no_feature`. + - A helper to parse and validate bools, IP addresses, MAC + addresses, filenames, percentages, dates, datetimes, and + durations passed as flags. + .. automodule:: pyutils.argparse_utils :members: :undoc-members: @@ -38,6 +95,17 @@ pyutils.argparse\_utils module pyutils.bootstrap module ------------------------ +Bootstrap module defines a decorator meant to wrap your main function. +This decorator will do several things for you: + + - If your code uses the :file:`config.py` module (see below), it invokes + `parse` automatically to initialize `config.config` from commandline + flags, environment variables, or other sources. + - It will optionally break into pdb in response to an unhandled + Exception at the top level of your code. + - It initializes logging for your program (see :file:`logging.py`). + - It can optionally run a code and/or memory profiler on your code. + .. automodule:: pyutils.bootstrap :members: :undoc-members: @@ -46,6 +114,22 @@ pyutils.bootstrap module pyutils.config module --------------------- +This module reads the program's configuration parameters from: the +commandline (using argparse), environment variables and/or a shared +zookeeper-based configuration. It stores this configuration state in +a dict-like structure that can be queried by your code at runtime. + +It handles creating a nice `--help` message for your code. + +It can optionally react to dynamic config changes and change state at +runtime (iff you name your flag with the *dynamic_* prefix and are +using zookeeper-based configs). + +It can optionally save and retrieve sets of arguments from files on +the local disk or on zookeeper. + +All of my examples use this as does the pyutils library itself. + .. automodule:: pyutils.config :members: :undoc-members: @@ -54,6 +138,8 @@ pyutils.config module pyutils.decorator\_utils module ------------------------------- +This is a grab bag of decorators. + .. automodule:: pyutils.decorator_utils :members: :undoc-members: @@ -62,6 +148,8 @@ pyutils.decorator\_utils module pyutils.dict\_utils module -------------------------- +A bunch of helpers for dealing with Python dicts. + .. automodule:: pyutils.dict_utils :members: :undoc-members: @@ -70,6 +158,8 @@ pyutils.dict\_utils module pyutils.exec\_utils module -------------------------- +Helper code for dealing with subprocesses. + .. automodule:: pyutils.exec_utils :members: :undoc-members: @@ -78,6 +168,8 @@ pyutils.exec\_utils module pyutils.function\_utils module ------------------------------ +Helper util for dealing with functions. + .. automodule:: pyutils.function_utils :members: :undoc-members: @@ -86,6 +178,8 @@ pyutils.function\_utils module pyutils.id\_generator module ---------------------------- +Generate unique identifiers. + .. automodule:: pyutils.id_generator :members: :undoc-members: @@ -94,6 +188,9 @@ pyutils.id\_generator module pyutils.iter\_utils module -------------------------- +Iterator utilities including a :class:PeekingIterator, :class:PushbackIterator, +and :class:SamplingIterator. + .. automodule:: pyutils.iter_utils :members: :undoc-members: @@ -102,6 +199,8 @@ pyutils.iter\_utils module pyutils.list\_utils module -------------------------- +Utilities for dealing with Python lists. + .. automodule:: pyutils.list_utils :members: :undoc-members: @@ -110,6 +209,20 @@ pyutils.list\_utils module pyutils.logging\_utils module ----------------------------- +This is a module that offers an opinionated take on how whole program logging +should be initialized and controlled. It uses standard Python logging but gives +you control, via commandline config, to: + + - Set the logging level of the program including overriding the + logging level for individual modules, + - Define the logging message format including easily adding a + PID/TID marker on all messages to help with multithreaded debugging, + - Control the destination (file, sys.stderr, syslog) of messages, + - Control the facility and logging level used with syslog, + - Squelch repeated messages, + - Log probalistically, + - Clear rogue logging handlers added by other imports. + .. automodule:: pyutils.logging_utils :members: :undoc-members: @@ -118,6 +231,11 @@ pyutils.logging\_utils module pyutils.math\_utils module -------------------------- +Helper utilities that are "mathy" such as a :class:NumericPopulation that +makes population summary statistics available to your code quickly, GCD +computation, literate float truncation, percentage <-> multiplier, prime +number determination, etc... + .. automodule:: pyutils.math_utils :members: :undoc-members: @@ -126,6 +244,8 @@ pyutils.math\_utils module pyutils.misc\_utils module -------------------------- +Miscellaneous utilities: are we running as root, and is a debugger attached? + .. automodule:: pyutils.misc_utils :members: :undoc-members: @@ -134,6 +254,11 @@ pyutils.misc\_utils module pyutils.persistent module ------------------------- +Persistent defines a class hierarchy and decorator for creating +singleton classes that (optionally/conditionally) load their state +from some external location and (optionally/conditionally) stave their +state to an external location at shutdown. + .. automodule:: pyutils.persistent :members: :undoc-members: @@ -142,6 +267,10 @@ pyutils.persistent module pyutils.remote\_worker module ----------------------------- +This module defines a helper that is invoked by the remote executor to +run pickled code on a remote machine. It is used by code marked with +@parallelize(method=Method.REMOTE) in the parallelize framework. + .. automodule:: pyutils.remote_worker :members: :undoc-members: @@ -150,6 +279,14 @@ pyutils.remote\_worker module pyutils.state\_tracker module ----------------------------- +This module defines several classes (:class:StateTracker, +:class:AutomaticStateTracker, and +:class:WaitableAutomaticStateTracker) that can be used as base +classes. These class patterns are meant to encapsulate and represent +state that dynamically changes. These classes update their state +(either automatically or when invoked to poll) and allow their callers +to wait on state changes. + .. automodule:: pyutils.state_tracker :members: :undoc-members: @@ -158,6 +295,9 @@ pyutils.state\_tracker module pyutils.stopwatch module ------------------------ +This is a stopwatch context that just times how long something took to +execute. + .. automodule:: pyutils.stopwatch :members: :undoc-members: @@ -166,6 +306,10 @@ pyutils.stopwatch module pyutils.string\_utils module ---------------------------- +A bunch of utilities for dealing with strings. Based on a really great +starting library from Davide Zanotti, I've added a pile of other string +functions so hopefully it will handle all of your string-needs. + .. automodule:: pyutils.string_utils :members: :undoc-members: @@ -174,6 +318,16 @@ pyutils.string\_utils module pyutils.text\_utils module -------------------------- +Utilities for dealing with and creating text chunks. For example: + + - Make a bar graph, + - make a spark line, + - left, right, center, justify text, + - word wrap text, + - indent text, + - create a header line, + - draw a box around some text. + .. automodule:: pyutils.text_utils :members: :undoc-members: @@ -182,6 +336,8 @@ pyutils.text\_utils module pyutils.unittest\_utils module ------------------------------ +Utilities to support smarter unit tests. + .. automodule:: pyutils.unittest_utils :members: :undoc-members: @@ -190,6 +346,8 @@ pyutils.unittest\_utils module pyutils.unscrambler module -------------------------- +Unscramble scrambled English words quickly. + .. automodule:: pyutils.unscrambler :members: :undoc-members: @@ -198,6 +356,8 @@ pyutils.unscrambler module pyutils.zookeeper module ------------------------ +A helper module for dealing with Zookeeper that adds some functionality. + .. automodule:: pyutils.zookeeper :members: :undoc-members: