library grew into pyutils: a set of collections, helpers and utilities
that I find useful and hope you will too.
-Code is under `src/pyutils/`. Most code includes documentation and inline
-doctests.
+Code is under `src/pyutils/
+<https://wannabe.guru.org/gitweb/?p=pyutils.git;a=tree;f=src/pyutils;h=e716e14b7a895e5c6206af90f4628bf756f040fe;hb=HEAD>`_.
+Most code includes inlint documentation and doctests. I've tried to
+organize it into logical packages based on the code's functionality.
+Note that when words would collide with a Python library or reserved
+word I've used a 'z' at the end, e.g. 'collectionz' instead of
+'collections', 'typez' instead of 'type', etc...
-Unit and integration tests are under `tests/*`. To run all tests::
+There's some example code that uses various features of this project checked
+in under `examples/ <https://wannabe.guru.org/gitweb/?p=pyutils.git;a=tree;f=examples;h=d9744bf2b171ba7a9ff21ae1d3862b673647fff4;hb=HEAD>`_ that you can check out.
+
+Unit and integration tests are under `tests/
+<https://wannabe.guru.org/gitweb/?p=pyutils.git;a=tree;f=tests;h=8c303f23cd89b6d2e4fbf214a5c7dcc0941151b4;hb=HEAD>`_.
+To run all tests::
cd tests/
./run_tests.py --all [--coverage]
-See the README under `tests/` and the code of `run_tests.py` for more
-options / information.
+See the `README <https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=tests/README;hb=HEAD>`_ under `tests/` and the code of `run_tests.py` for more
+options / information about running tests.
This package generates Sphinx docs which are available at:
https://wannabe.guru.org/pydocs/pyutils/pyutils.html
Package code is checked into a local git server and available to clone
-from git at https://wannabe.guru.org/git/pyutils.git or from a web browser
-at:
+from git at https://wannabe.guru.org/git/pyutils.git or to view in a
+web browser at:
https://wannabe.guru.org/gitweb/?p=pyutils.git;a=summary
For a long time this was just a local library on my machine that my
-tools imported but I've now decided to release it on PyPi. Early
+tools imported but I've now decided to release it on PyPi. Earlier
development happened in a different git repo:
https://wannabe.guru.org/gitweb/?p=python_utils.git;a=summary
-I hope you find this useful. LICENSE and NOTICE describe reusing this
-code and where everything came from. Drop me a line if you are using
-this, find a bug, have a question, or have a suggestion:
+The `LICENSE
+<https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=LICENSE;hb=HEAD>`_
+and `NOTICE
+<[https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=NOTICE;hb=HEAD>`_
+files at the root of the project describe reusing this code and where
+everything came from. Drop me a line if you are using this, find a
+bug, have a question, or have a suggestion:
.. toctree::
:maxdepth: 4
+ :name: mastertoc
pyutils.collectionz
pyutils.compress
-------------------
This file mainly contains code for changing the nature of text printed
-to the console via ANSI escape sequences. e.g. it can be used to emit
-text that is bolded, underlined, italicised, colorized, etc...
+to the console via ANSI escape sequences. For example, it can be used
+to emit text that is bolded, underlined, italicised, colorized, etc...
+
+It does *not* contain ANSI escape sequences that do things like move
+the cursor or record/restore its position. It is focused on text style
+only.
-It also contains a colorizing context that will apply color patterns
-based on regular expressions to any data emitted to stdout that may be
-useful in adding color to other programs' outputs, for instance.
+The file includes a colorizing context that will apply color patterns
+based on regular expressions / callables to any data emitted to stdout
+that may be useful in adding color to other programs' outputs, for
+instance.
-Here are the predefined color names it knows:
+In addition, it contains a mapping from color name to RGB value that it
+uses to enable friendlier color names in most of its functions. Here
+is the list of predefined color names it knows:
.. raw:: html
.. note::
You can also use raw RGB values with this module so you do not have to use
- the predefined color names.
+ these predefined color names unless you want to.
---
pyutils.argparse\_utils module
------------------------------
-I use the Python internal :py:mod:`argparse` module for commandline argument
-parsing but found it lacking in some ways. This module contains code to
-fill those gaps. It include stuff like:
-
- - An :class:`argparse.Action` to create pairs of flags such as
- `--feature` and `--no_feature`.
- - A helper to parse and validate bools, IP addresses, MAC
- addresses, filenames, percentages, dates, datetimes, and
- durations passed as flags.
+I use and love the Python internal :py:mod:`argparse` module for
+commandline argument parsing but found it lacking in some ways. This
+module contains code to fill those gaps. See also :py:mod:`pyutils.config`.
---
pyutils.bootstrap module
------------------------
-Bootstrap module defines a decorator meant to wrap your main function.
-This decorator will do several things for you:
-
- - If your code uses the :file:`config.py` module (see below), it invokes
- :py:meth:`parse` automatically to initialize `config.config` from commandline
- flags, environment variables, or other sources.
- - It will optionally break into pdb in response to an unhandled
- Exception at the top level of your code.
- - It initializes logging for your program (see :file:`logging.py`).
- - It can optionally run a code and/or memory profiler on your code.
+The bootstrap module defines a decorator meant to wrap your main function.
+This is optional, of course: you can use this library without using the
+bootstrap decorator on your main. If you choose to use it, though, it will
+do some work for you automatically.
---
pyutils.config module
---------------------
-This module reads the program's configuration parameters from: the
-commandline (using :py:mod:`argparse`), environment variables and/or a
-shared zookeeper-based configuration. It stores this configuration
-state in a dict-like structure that can be queried by your code at
-runtime.
+The config module is an opinionated way to set up input parameters to
+your program. It is enabled by using the :py:mod:`pyutils.bootstrap`
+decorator around your main entry point or by simply calling
+:py:meth:`pyutils.config.parse` early in main (which is what
+:py:meth:`pyutils.bootstrap.initialize` does for you).
-It handles creating a nice `--help` message for your code.
-
-It can optionally react to dynamic config changes and change state at
-runtime (if and only if you name your flag with the *dynamic_* prefix
-and are using zookeeper-based configs).
-
-It can optionally save and retrieve sets of arguments from files on
-the local disk or on Zookeeper.
-
-All of my examples use this as does the pyutils library itself.
+If you use this module, input parameters to your program come from
+the commandline (and are configured using Python's :py:mod:`argparse`).
+But they can also be be augmented or replaced using saved configuration
+files stored either on the local filesystem or on Apache Zookeeper.
+There is a provision for enabling dynamic arguments (i.e. that can change
+during runtime) via Zookeeper (which is disabled by default).
---
pyutils.decorator\_utils module
-------------------------------
-This is a grab bag of decorators.
-
----
-
.. automodule:: pyutils.decorator_utils
:members:
:undoc-members:
pyutils.dict\_utils module
--------------------------
-A bunch of helpers for dealing with Python dicts.
-
----
-
.. automodule:: pyutils.dict_utils
:members:
:undoc-members:
pyutils.exec\_utils module
--------------------------
-Helper code for dealing with subprocesses.
-
----
-
.. automodule:: pyutils.exec_utils
:members:
:undoc-members:
pyutils.function\_utils module
------------------------------
-Helper util for dealing with functions.
-
----
-
.. automodule:: pyutils.function_utils
:members:
:undoc-members:
pyutils.id\_generator module
----------------------------
-Generate unique identifiers.
-
----
-
.. automodule:: pyutils.id_generator
:members:
:undoc-members:
pyutils.iter\_utils module
--------------------------
-Iterator utilities including a :py:class:`PeekingIterator`, :py:class:`PushbackIterator`,
-and :py:class:`SamplingIterator`.
-
----
-
.. automodule:: pyutils.iter_utils
:members:
:undoc-members:
pyutils.list\_utils module
--------------------------
-Utilities for dealing with Python lists.
-
----
-
.. automodule:: pyutils.list_utils
:members:
:undoc-members:
pyutils.logging\_utils module
-----------------------------
-This is a module that offers an opinionated take on how whole program
-logging should be initialized and controlled. It uses standard Python
-:py:mod:`logging` but gives you control, via commandline config, to:
-
- - Set the logging level of the program including overriding the
- logging level for individual modules,
- - Define the logging message format including easily adding a
- PID/TID marker on all messages to help with multithreaded debugging,
- - Control the destination (file, `sys.stderr`, syslog) of messages,
- - Control the facility and logging level used with syslog,
- - Squelch repeated messages,
- - Log probalistically,
- - Clear rogue logging handlers added by other imports.
-
----
-
.. automodule:: pyutils.logging_utils
:members:
:undoc-members:
pyutils.math\_utils module
--------------------------
-Helper utilities that are "mathy" such as a :py:class:`NumericPopulation` that
-makes population summary statistics available to your code quickly, GCD
-computation, literate float truncation, percentage <-> multiplier, prime
-number determination, etc...
-
----
-
.. automodule:: pyutils.math_utils
:members:
:undoc-members:
pyutils.misc\_utils module
--------------------------
-Miscellaneous utilities: are we running as root, and is a debugger attached?
-
----
-
.. automodule:: pyutils.misc_utils
:members:
:undoc-members:
pyutils.persistent module
-------------------------
-Persistent defines a class hierarchy and decorator for creating
-singleton classes that (optionally / conditionally) load their state
-from some external location and (optionally / conditionally) save their
-state to an external location at shutdown.
-
----
-
.. automodule:: pyutils.persistent
:members:
:undoc-members:
pyutils.remote\_worker module
-----------------------------
-This module defines a helper that is invoked by the remote executor to
-run pickled code on a remote machine. It is used by code marked with
-`@parallelize(method=Method.REMOTE)` in the parallelize framework.
-
----
-
.. automodule:: pyutils.remote_worker
:members:
:undoc-members:
pyutils.state\_tracker module
-----------------------------
-This module defines several classes (:py:class:`StateTracker`,
-:py:class:`AutomaticStateTracker`, and
-:py:class:`WaitableAutomaticStateTracker`) that can be used as base
-classes. These class patterns are meant to encapsulate and represent
-state that dynamically changes. These classes update their state
-(either automatically or when invoked to poll) and allow their callers
-to wait on state changes.
-
----
-
.. automodule:: pyutils.state_tracker
:members:
:undoc-members:
pyutils.string\_utils module
----------------------------
-A bunch of utilities for dealing with strings. Based on a really great
-starting library from Davide Zanotti, I've added a pile of other string
-functions so hopefully it will handle all of your string-needs.
+A bunch of utilities for dealing with strings. Based on a really
+great starting library from Davide Zanotti (forked from
+https://github.com/daveoncode/python-string-utils/tree/master/string_utils),
+I've added a pile of other string functions (see `NOTICE
+<[https://wannabe.guru.org/gitweb/?p=pyutils.git;a=blob_plain;f=NOTICE;hb=HEAD>`_
+file in the root of this project for a detailed account of what was
+added and changed) so hopefully it will handle all of your
+string-needs.
---
pyutils.text\_utils module
--------------------------
-Utilities for dealing with and creating text chunks. For example:
-
- - Make a bar graph,
- - make a spark line,
- - left, right, center, justify text,
- - word wrap text,
- - indent text,
- - create a header line,
- - draw a box around some text.
-
----
-
.. automodule:: pyutils.text_utils
:members:
:undoc-members:
pyutils.unittest\_utils module
------------------------------
-Utilities to support smarter unit tests.
-
----
-
.. automodule:: pyutils.unittest_utils
:members:
:undoc-members:
pyutils.unscrambler module
--------------------------
-Unscramble scrambled English words quickly.
-
----
-
.. automodule:: pyutils.unscrambler
:members:
:undoc-members:
pyutils.zookeeper module
------------------------
-A helper module for dealing with Zookeeper that adds some functionality.
-
----
-
.. automodule:: pyutils.zookeeper
:members:
:undoc-members:
Module contents
---------------
----
-
.. automodule:: pyutils
:members:
:undoc-members:
projects / pyutils.git / blobdiff