4 When I was writing little tools in Python and found myself implementing
5 a generally useful pattern I stuffed it into a local library. That
6 library grew into pyutils: a set of collections, helpers and utilities
7 that I find useful and hope you will too.
9 Code is under `src/pyutils/`. Most code includes documentation and inline
12 Unit and integration tests are under `tests/*`. To run all tests::
15 ./run_tests.py --all [--coverage]
17 See the README under `tests/` and the code of `run_tests.py` for more
18 options / information.
20 This package generates Sphinx docs which are available at:
22 https://wannabe.guru.org/pydocs/pyutils/pyutils.html
24 Package code is checked into a local git server and available to clone
25 from git at https://wannabe.guru.org/git/pyutils.git or from a web browser
28 https://wannabe.guru.org/gitweb/?p=pyutils.git;a=summary
30 For a long time this was just a local library on my machine that my
31 tools imported but I've now decided to release it on PyPi. Early
32 development happened in a different git repo:
34 https://wannabe.guru.org/gitweb/?p=python_utils.git;a=summary
36 I hope you find this useful. LICENSE and NOTICE describe reusing it
37 and where everything came from. Drop me a line if you are using this,
38 find a bug, or have a question:
64 This file mainly contains code for changing the nature of text printed
65 to the console via ANSI escape sequences. e.g. it can be used to emit
66 text that is bolded, underlined, italicised, colorized, etc...
68 It also contains a colorizing context that will apply color patterns
69 based on regular expressions to any data emitted to stdout that may be
70 useful in adding color to other programs' outputs, for instance.
72 .. automodule:: pyutils.ansi
77 pyutils.argparse\_utils module
78 ------------------------------
80 I use the Python internal `argparse` module for commandline argument
81 parsing but found it lacking in some ways. This module contains code to
82 fill those gaps. It include stuff like:
84 - An `argparse.Action` to create pairs of flags such as
85 `--feature` and `--no_feature`.
86 - A helper to parse and validate bools, IP addresses, MAC
87 addresses, filenames, percentages, dates, datetimes, and
88 durations passed as flags.
90 .. automodule:: pyutils.argparse_utils
95 pyutils.bootstrap module
96 ------------------------
98 Bootstrap module defines a decorator meant to wrap your main function.
99 This decorator will do several things for you:
101 - If your code uses the :file:`config.py` module (see below), it invokes
102 `parse` automatically to initialize `config.config` from commandline
103 flags, environment variables, or other sources.
104 - It will optionally break into pdb in response to an unhandled
105 Exception at the top level of your code.
106 - It initializes logging for your program (see :file:`logging.py`).
107 - It can optionally run a code and/or memory profiler on your code.
109 .. automodule:: pyutils.bootstrap
114 pyutils.config module
115 ---------------------
117 This module reads the program's configuration parameters from: the
118 commandline (using argparse), environment variables and/or a shared
119 zookeeper-based configuration. It stores this configuration state in
120 a dict-like structure that can be queried by your code at runtime.
122 It handles creating a nice `--help` message for your code.
124 It can optionally react to dynamic config changes and change state at
125 runtime (iff you name your flag with the *dynamic_* prefix and are
126 using zookeeper-based configs).
128 It can optionally save and retrieve sets of arguments from files on
129 the local disk or on zookeeper.
131 All of my examples use this as does the pyutils library itself.
133 .. automodule:: pyutils.config
138 pyutils.decorator\_utils module
139 -------------------------------
141 This is a grab bag of decorators.
143 .. automodule:: pyutils.decorator_utils
148 pyutils.dict\_utils module
149 --------------------------
151 A bunch of helpers for dealing with Python dicts.
153 .. automodule:: pyutils.dict_utils
158 pyutils.exec\_utils module
159 --------------------------
161 Helper code for dealing with subprocesses.
163 .. automodule:: pyutils.exec_utils
168 pyutils.function\_utils module
169 ------------------------------
171 Helper util for dealing with functions.
173 .. automodule:: pyutils.function_utils
178 pyutils.id\_generator module
179 ----------------------------
181 Generate unique identifiers.
183 .. automodule:: pyutils.id_generator
188 pyutils.iter\_utils module
189 --------------------------
191 Iterator utilities including a :class:PeekingIterator, :class:PushbackIterator,
192 and :class:SamplingIterator.
194 .. automodule:: pyutils.iter_utils
199 pyutils.list\_utils module
200 --------------------------
202 Utilities for dealing with Python lists.
204 .. automodule:: pyutils.list_utils
209 pyutils.logging\_utils module
210 -----------------------------
212 This is a module that offers an opinionated take on how whole program logging
213 should be initialized and controlled. It uses standard Python logging but gives
214 you control, via commandline config, to:
216 - Set the logging level of the program including overriding the
217 logging level for individual modules,
218 - Define the logging message format including easily adding a
219 PID/TID marker on all messages to help with multithreaded debugging,
220 - Control the destination (file, sys.stderr, syslog) of messages,
221 - Control the facility and logging level used with syslog,
222 - Squelch repeated messages,
223 - Log probalistically,
224 - Clear rogue logging handlers added by other imports.
226 .. automodule:: pyutils.logging_utils
231 pyutils.math\_utils module
232 --------------------------
234 Helper utilities that are "mathy" such as a :class:NumericPopulation that
235 makes population summary statistics available to your code quickly, GCD
236 computation, literate float truncation, percentage <-> multiplier, prime
237 number determination, etc...
239 .. automodule:: pyutils.math_utils
244 pyutils.misc\_utils module
245 --------------------------
247 Miscellaneous utilities: are we running as root, and is a debugger attached?
249 .. automodule:: pyutils.misc_utils
254 pyutils.persistent module
255 -------------------------
257 Persistent defines a class hierarchy and decorator for creating
258 singleton classes that (optionally/conditionally) load their state
259 from some external location and (optionally/conditionally) stave their
260 state to an external location at shutdown.
262 .. automodule:: pyutils.persistent
267 pyutils.remote\_worker module
268 -----------------------------
270 This module defines a helper that is invoked by the remote executor to
271 run pickled code on a remote machine. It is used by code marked with
272 @parallelize(method=Method.REMOTE) in the parallelize framework.
274 .. automodule:: pyutils.remote_worker
279 pyutils.state\_tracker module
280 -----------------------------
282 This module defines several classes (:class:StateTracker,
283 :class:AutomaticStateTracker, and
284 :class:WaitableAutomaticStateTracker) that can be used as base
285 classes. These class patterns are meant to encapsulate and represent
286 state that dynamically changes. These classes update their state
287 (either automatically or when invoked to poll) and allow their callers
288 to wait on state changes.
290 .. automodule:: pyutils.state_tracker
295 pyutils.stopwatch module
296 ------------------------
298 This is a stopwatch context that just times how long something took to
301 .. automodule:: pyutils.stopwatch
306 pyutils.string\_utils module
307 ----------------------------
309 A bunch of utilities for dealing with strings. Based on a really great
310 starting library from Davide Zanotti, I've added a pile of other string
311 functions so hopefully it will handle all of your string-needs.
313 .. automodule:: pyutils.string_utils
318 pyutils.text\_utils module
319 --------------------------
321 Utilities for dealing with and creating text chunks. For example:
325 - left, right, center, justify text,
328 - create a header line,
329 - draw a box around some text.
331 .. automodule:: pyutils.text_utils
336 pyutils.unittest\_utils module
337 ------------------------------
339 Utilities to support smarter unit tests.
341 .. automodule:: pyutils.unittest_utils
346 pyutils.unscrambler module
347 --------------------------
349 Unscramble scrambled English words quickly.
351 .. automodule:: pyutils.unscrambler
356 pyutils.zookeeper module
357 ------------------------
359 A helper module for dealing with Zookeeper that adds some functionality.
361 .. automodule:: pyutils.zookeeper
369 .. automodule:: pyutils