pyutils.typez package

Submodules

pyutils.typez.centcount module

An amount of money represented as an integral count of cents so as to avoid floating point artifacts. Multiplication and division are performed using floating point arithmetic but the quotient is cast back to an integer number thus truncating the result and avoiding floating point arithmetic artifacts. See details below.

The type guards against inadvertent aggregation of instances with non-matching currencies, the division of one CentCount by another, and has a strict mode which disallows comparison or aggregation with non-CentCount operands (i.e. no comparison or aggregation with literal numbers).

Note

Multiplication and division are performed by converting the CentCount into a float and operating on two floating point numbers. The result is then cast back to an int which loses precision beyond the 1-cent granularity in order to avoid floating point representation artifacts.

This can cause “problems” such as the one illustrated below:

>>> c = CentCount(100.00)
>>> c
100.00 USD
>>> c = c * 2
>>> c
200.00 USD
>>> c = c / 3
>>> c
66.66 USD

Two-thirds of $100.00 is $66.66666… which might be expected to round upwards to $66.67 but it does not because the int cast truncates the result. Be aware of this and decide whether it’s suitable for your application.

See also the pyutils.typez.Money class which uses Python Decimals (see: https://docs.python.org/3/library/decimal.html) to represent monetary amounts.

class pyutils.typez.centcount.CentCount(centcount: int | float | str | CentCount = 0, currency: str = 'USD', *, strict_mode=False)

Bases: object

A class for representing monetary amounts potentially with different currencies meant to avoid floating point rounding issues by treating amount as a simple integral count of cents.

Parameters:

• centcount (int | float | str | CentCount) – the amount of money being represented; this can be a float, int, CentCount or str.

• currency (str) – optionally declare the currency being represented by this instance. If provided it will guard against operations such as attempting to add it to non-matching currencies.

• strict_mode – if True, the instance created will object (raise) if compared or aggregated with non-CentCount objects; that is, strict_mode disallows comparison with literal numbers or aggregation with literal numbers.

Raises:

ValueError – invalid money string passed in

classmethod parse(s: str) → CentCount

Parses a string format monetary amount and returns a CentCount if possible.

Parameters:

s (str) – the string to be parsed

Raises:

ValueError – input string cannot be parsed.

Return type:

CentCount

pyutils.typez.histogram module

This is a text-based histogram class. It creates output like this:

A Histogram helper class. Creates outputs like this:

  [5..6): ▏                                                     ( 0.10% n=1)
  [6..7): █▋                                                    ( 0.49% n=5)
  [7..8): █████▏                                                ( 1.46% n=15)
  [8..9): ███████████▉                                          ( 3.42% n=35)
 [9..10): ██████████████████████▏                               ( 6.35% n=65)
[10..11): ██████████████████████████████████▌                   ( 9.86% n=101)
[11..12): ██████████████████████████████████████████████▏       (13.18% n=135)
[12..13): ████████████████████████████████████████████████████▉ (15.14% n=155)
[13..14): ████████████████████████████████████████████████████▉ (15.14% n=155)
[14..15): ██████████████████████████████████████████████▏       (13.18% n=135)
[15..16): ██████████████████████████████████▌                   ( 9.86% n=101)
[16..17): ██████████████████████▏                               ( 6.35% n=65)
[17..18): ███████████▉                                          ( 3.42% n=35)
[18..19): █████▏                                                ( 1.46% n=15)
[19..20): █▋                                                    ( 0.49% n=5)
[20..21): ▏                                                     ( 0.10% n=1)
--------------------------------------------------------------------------------
 [5..21):                                                         pop(Σn)=1024
                                                                  mean(μ)=12.500
                                                              median(p50)=12.000
                                                                 mode(Mo)=12.000
                                                                 stdev(σ)=2.500

class pyutils.typez.histogram.BucketDetails(num_populated_buckets: int = 0, max_population: int | None = None, last_bucket_start: int | None = None, lowest_start: int | None = None, highest_end: int | None = None, max_label_width: int | None = None)

Bases: object

A collection of details about the internal histogram buckets.

Parameters:

• num_populated_buckets (int)

• max_population (int | None)

• last_bucket_start (int | None)

• lowest_start (int | None)

• highest_end (int | None)

• max_label_width (int | None)

highest_end: int | None = None

The highest populated bucket’s ending point

last_bucket_start: int | None = None

The last bucket starting point

lowest_start: int | None = None

The lowest populated bucket’s starting point

max_label_width: int | None = None

The maximum label width (for display purposes)

max_population: int | None = None

The max population in a bucket currently

num_populated_buckets: int = 0

Count of populated buckets

class pyutils.typez.histogram.SimpleHistogram(buckets: List[Tuple[int, int]])

Bases: Generic[T]

A simple histogram.

C’tor.

Parameters:

buckets (List[Tuple[int, int]]) – a list of [start..end] tuples that define the buckets we are counting population in. See also n_evenly_spaced_buckets() to generate these buckets more easily.

Raises:

ValueError – buckets overlap

NEGATIVE_INFINITY = -inf

POSITIVE_INFINITY = inf

add_item(item: T) → bool

Adds a single item to the histogram (reculting in us incrementing the population in the correct bucket.

Parameters:

item (T) – the item to be added

Returns:

True if the item was successfully added or False if the item is not within the bounds established during class construction.

Return type:

bool

add_items(lst: Iterable[T]) → bool

Adds a collection of items to the histogram and increments the correct bucket’s population for each item.

Parameters:

lst (Iterable[T]) – An iterable of items to be added

Returns:

True if all items were added successfully or False if any item was not able to be added because it was not within the bounds established at object construction.

Return type:

bool

static n_evenly_spaced_buckets(min_bound: T, max_bound: T, n: int) → List[Tuple[int, int]]

A helper method for generating the buckets argument to our c’tor provided that you want N evenly spaced buckets.

Parameters:

• min_bound (T) – the minimum possible value

• max_bound (T) – the maximum possible value

• n (int) – how many buckets to create

Returns:

A list of bounds that define N evenly spaced buckets

Raises:

ValueError – min is not < max

Return type:

List[Tuple[int, int]]

pyutils.typez.money module

A class to represent money. This class represents monetary amounts as Python Decimals (see https://docs.python.org/3/library/decimal.html) internally.

The type guards against inadvertent aggregation of instances with non-matching currencies, the division of one Money by another, and has a strict mode which disallows comparison or aggregation with non-Money operands (i.e. no comparison or aggregation with literal numbers).

See also pyutils.typez.centcount.CentCount which represents monetary amounts as an integral number of cents.

class pyutils.typez.money.Money(amount: Decimal | str | float | int | Money = Decimal('0'), currency: str = 'USD', *, strict_mode=False)

Bases: object

A class for representing monetary amounts potentially with different currencies.

Parameters:

• amount (Decimal | str | float | int | Money) – the initial monetary amount to be represented; can be a Money, int, float, Decimal, str, etc…

• currency (str) – if provided, indicates what currency this amount is units of and guards against operations such as attempting to aggregate Money instances with non-matching currencies directly. If not provided defaults to “USD”.

• strict_mode – if True, disallows comparison or arithmetic operations between Money instances and any non-Money types (e.g. literal numbers).

Raises:

ValueError – unable to parse a money string

AMOUNT_RE = re.compile('^([+|-]?)(\\d+)(\\.\\d+)$')

classmethod parse(s: str) → Money

Parses a string an attempts to create a Money instance.

Parameters:

s (str) – the string to parse

Raises:

ValueError – unable to parse a string

Return type:

Money

round_fractional_cents()

Rounds fractional cents being represented. e.g.

>>> m = Money(100.00)
>>> m *= 2
>>> m /= 3

At this point the internal representation of m is a long Decimal:

>>> m.amount
Decimal('66.66666666666666666666666667')

It will be rendered by __repr__ reasonably:

>>> m
66.67 USD

If you want to round this long decimal representation, this method will do that for you:

>>> m.round_fractional_cents()
Decimal('66.67')
>>> m.amount
Decimal('66.67')
>>> m
66.67 USD

See also truncate_fractional_cents()

truncate_fractional_cents()

Truncates fractional cents being represented. e.g.

>>> m = Money(100.00)
>>> m *= 2
>>> m /= 3

At this point the internal representation of m is a long Decimal:

>>> m.amount
Decimal('66.66666666666666666666666667')

It will be rendered by __repr__ reasonably:

>>> m
66.67 USD

If you want to truncate this long decimal representation, this method will do that for you:

>>> m.truncate_fractional_cents()
Decimal('66.66')
>>> m.amount
Decimal('66.66')
>>> m
66.66 USD

See also round_fractional_cents()

pyutils.typez.persistent module

This module defines a class hierarchy (base class Persistent) and a decorator (@persistent_autoloaded_singleton) that can be used to create objects that load and save their state from some external storage location automatically, optionally and conditionally.

A Persistent is just a class with a Persistent.load() and Persistent.save() method. Various subclasses such as JsonFileBasedPersistent and PicklingFileBasedPersistent define these methods to, save data in a particular format. The details of where and whether to save are left to your code to decide by implementing interface methods like FileBasedPersistent.get_filename() and FileBasedPersistent.should_we_load_data().

The subclasses such as JsonZookeeperFileBasedPersistent and PicklingZookeeperFileBasedPersistent can be used to automatically persist state in a zookeeper instance such that state can be shared and restored by code running on different machines.

This module inculdes some helpers to make deciding whether to load persisted state easier such as was_file_written_today() and was_file_written_within_n_seconds().

Persistent classes are good for things backed by persisted state that is loaded all or most of the time. For example, the high score list of a game, the configuration settings of a tool, etc… Really anything that wants to save/load state from storage and not bother with the plumbing to do so.

class pyutils.typez.persistent.FileBasedPersistent

Bases: Persistent

A Persistent subclass that uses a file to save/load data and knows the conditions under which the state should be saved/loaded.

abstractmethod static get_filename() → str

Returns:

The full path of the file in which we are saving/loading data.

Return type:

str

abstractmethod get_persistent_data() → Any

Returns:

The raw state data read from the filesystem. Can be any format.

Return type:

Any

abstractmethod static should_we_load_data(filename: str) → bool

Returns:

True if we should load persisted state now or False otherwise.

Parameters:

filename (str)

Return type:

bool

abstractmethod static should_we_save_data(filename: str) → bool

Returns:

True if we should save our state now or False otherwise.

Parameters:

filename (str)

Return type:

bool

class pyutils.typez.persistent.JsonFileBasedPersistent(data: Any | None)

Bases: FileBasedPersistent

A class that stores its state in a JSON format file.

Example usage:

from pyutils.typez import persistent

@persistent.persistent_autoloaded_singleton()
class MyClass(persistent.JsonFileBasedPersistent):
    def __init__(self, data: Optional[dict[str, Any]]):
        # load already deserialized the JSON data for you; it's
        # a "cooked" JSON dict of string -> values, lists, dicts,
        # etc...
        if data:
            #initialize youself from data...
        else:
            # if desired, initialize an empty state object
            # when json_data isn't provided.

    @staticmethod
    @overrides
    def get_filename() -> str:
        return "/path/to/where/you/want/to/save/data.json"

    @staticmethod
    @overrides
    def should_we_save_data(filename: str) -> bool:
        return true_if_we_should_save_the_data_this_time()

    @staticmethod
    @overrides
    def should_we_load_data(filename: str) -> bool:
        return persistent.was_file_written_within_n_seconds(whatever)

# Persistent will handle the plumbing to instantiate your
# class from its persisted state iff the
# :meth:`should_we_load_data` says it's ok to.  It will also
# persist the current in memory state to disk at program exit
# iff the :meth:`should_we_save_data methods` says to.
c = MyClass()

You should override this.

Parameters:

data (Optional[Any])

classmethod load() → JsonFileBasedPersistent | None

Load this thing from somewhere and give back an instance which will become the global singleton and which may (see below) be saved (via save()) at program exit time.

Oh, in case this is handy, here’s a reminder how to write a factory method that doesn’t call the c’tor in python:

@classmethod
def load_from_somewhere(cls, somewhere):
    # Note: __new__ does not call __init__.
    obj = cls.__new__(cls)

    # Don't forget to call any polymorphic base class initializers
    super(MyClass, obj).__init__()

    # Load the piece(s) of obj that you want to from somewhere.
    obj._state = load_from_somewhere(somewhere)
    return obj

Parameters:

cls – the class (type) that is being instantiated. That is, the type to load.

Returns:

An instance of the requested type or None to indicate failure.

Return type:

JsonFileBasedPersistent | None

save() → bool

Raises:

Exception – failure to save to file.

Return type:

bool

class pyutils.typez.persistent.JsonZookeeperFileBasedPersistent(data: Any | None)

Bases: FileBasedPersistent

This class is like JsonFileBasedPersistent except that it persists state on a zookeeper instance.

You should override this.

Parameters:

data (Optional[Any])

classmethod load() → JsonZookeeperFileBasedPersistent | None

Raises:

Exception – failure to load from file.

Return type:

JsonZookeeperFileBasedPersistent | None

save() → bool

Save this thing somewhere that you’ll remember when someone calls load() later on in a way that makes sense to your code.

Return type:

bool

class pyutils.typez.persistent.PersistAtShutdown(value)

Bases: Enum

An enum to describe the conditions under which state is persisted to disk. This is passed as an argument to the decorator below and is used to indicate when to call save() on a Persistent subclass.

• NEVER: never call save()

• IF_NOT_LOADED: call save() as long as we did not successfully load() its state.

• ALWAYS: always call save()

ALWAYS = (2,)

IF_NOT_LOADED = (1,)

NEVER = (0,)

class pyutils.typez.persistent.Persistent

Bases: ABC

A base class of an object with a load/save method. Classes that are decorated with @persistent_autoloaded_singleton should subclass this and implement their save() and load() methods.

classmethod load() → Persistent | None

Load this thing from somewhere and give back an instance which will become the global singleton and which may (see below) be saved (via save()) at program exit time.

Oh, in case this is handy, here’s a reminder how to write a factory method that doesn’t call the c’tor in python:

@classmethod
def load_from_somewhere(cls, somewhere):
    # Note: __new__ does not call __init__.
    obj = cls.__new__(cls)

    # Don't forget to call any polymorphic base class initializers
    super(MyClass, obj).__init__()

    # Load the piece(s) of obj that you want to from somewhere.
    obj._state = load_from_somewhere(somewhere)
    return obj

Parameters:

cls – the class (type) that is being instantiated. That is, the type to load.

Returns:

An instance of the requested type or None to indicate failure.

Return type:

Persistent | None

abstractmethod save() → bool

Save this thing somewhere that you’ll remember when someone calls load() later on in a way that makes sense to your code.

Return type:

bool

class pyutils.typez.persistent.PicklingFileBasedPersistent(data: Any | None = None)

Bases: FileBasedPersistent

A class that stores its state in a file as pickled Python objects.

Example usage:

from pyutils.typez import persistent

@persistent.persistent_autoloaded_singleton()
class MyClass(persistent.PicklingFileBasedPersistent):
    def __init__(self, data: Optional[Whatever]):
        if data:
            # initialize state from data
        else:
            # if desired, initialize an "empty" object with new state.

    @staticmethod
    @overrides
    def get_filename() -> str:
        return "/path/to/where/you/want/to/save/data.bin"

    @staticmethod
    @overrides
    def should_we_save_data(filename: str) -> bool:
        return true_if_we_should_save_the_data_this_time()

    @staticmethod
    @overrides
    def should_we_load_data(filename: str) -> bool:
        return persistent.was_file_written_within_n_seconds(whatever)

# Persistent will handle the plumbing to instantiate your class
# from its persisted state iff the :meth:`should_we_load_data`
# says it's ok to.  It will also persist the current in-memory
# state to disk at program exit iff the :meth:`should_we_save_data`
# methods says to.
c = MyClass()

You should override this.

Parameters:

data (Optional[Any])

classmethod load() → PicklingFileBasedPersistent | None

Load this thing from somewhere and give back an instance which will become the global singleton and which may (see below) be saved (via save()) at program exit time.

Oh, in case this is handy, here’s a reminder how to write a factory method that doesn’t call the c’tor in python:

@classmethod
def load_from_somewhere(cls, somewhere):
    # Note: __new__ does not call __init__.
    obj = cls.__new__(cls)

    # Don't forget to call any polymorphic base class initializers
    super(MyClass, obj).__init__()

    # Load the piece(s) of obj that you want to from somewhere.
    obj._state = load_from_somewhere(somewhere)
    return obj

Parameters:

cls – the class (type) that is being instantiated. That is, the type to load.

Returns:

An instance of the requested type or None to indicate failure.

Return type:

PicklingFileBasedPersistent | None

save() → bool

Raises:

Exception – failure to save to file.

Return type:

bool

class pyutils.typez.persistent.PicklingZookeeperFileBasedPersistent(data: Any | None = None)

Bases: FileBasedPersistent

This class is like PicklingFileBasedPersistent except for that it persists state on a zookeeper instance.

You should override this.

Parameters:

data (Optional[Any])

classmethod load() → PicklingZookeeperFileBasedPersistent | None

Load this thing from somewhere and give back an instance which will become the global singleton and which may (see below) be saved (via save()) at program exit time.

Oh, in case this is handy, here’s a reminder how to write a factory method that doesn’t call the c’tor in python:

@classmethod
def load_from_somewhere(cls, somewhere):
    # Note: __new__ does not call __init__.
    obj = cls.__new__(cls)

    # Don't forget to call any polymorphic base class initializers
    super(MyClass, obj).__init__()

    # Load the piece(s) of obj that you want to from somewhere.
    obj._state = load_from_somewhere(somewhere)
    return obj

Parameters:

cls – the class (type) that is being instantiated. That is, the type to load.

Returns:

An instance of the requested type or None to indicate failure.

Return type:

PicklingZookeeperFileBasedPersistent | None

save() → bool

Raises:

Exception – failure to save to file.

Return type:

bool

class pyutils.typez.persistent.persistent_autoloaded_singleton(*, persist_at_shutdown: PersistAtShutdown = PersistAtShutdown.IF_NOT_LOADED)

Bases: object

A decorator that can be applied to a Persistent subclass (i.e. a class with save() and load() methods. The decorator will intercept attempts to instantiate the class via it’s c’tor and, instead, invoke the class’ load() to give it a chance to read state from somewhere persistent (disk, db, whatever). Subsequent calls to construct instances of the wrapped class will return a single, global instance (i.e. the wrapped class is must be a singleton).

If load() fails (returns None), the class’ c’tor is invoked with the original args as a fallback.

Based upon the value of the optional argument persist_at_shutdown argument, (NEVER, IF_NOT_LOADED, ALWAYS), the save() method of the class will be invoked just before program shutdown to give the class a chance to save its state somewhere.

Note

The implementations of save() and load() and where the class persists its state are details left to the Persistent implementation. Essentially this decorator just handles the plumbing of calling your save/load and appropriate times and creates a transparent global singleton whose state can be persisted between runs. See example implementations such as JsonFileBasedPersistent and PicklingFileBasedPersistent.

Parameters:

persist_at_shutdown (PersistAtShutdown)

pyutils.typez.persistent.was_file_written_today(filename: str) → bool

Convenience wrapper around was_file_written_within_n_seconds().

Parameters:

filename (str) – path / filename to check

Returns:

True if filename was written today.

Return type:

bool

>>> import os
>>> filename = f'/tmp/testing_persistent_py_{os.getpid()}'
>>> os.system(f'touch {filename}')
0
>>> was_file_written_today(filename)
True
>>> os.system(f'touch -d 1974-04-15T01:02:03.99 {filename}')
0
>>> was_file_written_today(filename)
False
>>> os.system(f'/bin/rm -f {filename}')
0
>>> was_file_written_today(filename)
False

pyutils.typez.persistent.was_file_written_within_n_seconds(filename: str, limit_seconds: int) → bool

Helper for determining persisted state staleness.

Parameters:

• filename (str) – the filename to check

• limit_seconds (int) – how fresh, in seconds, it must be

Returns:

True if filename was written within the past limit_seconds or False otherwise (or on error).

Return type:

bool

>>> import os
>>> filename = f'/tmp/testing_persistent_py_{os.getpid()}'
>>> os.system(f'touch {filename}')
0
>>> was_file_written_within_n_seconds(filename, 60)
True
>>> import time
>>> time.sleep(2.0)
>>> was_file_written_within_n_seconds(filename, 2)
False
>>> os.system(f'/bin/rm -f {filename}')
0
>>> was_file_written_within_n_seconds(filename, 60)
False

pyutils.typez.rate module

A class to represent a rate of change.

class pyutils.typez.rate.Rate(multiplier: float | None = None, *, percentage: float | None = None, percent_change: float | None = None)

Bases: object

A class to represent a rate of change.

Constructs a new Rate from a multiplier, percentage, or percent change. One and only one of these may be passed. These are a little confusing so here’s an example…

Note

A multiplier of 1.5x is the same as a percentage of 150% and is also the same as a 50% change. Let’s examine an original amount of 100. Multiplying it by a 1.5x multiplier yields 150. Multiplying it by 150% yields 150. Increasing it by 50% also yields 150.

Parameters:

• multiplier (float | None) – provides the number that you would multiply a base amount by to modify it

• percentage (float | None) – provides the multiplier as a percentage

• percent_change (float | None) – provides the multiplier as a percent change to the base amount

Raises:

ValueError – if more than one of percentage, percent_change and multiplier is provided

apply_to(other)

Applies the rate to a base number.

Parameters:

other – the base to apply the change rate to.

Returns:

The result after the change.

of(other)

Applies the rate to a base number.

Parameters:

other – the base to apply the change rate to.

Returns:

The result after the change.

pyutils.typez.type_utils module

Utility functions for dealing with typing.

pyutils.typez.type_utils.unwrap_optional(x: Any | None) → Any

Unwrap an Optional[Type] argument returning a Type value back. Use this to satisfy most type checkers that a value that could be None isn’t so as to drop the Optional typing hint.

Parameters:

x (Any | None) – an Optional[Type] argument

Returns:

If the Optional[Type] argument is non-None, return it. If the Optional[Type] argument is None, however, raise an exception.

Raises:

AssertionError – the parameter is, indeed, of NoneType.

Return type:

Any

>>> x: Optional[bool] = True
>>> unwrap_optional(x)
True

>>> y: Optional[str] = None
>>> unwrap_optional(y)
Traceback (most recent call last):
...
AssertionError: Argument to unwrap_optional was unexpectedly None

pyutils.typez.type_hints module

My type hints.

class pyutils.typez.type_hints.Cloneable(*args, **kwargs)

Bases: Protocol

Something that can be cloned.

clone() → None

Return type:

None

class pyutils.typez.type_hints.Closable(*args, **kwargs)

Bases: Protocol

Something that can be closed.

close() → None

Return type:

None

class pyutils.typez.type_hints.Comparable(*args, **kwargs)

Bases: Protocol

Anything that implements basic comparison methods such that it can be compared to other instances of the same type.

Check out functools.total_ordering() (https://docs.python.org/3/library/functools.html#functools.total_ordering) for an easy way to make your type comparable.

class pyutils.typez.type_hints.Runnable(*args, **kwargs)

Bases: Protocol

Something that can be run.

run() → None

Return type:

None

Module contents