python_utils.git
2 years agoCleanup unittest_utils trying to get clean lint.
Scott Gasch [Wed, 9 Feb 2022 03:59:23 +0000 (19:59 -0800)]
Cleanup unittest_utils trying to get clean lint.

2 years agoChange the weights in the default executor's remote worker pool.
Scott Gasch [Wed, 9 Feb 2022 03:55:34 +0000 (19:55 -0800)]
Change the weights in the default executor's remote worker pool.

2 years agoKeep requirements up-to-date.
Scott Gasch [Wed, 9 Feb 2022 02:27:16 +0000 (18:27 -0800)]
Keep requirements up-to-date.

2 years agoMore cleanup, yey!
Scott Gasch [Wed, 9 Feb 2022 01:46:56 +0000 (17:46 -0800)]
More cleanup, yey!

2 years agoMore cleanup.
Scott Gasch [Tue, 8 Feb 2022 22:38:15 +0000 (14:38 -0800)]
More cleanup.

2 years agoHook in pylint to the pre-commit hook and start to fix some of its
Scott Gasch [Tue, 8 Feb 2022 21:08:52 +0000 (13:08 -0800)]
Hook in pylint to the pre-commit hook and start to fix some of its
warnings.  It seems pickier than flake8 which is probably a good
thing.

2 years agoHow to do the thing.
Scott Gasch [Mon, 7 Feb 2022 21:56:00 +0000 (13:56 -0800)]
How to do the thing.

2 years agoAdds unittest.
Scott Gasch [Mon, 7 Feb 2022 18:55:09 +0000 (10:55 -0800)]
Adds unittest.

2 years agoAdd a simple test to google_assistant code.
Scott Gasch [Mon, 7 Feb 2022 18:34:09 +0000 (10:34 -0800)]
Add a simple test to google_assistant code.

2 years agoIgnore integration test results in code coverage report.
Scott Gasch [Mon, 7 Feb 2022 02:36:50 +0000 (18:36 -0800)]
Ignore integration test results in code coverage report.

2 years agoAdd a doctest to type_utils.py.
Scott Gasch [Mon, 7 Feb 2022 02:34:05 +0000 (18:34 -0800)]
Add a doctest to type_utils.py.

2 years agoGet mypy to stop with this fucking "has no attribute" shit when
Scott Gasch [Mon, 7 Feb 2022 00:35:16 +0000 (16:35 -0800)]
Get mypy to stop with this fucking "has no attribute" shit when
importing a submodule.  I am probably doing something wrong here but I
have no idea what.

2 years agoGive deps more time to run.
Scott Gasch [Sun, 6 Feb 2022 23:14:14 +0000 (15:14 -0800)]
Give deps more time to run.

2 years agoAdd another profanity.
Scott Gasch [Sun, 6 Feb 2022 23:12:38 +0000 (15:12 -0800)]
Add another profanity.

2 years agoAdd a simple test for waitable presence.
Scott Gasch [Sun, 6 Feb 2022 21:45:21 +0000 (13:45 -0800)]
Add a simple test for waitable presence.

2 years agoAdd doctest to misc_utils.
Scott Gasch [Sat, 5 Feb 2022 22:09:59 +0000 (14:09 -0800)]
Add doctest to misc_utils.

2 years agoAdd a bunch of __init__.py's; maybe this will fix import
Scott Gasch [Sat, 5 Feb 2022 22:06:25 +0000 (14:06 -0800)]
Add a bunch of __init__.py's; maybe this will fix import
problems with mypy?

2 years agoChange threshold.
Scott Gasch [Sat, 5 Feb 2022 18:04:06 +0000 (10:04 -0800)]
Change threshold.

2 years agoMissed one.
Scott Gasch [Sat, 5 Feb 2022 17:41:31 +0000 (09:41 -0800)]
Missed one.

2 years agoAdd coding comments for files with utf8 characters in there.
Scott Gasch [Sat, 5 Feb 2022 17:38:32 +0000 (09:38 -0800)]
Add coding comments for files with utf8 characters in there.

2 years agoAdd sparkline helper.
Scott Gasch [Sat, 5 Feb 2022 17:30:07 +0000 (09:30 -0800)]
Add sparkline helper.

2 years agoAdds some doctests to site_config.
Scott Gasch [Fri, 4 Feb 2022 20:18:03 +0000 (12:18 -0800)]
Adds some doctests to site_config.

2 years agoHave bootstrap log the site_config at program initialization.
Scott Gasch [Fri, 4 Feb 2022 20:13:37 +0000 (12:13 -0800)]
Have bootstrap log the site_config at program initialization.

2 years agoStop trying to cache mac addresses from house and cabin in the same
Scott Gasch [Fri, 4 Feb 2022 17:48:56 +0000 (09:48 -0800)]
Stop trying to cache mac addresses from house and cabin in the same
file.

2 years agoFix a bug in device.py around the type of keywords, add logging and
Scott Gasch [Fri, 4 Feb 2022 16:23:14 +0000 (08:23 -0800)]
Fix a bug in device.py around the type of keywords, add logging and
add manage_switch_off_timers to the deps run at each checkin.

2 years agoMake it clear in arg message that this is a library. Dependency
Scott Gasch [Fri, 4 Feb 2022 03:39:40 +0000 (19:39 -0800)]
Make it clear in arg message that this is a library.  Dependency
inject the index file name to the c'tor to override the flag.

2 years agoMake it possible for users of this library to override argparse's
Scott Gasch [Fri, 4 Feb 2022 03:37:38 +0000 (19:37 -0800)]
Make it possible for users of this library to override argparse's
epilog message.

2 years agoAdd cron.py, a pretty important (and easy) one.
Scott Gasch [Fri, 4 Feb 2022 03:29:59 +0000 (19:29 -0800)]
Add cron.py, a pretty important (and easy) one.

2 years agoMake this thing actually work.
Scott Gasch [Thu, 3 Feb 2022 23:47:25 +0000 (15:47 -0800)]
Make this thing actually work.

2 years agoAdd a formatter that handles raw help text, optionally.
Scott Gasch [Thu, 3 Feb 2022 23:44:32 +0000 (15:44 -0800)]
Add a formatter that handles raw help text, optionally.

2 years agoMake the parallelize intergration test short enough to run at
Scott Gasch [Thu, 3 Feb 2022 23:42:44 +0000 (15:42 -0800)]
Make the parallelize intergration test short enough to run at
checkin time and enable it.

2 years agoAdd a "unittest" that runs stuff like reminder and unscramble
Scott Gasch [Thu, 3 Feb 2022 22:41:29 +0000 (14:41 -0800)]
Add a "unittest" that runs stuff like reminder and unscramble
etc... to just make sure they don't crash.

2 years agoChange settings in flake8 and black.
Scott Gasch [Thu, 3 Feb 2022 22:18:37 +0000 (14:18 -0800)]
Change settings in flake8 and black.

2 years agoTake the lock before unlinking the mmap'ed shared memory to ensure
Scott Gasch [Thu, 3 Feb 2022 22:04:07 +0000 (14:04 -0800)]
Take the lock before unlinking the mmap'ed shared memory to ensure
all writes are finished and visible.

2 years agoDisallow where're too.
Scott Gasch [Thu, 3 Feb 2022 19:53:51 +0000 (11:53 -0800)]
Disallow where're too.

2 years agoMerge branch 'master' of ssh://git.house/usr/local/git/base/python_utils
Scott Gasch [Thu, 3 Feb 2022 19:38:01 +0000 (11:38 -0800)]
Merge branch 'master' of ssh://git.house/usr/local/git/base/python_utils

2 years agoContractions and stuff in string_utils.
Scott Gasch [Thu, 3 Feb 2022 19:36:13 +0000 (11:36 -0800)]
Contractions and stuff in string_utils.

2 years agoContractions and stuff in string_utils.
Scott [Thu, 3 Feb 2022 19:18:21 +0000 (11:18 -0800)]
Contractions and stuff in string_utils.

2 years agoLet's call the base class' c'tor first, eh?
Scott [Thu, 3 Feb 2022 07:37:32 +0000 (23:37 -0800)]
Let's call the base class' c'tor first, eh?

2 years agoStop spamming the log from bg.
Scott [Thu, 3 Feb 2022 07:36:39 +0000 (23:36 -0800)]
Stop spamming the log from bg.

2 years agoFix a bug in histogram.
Scott [Thu, 3 Feb 2022 07:35:38 +0000 (23:35 -0800)]
Fix a bug in histogram.

2 years agoLet's be explicit with asserts; there was a bug in histogram
Scott [Wed, 2 Feb 2022 21:47:36 +0000 (13:47 -0800)]
Let's be explicit with asserts; there was a bug in histogram
caused by assert foo when foo was an int with valid valid 0.

2 years agoMake subdirs type clean too.
Scott [Wed, 2 Feb 2022 21:09:30 +0000 (13:09 -0800)]
Make subdirs type clean too.

2 years agoUsed isort to sort imports. Also added to the git pre-commit hook.
Scott [Wed, 2 Feb 2022 20:13:27 +0000 (12:13 -0800)]
Used isort to sort imports.  Also added to the git pre-commit hook.

2 years agomypy clean!
Scott [Wed, 2 Feb 2022 19:09:18 +0000 (11:09 -0800)]
mypy clean!

2 years agoFix a problem introduced with previous type cleanup change.
Scott [Wed, 2 Feb 2022 18:35:08 +0000 (10:35 -0800)]
Fix a problem introduced with previous type cleanup change.

2 years agoMore type annotations.
Scott [Wed, 2 Feb 2022 17:14:12 +0000 (09:14 -0800)]
More type annotations.

2 years agoMore mypy cleanup.
Scott [Wed, 2 Feb 2022 16:31:42 +0000 (08:31 -0800)]
More mypy cleanup.

2 years agoMore mypy cleanup... ugh.
Scott [Wed, 2 Feb 2022 04:14:13 +0000 (20:14 -0800)]
More mypy cleanup... ugh.

2 years agoMyre mypy fixes.
Scott [Tue, 1 Feb 2022 05:38:46 +0000 (21:38 -0800)]
Myre mypy fixes.

2 years agoClean up mypy type checker errors.
Scott [Tue, 1 Feb 2022 05:03:41 +0000 (21:03 -0800)]
Clean up mypy type checker errors.

2 years agoFix a couple of bugs in date parsing.
Scott [Mon, 31 Jan 2022 20:00:11 +0000 (12:00 -0800)]
Fix a couple of bugs in date parsing.

2 years agoAdds a quiet option to executor shutdown.
Scott [Mon, 31 Jan 2022 18:03:43 +0000 (10:03 -0800)]
Adds a quiet option to executor shutdown.

2 years agoTweak histograms in executors to include seconds label.
Scott [Mon, 31 Jan 2022 17:58:30 +0000 (09:58 -0800)]
Tweak histograms in executors to include seconds label.

2 years agoMake histogram auto-format labels.
Scott [Mon, 31 Jan 2022 17:55:30 +0000 (09:55 -0800)]
Make histogram auto-format labels.

2 years agoIf percent > 0, even if it's very small, render some little sliver of
Scott [Mon, 31 Jan 2022 17:13:00 +0000 (09:13 -0800)]
If percent > 0, even if it's very small, render some little sliver of
a bar.

2 years agoUse a label formatter for prettier histograms.
Scott [Mon, 31 Jan 2022 17:08:57 +0000 (09:08 -0800)]
Use a label formatter for prettier histograms.

2 years agoClean up histogram __repr__
Scott [Mon, 31 Jan 2022 17:04:07 +0000 (09:04 -0800)]
Clean up histogram __repr__

2 years agoMake run_tests.sh re-run a subset of the tests w/o coverage
Scott [Mon, 31 Jan 2022 17:02:59 +0000 (09:02 -0800)]
Make run_tests.sh re-run a subset of the tests w/o coverage
instrumentation even in --coverage mode.

2 years ago--unittests_ignore_perf shouldn't mess with the database / file
Scott [Mon, 31 Jan 2022 06:57:29 +0000 (22:57 -0800)]
--unittests_ignore_perf shouldn't mess with the database / file
layer at all.  When the db was down some of tests failed even
though they didn't care about perf.  Let's not do that.

2 years agoFix a recent bug in executors. Thread executor needs to return
Scott [Mon, 31 Jan 2022 06:09:43 +0000 (22:09 -0800)]
Fix a recent bug in executors.  Thread executor needs to return
its future.

2 years agoChange locking boundaries for shared dict. Add a unit test.
Scott [Mon, 31 Jan 2022 05:29:34 +0000 (21:29 -0800)]
Change locking boundaries for shared dict.  Add a unit test.
Make smart_futures re-raise exceptions that happened in futures.
Mess with file_utils.

2 years agoImprove the locking decorators; @synchronized(lock).
Scott [Mon, 31 Jan 2022 01:37:58 +0000 (17:37 -0800)]
Improve the locking decorators; @synchronized(lock).

2 years agoAdds a shutdown_if_idle method to the executors and also eliminates
Scott [Mon, 31 Jan 2022 01:36:47 +0000 (17:36 -0800)]
Adds a shutdown_if_idle method to the executors and also eliminates
a multiple-shutdown problem encountered when using them.

2 years agoAdds MIT license / copyright to string utils from original source
Scott [Mon, 31 Jan 2022 01:34:51 +0000 (17:34 -0800)]
Adds MIT license / copyright to string utils from original source
of code.

2 years agoAdd MIT License and copyright from the shared dict implementation
Scott [Mon, 31 Jan 2022 01:30:11 +0000 (17:30 -0800)]
Add MIT License and copyright from the shared dict implementation
I based this code on.

2 years agoAdd a MP shared dict.
Scott [Mon, 31 Jan 2022 01:25:59 +0000 (17:25 -0800)]
Add a MP shared dict.

2 years agoAdd some new helper methods to file utils.
Scott [Sun, 30 Jan 2022 22:53:33 +0000 (14:53 -0800)]
Add some new helper methods to file utils.

2 years agoOptionally surface exceptions that happen under executors by reading
Scott [Sun, 30 Jan 2022 22:49:52 +0000 (14:49 -0800)]
Optionally surface exceptions that happen under executors by reading
them off the returned futures.

2 years agoDo not adjust task count from a child process or thread; always
Scott [Sun, 30 Jan 2022 22:48:32 +0000 (14:48 -0800)]
Do not adjust task count from a child process or thread; always
do it in the launcher thread/process.

2 years agoGet rid of "fuzzy matches" in favor of a window size param.
Scott [Fri, 28 Jan 2022 19:05:03 +0000 (11:05 -0800)]
Get rid of "fuzzy matches" in favor of a window size param.

2 years agoAdding test code trying to improve test coverage.
Scott [Thu, 27 Jan 2022 21:42:09 +0000 (13:42 -0800)]
Adding test code trying to improve test coverage.

2 years agoStarted using flake8 to lint; removed some of its warnings.
Scott [Thu, 27 Jan 2022 20:09:50 +0000 (12:09 -0800)]
Started using flake8 to lint; removed some of its warnings.

2 years agoAdd --trace_memory to bootstrap.
Scott [Thu, 27 Jan 2022 19:03:36 +0000 (11:03 -0800)]
Add --trace_memory to bootstrap.

2 years agoAdd --run_profiler option.
Scott [Thu, 27 Jan 2022 18:34:30 +0000 (10:34 -0800)]
Add --run_profiler option.

2 years agoMake the worker selection heuristics work harder to avoid backup
Scott [Thu, 27 Jan 2022 17:29:07 +0000 (09:29 -0800)]
Make the worker selection heuristics work harder to avoid backup
bundles scheduled on the same machine as their primary.

2 years agoTesting auto formatting
Scott [Thu, 27 Jan 2022 16:45:48 +0000 (08:45 -0800)]
Testing auto formatting

2 years agoAnother formatter test, whee...
Scott [Thu, 27 Jan 2022 05:50:11 +0000 (21:50 -0800)]
Another formatter test, whee...

2 years agoTrying to get black to run in git's pre-commit hook and actually
Scott [Thu, 27 Jan 2022 05:46:30 +0000 (21:46 -0800)]
Trying to get black to run in git's pre-commit hook and actually
modify the files...

2 years agoAhem. Still running black?
Scott [Thu, 27 Jan 2022 05:35:20 +0000 (21:35 -0800)]
Ahem.  Still running black?

2 years agoRan black code formatter on everything.
Scott [Thu, 27 Jan 2022 05:34:26 +0000 (21:34 -0800)]
Ran black code formatter on everything.

2 years agoRemove missing import.
Scott [Thu, 27 Jan 2022 05:24:56 +0000 (21:24 -0800)]
Remove missing import.

2 years agoReformatting.
Scott [Thu, 27 Jan 2022 05:19:41 +0000 (21:19 -0800)]
Reformatting.

2 years agoFix missing f's.
Scott [Thu, 27 Jan 2022 05:18:33 +0000 (21:18 -0800)]
Fix missing f's.

2 years agoReformatting.
Scott [Wed, 26 Jan 2022 23:36:05 +0000 (15:36 -0800)]
Reformatting.

2 years agoMake sanity check optional.
Scott [Wed, 26 Jan 2022 19:26:55 +0000 (11:26 -0800)]
Make sanity check optional.

2 years agoSanity check list is sorted before binary search.
Scott [Wed, 26 Jan 2022 19:24:46 +0000 (11:24 -0800)]
Sanity check list is sorted before binary search.

2 years agoMinor tweaks.
Scott [Wed, 26 Jan 2022 19:15:14 +0000 (11:15 -0800)]
Minor tweaks.

2 years agoAdd type hints and cleanup.
Scott [Wed, 26 Jan 2022 18:31:53 +0000 (10:31 -0800)]
Add type hints and cleanup.

2 years agoMake unscrambler use a cache per instance instead of module and
Scott [Wed, 26 Jan 2022 17:28:53 +0000 (09:28 -0800)]
Make unscrambler use a cache per instance instead of module and
make it have non-static methods.

2 years agoUnscrambler.py is no longer a utility; see ~bin/unscramble.py.
Scott [Wed, 26 Jan 2022 06:16:24 +0000 (22:16 -0800)]
Unscrambler.py is no longer a utility; see ~bin/unscramble.py.

2 years agoClarify instructions for repopulation.
Scott [Wed, 26 Jan 2022 06:15:08 +0000 (22:15 -0800)]
Clarify instructions for repopulation.

2 years agoAdds unscrambler which contains the guts of a jumble/scrabble player
Scott [Wed, 26 Jan 2022 06:05:25 +0000 (22:05 -0800)]
Adds unscrambler which contains the guts of a jumble/scrabble player
I wrote outside of here.

2 years agoEnable scp compression on remote executor to try to improve data
Scott [Tue, 25 Jan 2022 04:20:16 +0000 (20:20 -0800)]
Enable scp compression on remote executor to try to improve data
xfer times.

2 years agoFix a newly introduced bug in run_tests.sh.
Scott [Tue, 25 Jan 2022 04:19:26 +0000 (20:19 -0800)]
Fix a newly introduced bug in run_tests.sh.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 23:55:24 +0000 (15:55 -0800)]
Improve run_tests.sh

2 years agoReport overall runtime in periodic status dumps.
Scott [Mon, 24 Jan 2022 23:42:09 +0000 (15:42 -0800)]
Report overall runtime in periodic status dumps.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 22:55:09 +0000 (14:55 -0800)]
Improve run_tests.sh

2 years agoMake parallelize remember to shutdown the default executors atexit.
Scott [Mon, 24 Jan 2022 22:51:17 +0000 (14:51 -0800)]
Make parallelize remember to shutdown the default executors atexit.

2 years agoAdd repro instructions for coverage in run_tests.sh.
Scott [Mon, 24 Jan 2022 22:48:49 +0000 (14:48 -0800)]
Add repro instructions for coverage in run_tests.sh.