python_utils.git
2 years agoAdd a "unittest" that runs stuff like reminder and unscramble
Scott Gasch [Thu, 3 Feb 2022 22:41:29 +0000 (14:41 -0800)]
Add a "unittest" that runs stuff like reminder and unscramble
etc... to just make sure they don't crash.

2 years agoChange settings in flake8 and black.
Scott Gasch [Thu, 3 Feb 2022 22:18:37 +0000 (14:18 -0800)]
Change settings in flake8 and black.

2 years agoTake the lock before unlinking the mmap'ed shared memory to ensure
Scott Gasch [Thu, 3 Feb 2022 22:04:07 +0000 (14:04 -0800)]
Take the lock before unlinking the mmap'ed shared memory to ensure
all writes are finished and visible.

2 years agoDisallow where're too.
Scott Gasch [Thu, 3 Feb 2022 19:53:51 +0000 (11:53 -0800)]
Disallow where're too.

2 years agoMerge branch 'master' of ssh://git.house/usr/local/git/base/python_utils
Scott Gasch [Thu, 3 Feb 2022 19:38:01 +0000 (11:38 -0800)]
Merge branch 'master' of ssh://git.house/usr/local/git/base/python_utils

2 years agoContractions and stuff in string_utils.
Scott Gasch [Thu, 3 Feb 2022 19:36:13 +0000 (11:36 -0800)]
Contractions and stuff in string_utils.

2 years agoContractions and stuff in string_utils.
Scott [Thu, 3 Feb 2022 19:18:21 +0000 (11:18 -0800)]
Contractions and stuff in string_utils.

2 years agoLet's call the base class' c'tor first, eh?
Scott [Thu, 3 Feb 2022 07:37:32 +0000 (23:37 -0800)]
Let's call the base class' c'tor first, eh?

2 years agoStop spamming the log from bg.
Scott [Thu, 3 Feb 2022 07:36:39 +0000 (23:36 -0800)]
Stop spamming the log from bg.

2 years agoFix a bug in histogram.
Scott [Thu, 3 Feb 2022 07:35:38 +0000 (23:35 -0800)]
Fix a bug in histogram.

2 years agoLet's be explicit with asserts; there was a bug in histogram
Scott [Wed, 2 Feb 2022 21:47:36 +0000 (13:47 -0800)]
Let's be explicit with asserts; there was a bug in histogram
caused by assert foo when foo was an int with valid valid 0.

2 years agoMake subdirs type clean too.
Scott [Wed, 2 Feb 2022 21:09:30 +0000 (13:09 -0800)]
Make subdirs type clean too.

2 years agoUsed isort to sort imports. Also added to the git pre-commit hook.
Scott [Wed, 2 Feb 2022 20:13:27 +0000 (12:13 -0800)]
Used isort to sort imports.  Also added to the git pre-commit hook.

2 years agomypy clean!
Scott [Wed, 2 Feb 2022 19:09:18 +0000 (11:09 -0800)]
mypy clean!

2 years agoFix a problem introduced with previous type cleanup change.
Scott [Wed, 2 Feb 2022 18:35:08 +0000 (10:35 -0800)]
Fix a problem introduced with previous type cleanup change.

2 years agoMore type annotations.
Scott [Wed, 2 Feb 2022 17:14:12 +0000 (09:14 -0800)]
More type annotations.

2 years agoMore mypy cleanup.
Scott [Wed, 2 Feb 2022 16:31:42 +0000 (08:31 -0800)]
More mypy cleanup.

2 years agoMore mypy cleanup... ugh.
Scott [Wed, 2 Feb 2022 04:14:13 +0000 (20:14 -0800)]
More mypy cleanup... ugh.

2 years agoMyre mypy fixes.
Scott [Tue, 1 Feb 2022 05:38:46 +0000 (21:38 -0800)]
Myre mypy fixes.

2 years agoClean up mypy type checker errors.
Scott [Tue, 1 Feb 2022 05:03:41 +0000 (21:03 -0800)]
Clean up mypy type checker errors.

2 years agoFix a couple of bugs in date parsing.
Scott [Mon, 31 Jan 2022 20:00:11 +0000 (12:00 -0800)]
Fix a couple of bugs in date parsing.

2 years agoAdds a quiet option to executor shutdown.
Scott [Mon, 31 Jan 2022 18:03:43 +0000 (10:03 -0800)]
Adds a quiet option to executor shutdown.

2 years agoTweak histograms in executors to include seconds label.
Scott [Mon, 31 Jan 2022 17:58:30 +0000 (09:58 -0800)]
Tweak histograms in executors to include seconds label.

2 years agoMake histogram auto-format labels.
Scott [Mon, 31 Jan 2022 17:55:30 +0000 (09:55 -0800)]
Make histogram auto-format labels.

2 years agoIf percent > 0, even if it's very small, render some little sliver of
Scott [Mon, 31 Jan 2022 17:13:00 +0000 (09:13 -0800)]
If percent > 0, even if it's very small, render some little sliver of
a bar.

2 years agoUse a label formatter for prettier histograms.
Scott [Mon, 31 Jan 2022 17:08:57 +0000 (09:08 -0800)]
Use a label formatter for prettier histograms.

2 years agoClean up histogram __repr__
Scott [Mon, 31 Jan 2022 17:04:07 +0000 (09:04 -0800)]
Clean up histogram __repr__

2 years agoMake run_tests.sh re-run a subset of the tests w/o coverage
Scott [Mon, 31 Jan 2022 17:02:59 +0000 (09:02 -0800)]
Make run_tests.sh re-run a subset of the tests w/o coverage
instrumentation even in --coverage mode.

2 years ago--unittests_ignore_perf shouldn't mess with the database / file
Scott [Mon, 31 Jan 2022 06:57:29 +0000 (22:57 -0800)]
--unittests_ignore_perf shouldn't mess with the database / file
layer at all.  When the db was down some of tests failed even
though they didn't care about perf.  Let's not do that.

2 years agoFix a recent bug in executors. Thread executor needs to return
Scott [Mon, 31 Jan 2022 06:09:43 +0000 (22:09 -0800)]
Fix a recent bug in executors.  Thread executor needs to return
its future.

2 years agoChange locking boundaries for shared dict. Add a unit test.
Scott [Mon, 31 Jan 2022 05:29:34 +0000 (21:29 -0800)]
Change locking boundaries for shared dict.  Add a unit test.
Make smart_futures re-raise exceptions that happened in futures.
Mess with file_utils.

2 years agoImprove the locking decorators; @synchronized(lock).
Scott [Mon, 31 Jan 2022 01:37:58 +0000 (17:37 -0800)]
Improve the locking decorators; @synchronized(lock).

2 years agoAdds a shutdown_if_idle method to the executors and also eliminates
Scott [Mon, 31 Jan 2022 01:36:47 +0000 (17:36 -0800)]
Adds a shutdown_if_idle method to the executors and also eliminates
a multiple-shutdown problem encountered when using them.

2 years agoAdds MIT license / copyright to string utils from original source
Scott [Mon, 31 Jan 2022 01:34:51 +0000 (17:34 -0800)]
Adds MIT license / copyright to string utils from original source
of code.

2 years agoAdd MIT License and copyright from the shared dict implementation
Scott [Mon, 31 Jan 2022 01:30:11 +0000 (17:30 -0800)]
Add MIT License and copyright from the shared dict implementation
I based this code on.

2 years agoAdd a MP shared dict.
Scott [Mon, 31 Jan 2022 01:25:59 +0000 (17:25 -0800)]
Add a MP shared dict.

2 years agoAdd some new helper methods to file utils.
Scott [Sun, 30 Jan 2022 22:53:33 +0000 (14:53 -0800)]
Add some new helper methods to file utils.

2 years agoOptionally surface exceptions that happen under executors by reading
Scott [Sun, 30 Jan 2022 22:49:52 +0000 (14:49 -0800)]
Optionally surface exceptions that happen under executors by reading
them off the returned futures.

2 years agoDo not adjust task count from a child process or thread; always
Scott [Sun, 30 Jan 2022 22:48:32 +0000 (14:48 -0800)]
Do not adjust task count from a child process or thread; always
do it in the launcher thread/process.

2 years agoGet rid of "fuzzy matches" in favor of a window size param.
Scott [Fri, 28 Jan 2022 19:05:03 +0000 (11:05 -0800)]
Get rid of "fuzzy matches" in favor of a window size param.

2 years agoAdding test code trying to improve test coverage.
Scott [Thu, 27 Jan 2022 21:42:09 +0000 (13:42 -0800)]
Adding test code trying to improve test coverage.

2 years agoStarted using flake8 to lint; removed some of its warnings.
Scott [Thu, 27 Jan 2022 20:09:50 +0000 (12:09 -0800)]
Started using flake8 to lint; removed some of its warnings.

2 years agoAdd --trace_memory to bootstrap.
Scott [Thu, 27 Jan 2022 19:03:36 +0000 (11:03 -0800)]
Add --trace_memory to bootstrap.

2 years agoAdd --run_profiler option.
Scott [Thu, 27 Jan 2022 18:34:30 +0000 (10:34 -0800)]
Add --run_profiler option.

2 years agoMake the worker selection heuristics work harder to avoid backup
Scott [Thu, 27 Jan 2022 17:29:07 +0000 (09:29 -0800)]
Make the worker selection heuristics work harder to avoid backup
bundles scheduled on the same machine as their primary.

2 years agoTesting auto formatting
Scott [Thu, 27 Jan 2022 16:45:48 +0000 (08:45 -0800)]
Testing auto formatting

2 years agoAnother formatter test, whee...
Scott [Thu, 27 Jan 2022 05:50:11 +0000 (21:50 -0800)]
Another formatter test, whee...

2 years agoTrying to get black to run in git's pre-commit hook and actually
Scott [Thu, 27 Jan 2022 05:46:30 +0000 (21:46 -0800)]
Trying to get black to run in git's pre-commit hook and actually
modify the files...

2 years agoAhem. Still running black?
Scott [Thu, 27 Jan 2022 05:35:20 +0000 (21:35 -0800)]
Ahem.  Still running black?

2 years agoRan black code formatter on everything.
Scott [Thu, 27 Jan 2022 05:34:26 +0000 (21:34 -0800)]
Ran black code formatter on everything.

2 years agoRemove missing import.
Scott [Thu, 27 Jan 2022 05:24:56 +0000 (21:24 -0800)]
Remove missing import.

2 years agoReformatting.
Scott [Thu, 27 Jan 2022 05:19:41 +0000 (21:19 -0800)]
Reformatting.

2 years agoFix missing f's.
Scott [Thu, 27 Jan 2022 05:18:33 +0000 (21:18 -0800)]
Fix missing f's.

2 years agoReformatting.
Scott [Wed, 26 Jan 2022 23:36:05 +0000 (15:36 -0800)]
Reformatting.

2 years agoMake sanity check optional.
Scott [Wed, 26 Jan 2022 19:26:55 +0000 (11:26 -0800)]
Make sanity check optional.

2 years agoSanity check list is sorted before binary search.
Scott [Wed, 26 Jan 2022 19:24:46 +0000 (11:24 -0800)]
Sanity check list is sorted before binary search.

2 years agoMinor tweaks.
Scott [Wed, 26 Jan 2022 19:15:14 +0000 (11:15 -0800)]
Minor tweaks.

2 years agoAdd type hints and cleanup.
Scott [Wed, 26 Jan 2022 18:31:53 +0000 (10:31 -0800)]
Add type hints and cleanup.

2 years agoMake unscrambler use a cache per instance instead of module and
Scott [Wed, 26 Jan 2022 17:28:53 +0000 (09:28 -0800)]
Make unscrambler use a cache per instance instead of module and
make it have non-static methods.

2 years agoUnscrambler.py is no longer a utility; see ~bin/unscramble.py.
Scott [Wed, 26 Jan 2022 06:16:24 +0000 (22:16 -0800)]
Unscrambler.py is no longer a utility; see ~bin/unscramble.py.

2 years agoClarify instructions for repopulation.
Scott [Wed, 26 Jan 2022 06:15:08 +0000 (22:15 -0800)]
Clarify instructions for repopulation.

2 years agoAdds unscrambler which contains the guts of a jumble/scrabble player
Scott [Wed, 26 Jan 2022 06:05:25 +0000 (22:05 -0800)]
Adds unscrambler which contains the guts of a jumble/scrabble player
I wrote outside of here.

2 years agoEnable scp compression on remote executor to try to improve data
Scott [Tue, 25 Jan 2022 04:20:16 +0000 (20:20 -0800)]
Enable scp compression on remote executor to try to improve data
xfer times.

2 years agoFix a newly introduced bug in run_tests.sh.
Scott [Tue, 25 Jan 2022 04:19:26 +0000 (20:19 -0800)]
Fix a newly introduced bug in run_tests.sh.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 23:55:24 +0000 (15:55 -0800)]
Improve run_tests.sh

2 years agoReport overall runtime in periodic status dumps.
Scott [Mon, 24 Jan 2022 23:42:09 +0000 (15:42 -0800)]
Report overall runtime in periodic status dumps.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 22:55:09 +0000 (14:55 -0800)]
Improve run_tests.sh

2 years agoMake parallelize remember to shutdown the default executors atexit.
Scott [Mon, 24 Jan 2022 22:51:17 +0000 (14:51 -0800)]
Make parallelize remember to shutdown the default executors atexit.

2 years agoAdd repro instructions for coverage in run_tests.sh.
Scott [Mon, 24 Jan 2022 22:48:49 +0000 (14:48 -0800)]
Add repro instructions for coverage in run_tests.sh.

2 years agoFinish up this coverage stuff.
Scott [Mon, 24 Jan 2022 21:57:51 +0000 (13:57 -0800)]
Finish up this coverage stuff.

2 years agoAdds optional code coverage reporting to run_tests.sh.
Scott [Mon, 24 Jan 2022 21:47:27 +0000 (13:47 -0800)]
Adds optional code coverage reporting to run_tests.sh.

2 years agoThis stuff all still sucks but this is slightly better.
Scott [Mon, 24 Jan 2022 16:47:00 +0000 (08:47 -0800)]
This stuff all still sucks but this is slightly better.

2 years agoAdd my own splitter.
Scott [Mon, 24 Jan 2022 16:25:33 +0000 (08:25 -0800)]
Add my own splitter.

2 years agoMake profanity filter catch foo/bar where foo and/or bar are bad
Scott [Mon, 24 Jan 2022 00:13:44 +0000 (16:13 -0800)]
Make profanity filter catch foo/bar where foo and/or bar are bad
words.

2 years agoMinor cleanup.
Scott [Sun, 23 Jan 2022 20:58:13 +0000 (12:58 -0800)]
Minor cleanup.

2 years agoClean up the remote executor stuff and create a dedicated heartbeat
Scott [Sun, 23 Jan 2022 20:43:46 +0000 (12:43 -0800)]
Clean up the remote executor stuff and create a dedicated heartbeat
thread.  Fix the parallelize integration test.  Other changes that
depended on these fixes.

2 years agoAdded some helpers to file_utils and improved the docs/doctests.
Scott [Sun, 23 Jan 2022 18:08:17 +0000 (10:08 -0800)]
Added some helpers to file_utils and improved the docs/doctests.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:30:42 +0000 (17:30 -0800)]
Reformatting.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:29:09 +0000 (17:29 -0800)]
Reformatting.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:27:17 +0000 (17:27 -0800)]
Reformatting.

2 years agoWatch for grandparent ssh termination by default.
Scott [Sun, 23 Jan 2022 00:57:12 +0000 (16:57 -0800)]
Watch for grandparent ssh termination by default.

2 years agoTrying again to make sure all output is seen here.
Scott [Fri, 21 Jan 2022 19:40:56 +0000 (11:40 -0800)]
Trying again to make sure all output is seen here.

2 years agoRemove spurious print message.
Scott [Fri, 21 Jan 2022 19:37:20 +0000 (11:37 -0800)]
Remove spurious print message.

2 years agoDon't be tricked by speaker groups living on the same IP address
Scott [Fri, 21 Jan 2022 18:20:02 +0000 (10:20 -0800)]
Don't be tricked by speaker groups living on the same IP address
as a chromecast; we want the actual device.

2 years agoWait for both streams to end.
Scott [Fri, 21 Jan 2022 18:19:34 +0000 (10:19 -0800)]
Wait for both streams to end.

2 years agoStop using rsync in executors; this was a hack to work around some
Scott [Thu, 20 Jan 2022 23:18:34 +0000 (15:18 -0800)]
Stop using rsync in executors; this was a hack to work around some
terrible "linux" box that was running linux in emulation under windows
and, for some strange reason, didn't like scp.

2 years agoCleanup docs, add a timeout, in exec_utils.py.
Scott [Thu, 20 Jan 2022 23:17:50 +0000 (15:17 -0800)]
Cleanup docs, add a timeout, in exec_utils.py.

2 years agoMake the new cmd_showing_output select and display data from stderr in
Scott [Thu, 20 Jan 2022 22:45:34 +0000 (14:45 -0800)]
Make the new cmd_showing_output select and display data from stderr in
addition to stdout.

2 years agoMake cmd_showing_output work with subprocesses that use '\r'.
Scott [Wed, 19 Jan 2022 22:10:35 +0000 (14:10 -0800)]
Make cmd_showing_output work with subprocesses that use '\r'.

2 years agoDon't let chromecasts steal the thread for too long.
Scott [Tue, 18 Jan 2022 23:18:21 +0000 (15:18 -0800)]
Don't let chromecasts steal the thread for too long.

2 years agoFix state determination in tplink kasa lights.
Scott [Tue, 18 Jan 2022 21:41:12 +0000 (13:41 -0800)]
Fix state determination in tplink kasa lights.

2 years agoFix chromecasts to work with new version of pychromecast; rename
Scott [Tue, 18 Jan 2022 21:13:21 +0000 (13:13 -0800)]
Fix chromecasts to work with new version of pychromecast; rename
some cameras, add debugging log messages.

2 years agoCode cleanup.
Scott [Tue, 18 Jan 2022 04:40:49 +0000 (20:40 -0800)]
Code cleanup.

2 years agoCreates a function_utils and pull a function_identifer method into
Scott [Tue, 18 Jan 2022 04:30:08 +0000 (20:30 -0800)]
Creates a function_utils and pull a function_identifer method into
it.

2 years agoMake unittest_utils log its perf data to a database, optionally.
Scott [Mon, 17 Jan 2022 23:54:14 +0000 (15:54 -0800)]
Make unittest_utils log its perf data to a database, optionally.

2 years agoArgparse choices wants a list, not a tuple.
Scott [Mon, 17 Jan 2022 23:52:24 +0000 (15:52 -0800)]
Argparse choices wants a list, not a tuple.

2 years agoMake logging_utils remove preexisting logging turds, fix a small bug.
Scott [Mon, 17 Jan 2022 23:51:44 +0000 (15:51 -0800)]
Make logging_utils remove preexisting logging turds, fix a small bug.

2 years ago"Fix" a flaky test and cleanup an unused import.
Scott [Thu, 13 Jan 2022 04:59:00 +0000 (20:59 -0800)]
"Fix" a flaky test and cleanup an unused import.

2 years agoAdds doctests.
Scott [Thu, 13 Jan 2022 04:58:40 +0000 (20:58 -0800)]
Adds doctests.

2 years agoAdds doctests.
Scott [Thu, 13 Jan 2022 04:57:11 +0000 (20:57 -0800)]
Adds doctests.