python_utils.git
2 years agoOptionally surface exceptions that happen under executors by reading
Scott [Sun, 30 Jan 2022 22:49:52 +0000 (14:49 -0800)]
Optionally surface exceptions that happen under executors by reading
them off the returned futures.

2 years agoDo not adjust task count from a child process or thread; always
Scott [Sun, 30 Jan 2022 22:48:32 +0000 (14:48 -0800)]
Do not adjust task count from a child process or thread; always
do it in the launcher thread/process.

2 years agoGet rid of "fuzzy matches" in favor of a window size param.
Scott [Fri, 28 Jan 2022 19:05:03 +0000 (11:05 -0800)]
Get rid of "fuzzy matches" in favor of a window size param.

2 years agoAdding test code trying to improve test coverage.
Scott [Thu, 27 Jan 2022 21:42:09 +0000 (13:42 -0800)]
Adding test code trying to improve test coverage.

2 years agoStarted using flake8 to lint; removed some of its warnings.
Scott [Thu, 27 Jan 2022 20:09:50 +0000 (12:09 -0800)]
Started using flake8 to lint; removed some of its warnings.

2 years agoAdd --trace_memory to bootstrap.
Scott [Thu, 27 Jan 2022 19:03:36 +0000 (11:03 -0800)]
Add --trace_memory to bootstrap.

2 years agoAdd --run_profiler option.
Scott [Thu, 27 Jan 2022 18:34:30 +0000 (10:34 -0800)]
Add --run_profiler option.

2 years agoMake the worker selection heuristics work harder to avoid backup
Scott [Thu, 27 Jan 2022 17:29:07 +0000 (09:29 -0800)]
Make the worker selection heuristics work harder to avoid backup
bundles scheduled on the same machine as their primary.

2 years agoTesting auto formatting
Scott [Thu, 27 Jan 2022 16:45:48 +0000 (08:45 -0800)]
Testing auto formatting

2 years agoAnother formatter test, whee...
Scott [Thu, 27 Jan 2022 05:50:11 +0000 (21:50 -0800)]
Another formatter test, whee...

2 years agoTrying to get black to run in git's pre-commit hook and actually
Scott [Thu, 27 Jan 2022 05:46:30 +0000 (21:46 -0800)]
Trying to get black to run in git's pre-commit hook and actually
modify the files...

2 years agoAhem. Still running black?
Scott [Thu, 27 Jan 2022 05:35:20 +0000 (21:35 -0800)]
Ahem.  Still running black?

2 years agoRan black code formatter on everything.
Scott [Thu, 27 Jan 2022 05:34:26 +0000 (21:34 -0800)]
Ran black code formatter on everything.

2 years agoRemove missing import.
Scott [Thu, 27 Jan 2022 05:24:56 +0000 (21:24 -0800)]
Remove missing import.

2 years agoReformatting.
Scott [Thu, 27 Jan 2022 05:19:41 +0000 (21:19 -0800)]
Reformatting.

2 years agoFix missing f's.
Scott [Thu, 27 Jan 2022 05:18:33 +0000 (21:18 -0800)]
Fix missing f's.

2 years agoReformatting.
Scott [Wed, 26 Jan 2022 23:36:05 +0000 (15:36 -0800)]
Reformatting.

2 years agoMake sanity check optional.
Scott [Wed, 26 Jan 2022 19:26:55 +0000 (11:26 -0800)]
Make sanity check optional.

2 years agoSanity check list is sorted before binary search.
Scott [Wed, 26 Jan 2022 19:24:46 +0000 (11:24 -0800)]
Sanity check list is sorted before binary search.

2 years agoMinor tweaks.
Scott [Wed, 26 Jan 2022 19:15:14 +0000 (11:15 -0800)]
Minor tweaks.

2 years agoAdd type hints and cleanup.
Scott [Wed, 26 Jan 2022 18:31:53 +0000 (10:31 -0800)]
Add type hints and cleanup.

2 years agoMake unscrambler use a cache per instance instead of module and
Scott [Wed, 26 Jan 2022 17:28:53 +0000 (09:28 -0800)]
Make unscrambler use a cache per instance instead of module and
make it have non-static methods.

2 years agoUnscrambler.py is no longer a utility; see ~bin/unscramble.py.
Scott [Wed, 26 Jan 2022 06:16:24 +0000 (22:16 -0800)]
Unscrambler.py is no longer a utility; see ~bin/unscramble.py.

2 years agoClarify instructions for repopulation.
Scott [Wed, 26 Jan 2022 06:15:08 +0000 (22:15 -0800)]
Clarify instructions for repopulation.

2 years agoAdds unscrambler which contains the guts of a jumble/scrabble player
Scott [Wed, 26 Jan 2022 06:05:25 +0000 (22:05 -0800)]
Adds unscrambler which contains the guts of a jumble/scrabble player
I wrote outside of here.

2 years agoEnable scp compression on remote executor to try to improve data
Scott [Tue, 25 Jan 2022 04:20:16 +0000 (20:20 -0800)]
Enable scp compression on remote executor to try to improve data
xfer times.

2 years agoFix a newly introduced bug in run_tests.sh.
Scott [Tue, 25 Jan 2022 04:19:26 +0000 (20:19 -0800)]
Fix a newly introduced bug in run_tests.sh.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 23:55:24 +0000 (15:55 -0800)]
Improve run_tests.sh

2 years agoReport overall runtime in periodic status dumps.
Scott [Mon, 24 Jan 2022 23:42:09 +0000 (15:42 -0800)]
Report overall runtime in periodic status dumps.

2 years agoImprove run_tests.sh
Scott [Mon, 24 Jan 2022 22:55:09 +0000 (14:55 -0800)]
Improve run_tests.sh

2 years agoMake parallelize remember to shutdown the default executors atexit.
Scott [Mon, 24 Jan 2022 22:51:17 +0000 (14:51 -0800)]
Make parallelize remember to shutdown the default executors atexit.

2 years agoAdd repro instructions for coverage in run_tests.sh.
Scott [Mon, 24 Jan 2022 22:48:49 +0000 (14:48 -0800)]
Add repro instructions for coverage in run_tests.sh.

2 years agoFinish up this coverage stuff.
Scott [Mon, 24 Jan 2022 21:57:51 +0000 (13:57 -0800)]
Finish up this coverage stuff.

2 years agoAdds optional code coverage reporting to run_tests.sh.
Scott [Mon, 24 Jan 2022 21:47:27 +0000 (13:47 -0800)]
Adds optional code coverage reporting to run_tests.sh.

2 years agoThis stuff all still sucks but this is slightly better.
Scott [Mon, 24 Jan 2022 16:47:00 +0000 (08:47 -0800)]
This stuff all still sucks but this is slightly better.

2 years agoAdd my own splitter.
Scott [Mon, 24 Jan 2022 16:25:33 +0000 (08:25 -0800)]
Add my own splitter.

2 years agoMake profanity filter catch foo/bar where foo and/or bar are bad
Scott [Mon, 24 Jan 2022 00:13:44 +0000 (16:13 -0800)]
Make profanity filter catch foo/bar where foo and/or bar are bad
words.

2 years agoMinor cleanup.
Scott [Sun, 23 Jan 2022 20:58:13 +0000 (12:58 -0800)]
Minor cleanup.

2 years agoClean up the remote executor stuff and create a dedicated heartbeat
Scott [Sun, 23 Jan 2022 20:43:46 +0000 (12:43 -0800)]
Clean up the remote executor stuff and create a dedicated heartbeat
thread.  Fix the parallelize integration test.  Other changes that
depended on these fixes.

2 years agoAdded some helpers to file_utils and improved the docs/doctests.
Scott [Sun, 23 Jan 2022 18:08:17 +0000 (10:08 -0800)]
Added some helpers to file_utils and improved the docs/doctests.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:30:42 +0000 (17:30 -0800)]
Reformatting.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:29:09 +0000 (17:29 -0800)]
Reformatting.

2 years agoReformatting.
Scott [Sun, 23 Jan 2022 01:27:17 +0000 (17:27 -0800)]
Reformatting.

2 years agoWatch for grandparent ssh termination by default.
Scott [Sun, 23 Jan 2022 00:57:12 +0000 (16:57 -0800)]
Watch for grandparent ssh termination by default.

2 years agoTrying again to make sure all output is seen here.
Scott [Fri, 21 Jan 2022 19:40:56 +0000 (11:40 -0800)]
Trying again to make sure all output is seen here.

2 years agoRemove spurious print message.
Scott [Fri, 21 Jan 2022 19:37:20 +0000 (11:37 -0800)]
Remove spurious print message.

2 years agoDon't be tricked by speaker groups living on the same IP address
Scott [Fri, 21 Jan 2022 18:20:02 +0000 (10:20 -0800)]
Don't be tricked by speaker groups living on the same IP address
as a chromecast; we want the actual device.

2 years agoWait for both streams to end.
Scott [Fri, 21 Jan 2022 18:19:34 +0000 (10:19 -0800)]
Wait for both streams to end.

2 years agoStop using rsync in executors; this was a hack to work around some
Scott [Thu, 20 Jan 2022 23:18:34 +0000 (15:18 -0800)]
Stop using rsync in executors; this was a hack to work around some
terrible "linux" box that was running linux in emulation under windows
and, for some strange reason, didn't like scp.

2 years agoCleanup docs, add a timeout, in exec_utils.py.
Scott [Thu, 20 Jan 2022 23:17:50 +0000 (15:17 -0800)]
Cleanup docs, add a timeout, in exec_utils.py.

2 years agoMake the new cmd_showing_output select and display data from stderr in
Scott [Thu, 20 Jan 2022 22:45:34 +0000 (14:45 -0800)]
Make the new cmd_showing_output select and display data from stderr in
addition to stdout.

2 years agoMake cmd_showing_output work with subprocesses that use '\r'.
Scott [Wed, 19 Jan 2022 22:10:35 +0000 (14:10 -0800)]
Make cmd_showing_output work with subprocesses that use '\r'.

2 years agoDon't let chromecasts steal the thread for too long.
Scott [Tue, 18 Jan 2022 23:18:21 +0000 (15:18 -0800)]
Don't let chromecasts steal the thread for too long.

2 years agoFix state determination in tplink kasa lights.
Scott [Tue, 18 Jan 2022 21:41:12 +0000 (13:41 -0800)]
Fix state determination in tplink kasa lights.

2 years agoFix chromecasts to work with new version of pychromecast; rename
Scott [Tue, 18 Jan 2022 21:13:21 +0000 (13:13 -0800)]
Fix chromecasts to work with new version of pychromecast; rename
some cameras, add debugging log messages.

2 years agoCode cleanup.
Scott [Tue, 18 Jan 2022 04:40:49 +0000 (20:40 -0800)]
Code cleanup.

2 years agoCreates a function_utils and pull a function_identifer method into
Scott [Tue, 18 Jan 2022 04:30:08 +0000 (20:30 -0800)]
Creates a function_utils and pull a function_identifer method into
it.

2 years agoMake unittest_utils log its perf data to a database, optionally.
Scott [Mon, 17 Jan 2022 23:54:14 +0000 (15:54 -0800)]
Make unittest_utils log its perf data to a database, optionally.

2 years agoArgparse choices wants a list, not a tuple.
Scott [Mon, 17 Jan 2022 23:52:24 +0000 (15:52 -0800)]
Argparse choices wants a list, not a tuple.

2 years agoMake logging_utils remove preexisting logging turds, fix a small bug.
Scott [Mon, 17 Jan 2022 23:51:44 +0000 (15:51 -0800)]
Make logging_utils remove preexisting logging turds, fix a small bug.

2 years ago"Fix" a flaky test and cleanup an unused import.
Scott [Thu, 13 Jan 2022 04:59:00 +0000 (20:59 -0800)]
"Fix" a flaky test and cleanup an unused import.

2 years agoAdds doctests.
Scott [Thu, 13 Jan 2022 04:58:40 +0000 (20:58 -0800)]
Adds doctests.

2 years agoAdds doctests.
Scott [Thu, 13 Jan 2022 04:57:11 +0000 (20:57 -0800)]
Adds doctests.

2 years agoUpdate comments.
Scott [Thu, 13 Jan 2022 04:56:31 +0000 (20:56 -0800)]
Update comments.

2 years agoAdd doctests, general cleanup.
Scott [Thu, 13 Jan 2022 00:19:52 +0000 (16:19 -0800)]
Add doctests, general cleanup.

2 years agoScale back warnings.warn and add stacklevels= where appropriate.
Scott [Wed, 12 Jan 2022 17:06:29 +0000 (09:06 -0800)]
Scale back warnings.warn and add stacklevels= where appropriate.

2 years agoFix a couple of bugs in executors. Use run_tests.sh as a pre commit
Scott [Tue, 11 Jan 2022 21:41:25 +0000 (13:41 -0800)]
Fix a couple of bugs in executors.  Use run_tests.sh as a pre commit
hook.

2 years agoStart using warnings from stdlib.
Scott [Tue, 11 Jan 2022 21:33:24 +0000 (13:33 -0800)]
Start using warnings from stdlib.

2 years agoFix requirements.txt
Scott [Tue, 11 Jan 2022 05:51:39 +0000 (21:51 -0800)]
Fix requirements.txt

2 years agoAdd multi replace.
Scott [Tue, 11 Jan 2022 05:51:30 +0000 (21:51 -0800)]
Add multi replace.

2 years agoFix a missing import bug.
Scott [Tue, 11 Jan 2022 02:34:19 +0000 (18:34 -0800)]
Fix a missing import bug.

2 years agoMake it work.
Scott [Tue, 11 Jan 2022 01:52:32 +0000 (17:52 -0800)]
Make it work.

2 years agoAdd requirements.txt
Scott [Tue, 11 Jan 2022 01:49:58 +0000 (17:49 -0800)]
Add requirements.txt

2 years agoAdd another doctest.
Scott [Tue, 11 Jan 2022 01:22:08 +0000 (17:22 -0800)]
Add another doctest.

2 years agoAdds some doctests to decorators.
Scott [Mon, 10 Jan 2022 23:31:11 +0000 (15:31 -0800)]
Adds some doctests to decorators.

2 years agoMake rate_limited work and add doctest.
Scott [Mon, 10 Jan 2022 22:57:17 +0000 (14:57 -0800)]
Make rate_limited work and add doctest.

2 years agoUpdate MAC address.
Scott [Mon, 10 Jan 2022 22:56:55 +0000 (14:56 -0800)]
Update MAC address.

2 years agoExperiment with audit events in bootstrap.
Scott [Mon, 10 Jan 2022 17:45:31 +0000 (09:45 -0800)]
Experiment with audit events in bootstrap.

2 years agoMake rate_limited use cvs.
Scott [Mon, 10 Jan 2022 06:15:45 +0000 (22:15 -0800)]
Make rate_limited use cvs.

2 years agoAdd rate limiter decorator.
Scott [Sat, 8 Jan 2022 23:21:25 +0000 (15:21 -0800)]
Add rate limiter decorator.

2 years agoCleanup logging module.
Scott [Sat, 8 Jan 2022 21:01:47 +0000 (13:01 -0800)]
Cleanup logging module.

2 years agoMake logging optionally remove global handlers added by (shitty) pip
Scott [Fri, 7 Jan 2022 19:14:29 +0000 (11:14 -0800)]
Make logging optionally remove global handlers added by (shitty) pip
modules.  Make config optionally halt on unrecognized arguments.
Make profanity filter smarter.

2 years agoA bunch of changes...
Scott [Thu, 6 Jan 2022 20:13:34 +0000 (12:13 -0800)]
A bunch of changes...

2 years agoMove stuff around.
Scott [Mon, 6 Dec 2021 23:43:17 +0000 (15:43 -0800)]
Move stuff around.

2 years agoMake PresenceDetection self-update periodically.
Scott [Mon, 6 Dec 2021 05:43:24 +0000 (21:43 -0800)]
Make PresenceDetection self-update periodically.

2 years agoMake presence detection work from cabin or house and deal with a
Scott [Mon, 6 Dec 2021 04:50:59 +0000 (20:50 -0800)]
Make presence detection work from cabin or house and deal with a
bad VPN link between them better.

2 years agoVarious changes.
Scott [Sun, 5 Dec 2021 05:23:11 +0000 (21:23 -0800)]
Various changes.

2 years agoMake smart futures avoid polling.
Scott Gasch [Fri, 19 Nov 2021 04:39:12 +0000 (20:39 -0800)]
Make smart futures avoid polling.
Much fucking around with the backup strategy in executors.
Tweaks to presence.

2 years agoWaitableState thingy and some stuff in the ML world.
Scott Gasch [Thu, 18 Nov 2021 06:14:37 +0000 (22:14 -0800)]
WaitableState thingy and some stuff in the ML world.

2 years agoBugfixes in executors.
Scott Gasch [Wed, 17 Nov 2021 23:26:43 +0000 (15:26 -0800)]
Bugfixes in executors.

2 years agoMake remote workers die if no longer needed; cleanups in executors.
Scott Gasch [Wed, 17 Nov 2021 22:40:34 +0000 (14:40 -0800)]
Make remote workers die if no longer needed; cleanups in executors.

2 years agoMaking remote training work better.
Scott Gasch [Tue, 16 Nov 2021 00:55:26 +0000 (16:55 -0800)]
Making remote training work better.

2 years agoTighten up the remote executor.
Scott Gasch [Mon, 15 Nov 2021 06:48:04 +0000 (22:48 -0800)]
Tighten up the remote executor.

2 years agoSmall changes to decorators.
Scott Gasch [Sun, 14 Nov 2021 23:46:39 +0000 (15:46 -0800)]
Small changes to decorators.

2 years agoSmall bugfixes; also, add a new machine to the remote executor pool.
Scott Gasch [Sat, 13 Nov 2021 15:06:31 +0000 (07:06 -0800)]
Small bugfixes; also, add a new machine to the remote executor pool.

2 years agoAdd device_utils.py
Scott Gasch [Tue, 2 Nov 2021 03:05:34 +0000 (20:05 -0700)]
Add device_utils.py

2 years agoOverrides + debugging modules / functions in logging.
Scott Gasch [Mon, 1 Nov 2021 21:10:02 +0000 (14:10 -0700)]
Overrides + debugging modules / functions in logging.

2 years agoUgh, a bunch of things. @overrides. --lmodule. Chromecasts. etc...
Scott Gasch [Sun, 31 Oct 2021 20:08:51 +0000 (13:08 -0700)]
Ugh, a bunch of things.  @overrides.  --lmodule.  Chromecasts.  etc...

2 years agoFix Arper, make IP address a property of device
Scott Gasch [Fri, 29 Oct 2021 18:26:45 +0000 (11:26 -0700)]
Fix Arper, make IP address a property of device

2 years agoAdds Arper, cleans up other stuff.
Scott Gasch [Fri, 29 Oct 2021 17:33:07 +0000 (10:33 -0700)]
Adds Arper, cleans up other stuff.