Fix silent crawler failures, dropped regex matches, and stale-branch detection - #405
Fix silent crawler failures, dropped regex matches, and stale-branch detection#405johnnyc20 wants to merge 2 commits into
Conversation
…detection - Pass base_url through to parse_links() when building the link tree so relative hrefs resolve instead of being silently dropped; update the integration test that had encoded the old dropped-links behavior as expected. - Fix get_intel/get_bitcoin_address in info.py running regex against the raw httpx.Response instead of response.text (always raised TypeError, swallowed by a bare except). - Fix updater.py: strip trailing newline from `git rev-parse` output before comparing to "master" (comparison never matched), and fix a typo'd fallback remote URL (TorBoT.git -> TorBot.git). - Remove deprecated `toml` dependency, now unused. (This branch originally also restructured the CLI into src/torbot/main.py and fixed the httpx.Client(proxies=...) argument rename, to fix a ModuleNotFoundError crash on install. Both are dropped from this PR: dev's own CLI restructuring into src/torbot/cli.py already fixes the same install/CLI crash via a different module layout, and already includes the corrected httpx Client argument.) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
f3c9798 to
2ae88a1
Compare
KingAkeem
left a comment
There was a problem hiding this comment.
Thanks for tightening this up. I think this needs one more pass before merge.
Two behavior fixes still need direct regression coverage: get_intel() and get_bitcoin_address() should be tested against an httpx.Response-like object/body so we know the swallowed TypeError does not come back. updater.check_version() also needs a focused test for the rev-parse newline case so the master branch path is covered.
I also noticed toml is removed from pyproject.toml and requirements.txt, but src/torbot/cli.py still imports toml and uses toml.load() in the source-checkout version fallback. In a clean install where toml is no longer present, importing the CLI can fail before the command starts. Either keep the dependency or switch this fallback to tomllib/tomli.
One functional issue remains in get_bitcoin_address(): switching to response.text fixes the type error, but the regex is still anchored with ^...$, so it only matches when the entire response body is exactly one BTC address. It will still miss addresses embedded in HTML or normal page text.
KingAkeem
left a comment
There was a problem hiding this comment.
The TOML fallback is fixed now, and the new version test covers that path. There are still three blockers before this should merge:
get_bitcoin_address()still applies^...$to the entireresponse.text, so it only finds an address when the whole response body is exactly that address. Addresses embedded in HTML or normal page text are still missed.get_intel()now readsresponse.text, but the matches are only added to a local set and then discarded. Nothing is returned, displayed, or persisted, so the function still has no observable result.- There are no direct regression tests for
get_intel(),get_bitcoin_address(), or the newline/branch behavior inupdater.check_version(). The added tests cover the version fallback and link-tree change, but not these behavior fixes.
Please fix those paths and add focused tests before merging. GitHub Actions also has not run on this head because the workflow is still awaiting approval.
Summary
base_urlthrough toparse_links()when building the link tree so relative hrefs resolve instead of being silently dropped; updated the integration test that had encoded the old dropped-links behavior as expected.get_intel/get_bitcoin_addressininfo.pyrunning regex against the rawhttpx.Responseinstead ofresponse.text(always raisedTypeError, swallowed by a bare except).updater.py: strip trailing newline fromgit rev-parseoutput before comparing to"master"(comparison never matched), and fix a typo'd fallback remote URL (TorBoT.git->TorBot.git).tomldependency, now unused.Note: this branch originally also restructured the CLI into
src/torbot/main.pyand fixed thehttpx.Client(proxies=...)argument rename, to fix aModuleNotFoundErrorcrash on install. Both are dropped from this PR —dev's own CLI restructuring intosrc/torbot/cli.pyalready fixes the same install/CLI crash via a different module layout, and already includes the correctedhttpx.Clientargument.Test plan
tests/test_linktree_tree.pyupdated for correct relative-href resolution