Runs pytest tests/ on every push and PR. Ubuntu runner with
Python 3.11, libmpv, and QT_QPA_PLATFORM=offscreen for headless
Qt. Badge in README links to the Actions tab.
117 tests, ~0.2s locally. CI time depends on PySide6 install
(~2 min) + apt deps (~30s) + tests (~1s).
Remove tests/ from .gitignore and track the existing test suite:
tests/core/test_db.py — DB schema, migration, CRUD
tests/core/test_cache.py — cache helpers
tests/core/test_config.py — config/path helpers
tests/core/test_concurrency.py — app loop accessor
tests/core/api/test_base.py — Post dataclass, BooruClient
tests/gui/popout/test_state.py — 57 state machine tests
All pure Python, no secrets, no external deps. Uses temp DBs and
synthetic data. Run with: pytest tests/
Library thumbnails are saved by post_id (_copy_library_thumb uses
f"{post.id}.jpg") but the library viewer looked them up by file
stem (f"{filepath.stem}.jpg"). For digit-stem files (12345.jpg)
these are the same. For templated files (artist_12345.jpg) the
stem is "artist_12345" which doesn't match the thumbnail named
"12345.jpg" — wrong or missing thumbnails.
Fix: resolve post_id from the filename via
get_library_post_id_by_filename, then look up the thumbnail as
f"{post_id}.jpg". Generated thumbnails (for files without a
cached browse thumbnail) also store by post_id now, so
everything stays consistent.
The tag search filter in refresh() used f.stem.isdigit() to
extract post_id — templated filenames like artist_12345.jpg
failed the check and got filtered out even when their post_id
matched the search query.
Fix: look up post_id via db.get_library_post_id_by_filename
first (handles templated filenames), fall back to int(stem) for
legacy digit-stem files. Same pattern as the delete and saved-dot
fixes from earlier in this refactor.
Logs when GL render context is actually initialized (not on the no-op
path). Confirms GL init fires once per widget lifetime, not on every
video click. Kept permanently for future debugging.
behavior change: when streaming=True (uncached video handed directly to
mpv), _load now early-returns instead of running download_image in
parallel. mpv's stream-record option (added in the previous commit)
handles cache population, so the parallel httpx download was a second
TCP+TLS connection to the same CDN URL contending with mpv for
bandwidth. Single connection per uncached video after this commit.
Replaces the parallel httpx download with mpv's stream-record per-file
option. When play_file receives an HTTP URL, it passes stream_record
pointing at a .part temp file alongside the URL. mpv writes the incoming
network stream to disk as it decodes, so a single HTTP connection serves
both playback and cache population.
On clean EOF the .part is promoted to the real cache path via os.replace.
Seeks invalidate the recording (mpv may skip byte ranges), so
_seeked_during_record flags it for discard. stop() and rapid-click
cleanup also discard incomplete .part files.
At this commit both pipelines are active — _load still runs the httpx
download in parallel. Whichever finishes second wins os.replace. The
next commit removes the httpx path.
Prevents cache eviction from deleting a .part temp file that mpv's
stream-record is actively writing to. Prerequisite for the stream-record
plumbing in video_player.py.
behavior change: mpv now uses explicit cache=yes, cache_pause=no
(stutter over pause for short clips), 50MiB demuxer buffer cap,
20s read-ahead, and 10s network timeout (down from ~60s default).
Improves first-frame latency on uncached video streams and surfaces
stalled-connection errors faster.
behavior change: _apply_load_video now switches the stack to the video
surface BEFORE calling play_file so mpv's first frame lands on a visible
widget instead of a cleared image viewer. Removes the redundant stop()
call — loadfile("replace") atomically replaces the current file.
Also fixes video position not surviving popout close: StopMedia (part of
CloseRequested effects) destroyed mpv's time_pos before get_video_state
could read it. Now closeEvent snapshots position_ms before dispatching
CloseRequested, and get_video_state returns the snapshot.
behavior change: _on_video_stream no longer calls stop() on the
embedded preview's mpv when the popout is the visible target. The
embedded preview is hidden and idle — the synchronous command('stop')
round-trip was wasting ~50-100ms on the click-to-first-frame critical
path with no visible benefit. loadfile("replace") in the popout's
play_file handles the media swap atomically.
Library items' Post objects were constructed without width/height
(library_meta doesn't store them), so the popout got 0/0 and
_fit_to_content returned early without setting keep_aspect_ratio.
Videos were unaffected because mpv reports dimensions later via
VideoSizeKnown. Images had no second chance — the aspect lock
was never set, and manual window resizing stretched them freely.
Fix: new _image_dimensions(path) reads the actual pixel size from
the file via QPixmap before constructing the Post. The Post now
carries real width/height. _update_fullscreen moved to run AFTER
Post construction so cp.width/cp.height are populated when the
popout reads them for pre-fit + aspect lock.
Not a regression from the templates refactor — pre-existing gap
in the library display path.
The flag was set in _ensure_post_categories_async which runs AFTER
_on_post_selected calls info_panel.set_post. By the time the flag
was True, the flat tags had already rendered. The flash persisted.
Fix: check whether a fetch is needed and set the flag in
_on_post_selected, right before set_post. The info panel sees the
flag and skips the flat-tag fallback on its first render.
When a category fetch is about to fire (Rule34/Safebooru.org/
Moebooru on first click), the info panel was rendering the full
flat tag list, then ~200ms later re-rendering with categorized
tags. The re-layout from flat→categorized looked like a visual
hitch.
Fix: new _categories_pending flag on InfoPanel. When True, the
flat-tag fallback branch is skipped — the tags area stays empty
until categories arrive and render in one clean pass.
_ensure_post_categories_async sets _categories_pending = True
before scheduling the fetch (or False if no fetcher = Danbooru)
_on_categories_updated clears _categories_pending = False
Visual result:
Danbooru/e621: instant (inline, no flag)
Gelbooru with auth: instant (background prefetch beat the click)
Rule34/SB.org/Moebooru: empty ~200ms → categories appear cleanly
(no flat→categorized re-layout)
When _batch_api_works is True (Gelbooru proper with auth, persisted
from a prior session's probe), search() fires prefetch_batch in the
background. The batch tag API covers the entire page's tags in 1-2
requests during the time between grid render and user click — the
cache is warm before the info panel opens, so categories appear
instantly with no flash of flat tags.
Gated on _batch_api_works is True (not None, not False):
- Gelbooru proper: prefetches (batch API known good)
- Rule34: skips (batch_api_works = False, persisted)
- Safebooru.org: skips (no auth → fetcher skips batch capability)
Rule34 / Safebooru.org / Moebooru stay on-demand: the ~200ms
per-click HTML scrape is unavoidable for those sites because their
only path is per-post page fetching, which can't be batched.
The Library tab's single-delete and multi-delete context menu
actions called .unlink() directly, bypassing delete_from_library
entirely. They only extracted post_id from digit-stem filenames
(int(stem) if stem.isdigit()), so templated files like
artist_12345.jpg got deleted from disk but left orphan
library_meta rows that made get_saved_post_ids lie forever.
Fix: resolve post_id via db.get_library_post_id_by_filename first
(handles templated filenames), fall back to int(stem) for legacy
digit-stem files, then call db.remove_library_meta(post_id) after
unlinking. Both single-delete and multi-delete paths are fixed.
This was the last source of orphan library_meta rows. With this
fix + the earlier delete_from_library cleanup, every deletion
path in the app now cleans up its meta row:
- Library tab single delete (this commit)
- Library tab multi delete (this commit)
- Browse/preview "Unsave from Library" (via delete_from_library)
- Browse multi-select "Unsave All" (via delete_from_library)
- Bookmarks "Unsave from Library" (via delete_from_library)
- Bookmarks multi-select "Unsave All" (via delete_from_library)
The primary search result handler (_on_search_done) was still using
the old filesystem walk + stem.isdigit() filter to build the saved-
post-id set. The two other call sites (_on_load_more and the
blacklist rebuild) were fixed in the earlier saved-dot sweep but
this one was missed. Templated filenames like artist_12345.jpg
were invisible, so the saved-dot disappeared after any grid
rebuild (new search, page change, etc).
Fix: use self._db.get_saved_post_ids() (one indexed SELECT,
format-agnostic) like the other two sites already do. Also drops
the saved_dir import that was only needed for the filesystem walk.
_do_ensure only tried the batch API when _batch_api_works was True,
but after removing the search-time prefetch (where the probe used
to run), _batch_api_works stayed None forever. Gelbooru's only
viable path IS the batch API (its post-view HTML has no tag links),
so clicks on Gelbooru posts produced zero categories.
Fix: _do_ensure now tries the batch API when _batch_api_works is
not False (i.e., both True and None). When None, the call doubles
as an inline probe: if the batch produced categories, save True;
if nothing useful came back, save False and fall to HTML.
This is simpler than the old prefetch_batch probe because it runs
on ONE post at a time — no batch/HTML mixing concerns, no "single
path per invocation" rule. The probe result is persisted to DB so
it only fires once per site ever.
Dispatch matrix in _do_ensure:
_batch_api_works True + auth → batch API (Gelbooru proper)
_batch_api_works None + auth → batch as probe → True or False
_batch_api_works False → HTML scrape (Rule34)
no auth → HTML scrape (Safebooru.org)
transient error → stays None, retry next click
Verified all three sites from clean cache: Gelbooru 55/56+49/50
(batch), Rule34 40/40+38/38 (HTML), Safebooru.org 47/47+47/47
(HTML).
Removes the asyncio.create_task(prefetch_batch) calls from
search() and get_post() in both clients. Tags are now fetched
ONLY when the user actually clicks a post (via ensure_categories
in the info panel path) or saves with a category-token template.
The background prefetch was the source of most of the complexity:
probe timing, early-exit bugs from partial composes racing with
on-click ensures, Rule34's slow probe blocking the prefetch
window. All gone.
New flow:
search() → fast, returns posts with flat tags only
click → ensure_categories fires, ~200ms HTML scrape or
batch API, categories arrive, signal re-renders
re-click → instant (cache compose, no HTTP)
save → ensure in save_post_file, same path
The ~200ms per first-click is invisible during the image load.
The cache compounds across posts and sessions. The prefetch_batch
method stays in CategoryFetcher for potential future use but
nothing calls it from the hot path anymore.
The probe that detects whether a site's batch tag API works
(Gelbooru proper: yes, Rule34: no) now persists its result in the
tag_types table using a sentinel key (__batch_api_probe__). On
subsequent app launches, the fetcher reads the saved result at
construction time and skips the probe entirely.
Before: every session with Rule34 wasted ~0.6s on a probe request
that always fails (Rule34 returns garbage for names=). During that
time the background prefetch couldn't start HTML scraping, so the
first few post clicks paid ~0.3s each.
After: first ever session probes Rule34 once, stores False. Every
subsequent session reads False from DB, skips the probe, and the
background prefetch immediately starts HTML scraping. By the time
the user clicks any post, the scrape is usually done.
Gelbooru proper: probe succeeds on first session, stores True.
Future sessions use the batch API without probing. No change in
speed (already fast), just saves the probe roundtrip.
Persisted per site_id so different Gelbooru-shaped sites get their
own probe result. The clear_tag_cache method wipes probe results
along with tag data (the sentinel key lives in the same table).
Two places checked `if post.tag_categories: return` before doing
a full cache-coverage check, causing posts with partial cache
composes (e.g. 5/40 tags from the background prefetch) to get
stuck at low coverage forever:
ensure_categories: removed the post.tag_categories early exit.
Now ALWAYS runs try_compose_from_cache first. Only the 100%
coverage return (True) is trusted as "done." Partial composes
return False and fall through to the fetch path.
_ensure_post_categories_async: removed the post.tag_categories
guard. Danbooru/e621 are filtered by the client.category_fetcher
is None check instead (they categorize inline, no fetcher).
For Gelbooru-style sites, always schedules ensure_categories
regardless of current post state.
Root cause: the partial-compose fix (try_compose_from_cache
populates tag_categories even when cache coverage is <100%)
conflicted with the early-exit guards that assumed non-empty
tag_categories = fully categorized. Now the only "fully done"
signal is try_compose_from_cache returning True (100% coverage).
try_compose_from_cache was returning True on ANY partial cache hit
(even 1/38 tags). ensure_categories then saw non-empty
tag_categories and returned immediately, leaving the post stuck at
1/38 coverage. The bug showed on Rule34: post 1 got fully scraped
(40/40), its tags got cached, then post 2's compose found one
matching tag and declared victory.
Fix: try_compose_from_cache now returns True ONLY when 100% of
unique tags have cached labels (no fetch needed). It STILL
populates post.tag_categories with whatever IS cached (for
immediate partial display), but returning False signals
ensure_categories to continue to the fetch path.
This is the correct semantic split:
- populate → always (for display)
- return True → only when complete (for dispatch)
Verified:
Rule34: 40/40 + 38/38 (was 40/40 + 1/38)
Gelbooru: 55/56 + 49/50 (batch API, one rare tag)
Safebooru.org: 47/47 + 47/47 (HTML scrape, full)
Three fixes:
1. HTML parser completely rewritten with two-pass approach:
- Pass 1: regex finds each tag-type element and its full inner
content (up to closing </li|span|td|div>)
- Pass 2: within the content, extracts the tag name from the
tags=NAME URL parameter in the search link
The old single-pass regex captured the ? wiki-link (first <a>)
instead of the tag name (second <a>). The URL-param extraction
works on Rule34 (40 tags), Safebooru.org (47 tags), and
yande.re (3 tags). Gelbooru proper returns 0 (post page only
has ? links with no tags= param) which is correct — Gelbooru
uses the batch tag API instead.
2. prefetch_batch is now truly fire-and-forget:
gelbooru.py and moebooru.py use asyncio.create_task instead of
await for prefetch_batch. search() returns immediately. The
probe + batch/HTML fetch runs in the background. Previously
search() blocked on the probe, which made Rule34 searches take
5+ seconds (slow/broken Rule34 API response time).
3. Partial cache compose already fixed in the previous commit
complements this: posts with 49/50 cached tags now show all
available categories instead of nothing.
try_compose_from_cache previously required 100% cache coverage —
every tag in the post had to have a cached label or it returned
False and populated nothing. One rare uncached tag out of 50
blocked the entire composition, leaving the post with zero
categories even though 49/50 labels were available.
Fix: compose whatever IS cached, return True when at least one
tag got categorized. Tags not in the cache are simply absent from
the categories dict (they stay in the flat tags string). The
return value now means "the post has usable categories" rather
than "the post has complete categories." This distinction matters
because the dispatch logic uses the return value to decide
whether to skip the fetch path — partial coverage is better than
no coverage, and the missing tags get cached eventually when
other posts that contain them get fetched.
Verified against Gelbooru: post with 50 tags where 49 were cached
now gets 49/50 categorized (Artist, Character, Copyright, General,
Meta) instead of 0/50.
Two bookmark save sites updated for save_post_file's sync→async
signature change:
_save_bookmark_to_library: wraps the save in an async closure
and schedules via run_on_app_loop (already imported for the
thumbnail download path). Fire-and-forget; the source file is
already cached so the save is near-instant.
Save As action: same async wrapper pattern. The dialog runs
synchronously (user picks destination), then the actual file
copy is scheduled on the async loop.
Neither site passes a category_fetcher — bookmarks don't have a
direct reference to the active BooruClient. The save flow's
ensure_categories check in library_save.py short-circuits (the
fetcher is None), so template rendering uses whatever categories
are already on the post object. For bookmark→library saves, the
user typically hasn't clicked the post in the browse grid, so
categories may be empty — the template falls back to %id% for
category tokens, same as before. Full categorization on the
bookmark save path is a future enhancement (would require passing
the client through from main_window).
Four save call sites updated to await save_post_file (now async)
and pass category_fetcher so the template-render ensure check can
fire when needed.
_bulk_save: creates fetcher once at the top of the async closure,
shared across all posts in the batch. Probe state persists
within the batch.
_save_to_library: creates fetcher per invocation (single post).
_save_as: wrapped in an async closure (was sync before) since
save_post_file is now async. Uses bookmark_done/error signals
for status instead of direct showMessage.
_batch_download_to: creates fetcher once at the top, shared
across the batch.
New _get_category_fetcher helper returns the fetcher from a fresh
client (lightweight — shares the global httpx pool) or None if no
site is active.
save_post_file is now async and gains an optional
category_fetcher parameter. When the template uses any category
token (%artist%, %character%, %copyright%, %general%, %meta%,
%species%) AND the post's tag_categories is empty AND a fetcher
is available, it awaits ensure_categories(post) before calling
render_filename_template. This guarantees the filename is
correct even when saving a post the user hasn't clicked
(bypassing the info panel's on-display trigger).
When the template uses only non-category tokens (%id%, %md5%,
%score%, %rating%, %ext%) or is empty, the ensure check is
skipped entirely — no HTTP overhead for the common case.
Every existing caller already runs from _run_async closures,
so the sync→async signature change is mechanical. The callers
are updated in the next two commits to pass category_fetcher.
Three changes:
1. _make_client passes db=self._db, site_id=s.id so Gelbooru and
Moebooru clients get a CategoryFetcher attached via the factory.
2. _on_post_activated calls _ensure_post_categories_async(post)
after setting up the preview. If the post has empty categories
(background prefetch hasn't reached it yet, or cache miss),
this schedules ensure_categories on the async loop. When it
completes, it emits categories_updated via the Qt signal.
3. _on_categories_updated slot re-renders the info panel and
preview pane tag display when the currently-selected post's
categories arrive. Stale updates (user clicked a different post
before the fill completed) are silently dropped by the post.id
check.
client_for_type gains optional db + site_id kwargs. When both are
passed and api_type is gelbooru or moebooru, a CategoryFetcher is
constructed and assigned to client.category_fetcher. The fetcher
owns the per-tag cache, the batch tag API fast path, and the
per-post HTML scrape fallback.
Danbooru and e621 never get a fetcher — their inline JSON
categorization is already optimal.
Test Connection dialog and scripts don't pass db/site_id, so they
get fetcher-less clients with the existing search behavior.
Override _post_view_url to return /post/show/{id} for the per-post
HTML scrape path. No _tag_api_url override — Moebooru has no batch
tag DAPI; the CategoryFetcher dispatch goes straight to per-post
HTML for these sites.
search() and get_post() now call prefetch_batch when a fetcher is
attached, same fire-and-forget pattern as gelbooru.py.
