Add Feed response fields (description, next_check_at, no_media_player,
icon, and notification fields), the no_media_player and description
fields to the feed creation and modification requests, and a Tags
filter for entry queries.
The entry list path applied the category_id query parameter twice: once
with validation in findEntries and again without validation in
configureFilters. Drop the unvalidated second application so an invalid
category_id consistently returns a bad request.
The categories handler parsed the counts query parameter by comparing
the raw string to "true". Use QueryBoolParam for consistency with the
other boolean query parameters in the package.
The admin user lookup handlers reported a database error from UserByID
and UserByUsername as a bad request with a generic message. Return a
server error instead, consistent with the other user handlers.
The update-feed handler returned a not found response for any error
from FeedByID, masking genuine database failures. Return a server error
on failure and keep the not found response for a nil feed.
The current-user and categories handlers used the value returned by
UserByID without checking whether it was nil. Since UserByID returns no
error when the user does not exist, the categories handler could
dereference a nil user. Return a not found response when the user is nil.
The mark-user-as-read and integrations-status handlers treated any
error from UserByID as a 404, which masked genuine database failures
and never detected a missing user since UserByID returns no error when
the user does not exist. Return a server error on failure and a not
found response only when the user is nil.
The single-entry endpoint proxified enclosure URLs while the list
endpoints did not, so the same enclosure was returned with different
URLs depending on the endpoint used.
When attempting to set a proxy in a feed, due to a regression caused by 4cd9dd6af7
the form validation would now only accept HTTP(S) proxies, and reject SOCKS proxies, despite this being
a valid configuration before. Adding a new function to validate proxy URLs separates the concerns
and allows using a SOCKS proxy url again.
The subscription finder holds a shared RequestBuilder whose methods mutate
in place. Disabling redirects while probing well-known feed URLs flipped
that flag on the shared builder permanently, leaking into the finder's
other requests instead of scoping to the probe.
Add a Clone method and derive an isolated builder for the probe so the
redirect setting no longer escapes the loop.
The R keyboard shortcut used a fetch request that followed the redirect
to /feeds automatically, consuming the one-time flash message before the
browser navigated there a second time, so no success alert appeared.
Submit a real form POST instead so the browser follows the redirect once
and renders the flash message, matching the menu button behavior.
Fixes: #4358
This switches plainto_tsquery to instead use websearch_to_tsquery,
introduced in PostgreSQL 11. With unquoted text it behaves the same, but
allows to use quoted text and OR and negation in the search terms.
Every build feed method does similar things in a different way. That makes it harder to read these implementations.
Remove nesting creep in loops to make code simpler.
Add GET /v1/entries/ids to return paginated entry IDs for the current user.
The endpoint supports status and starred filters, returns the total matching
count, and exposes matching client methods.
PostgreSQL 18 disables MD5 when running in FIPS mode, which made
md5(url) unusable. This broke enclosure creation with:
store: unable to create enclosure: pq: could not compute MD5
hash: unsupported (XX000)
Replace md5(url) with encode(sha256(url::bytea), 'hex') everywhere:
- The historical migrations that created the enclosures index are
changed to sha256 so fresh installs no longer fail while replaying
them on FIPS-mode PostgreSQL 18.
- A new migration rebuilds enclosures_user_entry_url_unique_idx with
sha256 to convert existing installs.
- The ON CONFLICT clause in createEnclosure is updated to match the
new expression index.
According to `openssl speed -bytes 256 md5 sha256`, this is a performance
improvement as well :D
Finally, the PostgreSQL minimum version was bumped from 9.5 to 11, the lowest
version to support SHA256.
Fixes#4350
The /reader/api/0/subscription/quickadd endpoint was creating its
request builder without calling WithUserAgent, so outbound feed
fetches used Go's default user agent (Go-http-client/2.x) instead
of the operator-configured HTTP_CLIENT_USER_AGENT.
Sites like Reddit that block Go's default user agent would return
403, causing quickadd to fail even though adding the same feed via
the Miniflux web UI succeeded (the UI's subscription handler correctly
sets the configured user agent).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Translation forms that intentionally omit the count, such as the Arabic
dual, no longer render a trailing %!(EXTRA ...) marker. The printer now
formats with the supplied arguments only when the string has a real
directive, and skips them otherwise while still unescaping percents.
This also fixes the same issue in the Polish and Romanian one-forms, and
makes the missing-translation fallback return the bare key.
iconPath() is called a bunch of times on virtually every pages. Each call did a
fmt.Sprintf to build a string of the form
"{basePath}/icon/{checksum}/{filename}". Since both the base path and
BinaryBundles are fixed at startup, all results are determinate.
This commit precomputes the full URL for every embedded bundle once in
funcMap.Map() into a map[string]string and turns iconPath into a single map
lookup. The "_/" fallback for unknown filenames was unused, as every caller
passes a compile-time literal present in bin/, so it was removed.
Microbenchmarks are showing stupidly high gains of course, but anything macro
is non-trivial, and I gave up on it. Knowing that it removes at least one heap
allocation and a couple of reflection calls on every iconPath call is enough to
bring me joy.
While inspecting the network requests done by miniflux' web interface, I
noticed that immutable assets with the `immutable` header were still fetched.
So I reread
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Cache-Control:
> immutable tells a cache that the response is immutable while it's fresh and
avoids those kinds of unnecessary conditional requests to the server.
So it needs to have max-age as well, otherwise the browser will consider the
resource as not fresh, and will perform a network request (and get a 304).
When a non-existent username was submitted, CheckPassword returned
immediately without performing a bcrypt comparison, making it possible
to distinguish valid from invalid usernames by measuring response time.
Perform a dummy bcrypt comparison against a fixed cost-10 hash when
the user is not found so the response time is indistinguishable from
a real password check.
Filter rules are evaluated once per entry on every feed refresh. The
previous code called regexp.Compile / regexp.MatchString per call,
recompiling the same patterns N times per refresh: once per entry per
filter rule, plus once per entry for feed.BlocklistRules and
feed.KeeplistRules.
This commit routes all regex compilations through a small cachedRegex() helper
that memoizes results in a process-wide map (RWMutex protected, since we need
len() and atomic reset that sync.Map doesn't expose). A nil cached value means
the pattern previously failed to compile.
To prevent unbounded memory growth from an authenticated user churning
distinct patterns, the cache is completely reset once it reaches
maxCachedRegexes entries.
Benchmarked on a 50-entry refresh with 6 distinct regex rules:
before: ~408 µs/op 11,580 B/op 131 allocs/op
after: ~271 µs/op ~0 B/op 0 allocs/op
Making it roughly 33% faster with zero allocations per feed-refresh batch,
scaling linearly with entry count, yay.
The old query ran two correlated subqueries per category row: one to
count feeds and one to count unread entries. For N categories this meant
2xN subquery executions.
This commit replaces them with two pre-aggregated subqueries joined once to
categories.
The change was validated against my live database with 5 categories, ~500 feeds
and 12k unreads:
| | Old (correlated) | New (pre-aggregated) |
|---------------------|---------------------|------------------------|
| Execution time | 3.216 ms | 3.123 ms |
| Planning time | 0.732 ms | 0.778 ms |
| Buffer hits | 3,141 | 540 |
| Index searches | 1,237 | 1 |
| Seq scans on feeds | 10 (2 per category) | 2 (once each subquery) |
| Subquery executions | 10 (2x5 categories) | 2 (fixed) |
A non-admin could bind an arbitrary OAuth identity to their own account
by patching google_id or openid_connect_id, bypassing the duplicate
check enforced in the OAuth callback. Remove both fields from the
update request so binding only happens through the OAuth flow.
Browsers normalize backslashes to forward slashes, so a redirect target
like "/\evil.com" parsed as a relative path by url.Parse and resolved to
//evil.com by the browser, resulting in an open redirect. Reject any link
containing a backslash before validating it as a relative path.
WithTags previously emitted one "LOWER($i) = ANY(LOWER(e.tags::text)::text[])"
condition and one parameter per filter tag. With K tags this means K
predicates, K parameters, and K evaluations of the LOWER(e.tags::text)::text[]
sub-expression per candidate row.
This commit replaces the loop with a single predicate using the array
containment operator:
LOWER(e.tags::text)::text[] @> LOWER($N::text)::text[]
Same case-insensitive "row must contain all listed tags" semantics, fewer
predicates for the planner, and the row-side expression is referenced once
instead of K times.
findSubscriptionsFromWebPage was running four separate goquery.Find passes, one
per supported MIME type. This commit replaces it with one
doc.Find("link[type]") traversal and a switch on the type attribute, which is
functionally equivalent, but it visits the DOM once instead of four times and
emits results in document order. Microbenchmark on a representative <head> (8
link tags + ~100 unrelated head children):
before: ~25 µs/op, 1536 B/op, 50 allocs/op
after: ~9 µs/op, 744 B/op, 21 allocs/op
- Don't call strings.TrimSpace twice on canonicalHref
- Use doc.FindMatcher+goquery.Single instead of doc.Find+First, as is done
everywhere else in the codebase.
Replace LEFT JOIN with INNER JOIN in queries where the WHERE clause or
foreign key constraints already guarantee matching rows exist:
- Icons(): filters on feeds.user_id
- UserByAPIKey(): filters on api_keys.token
- fetchFeedCounter(): filters on feeds columns when counterJoinFeeds is set
- fetchEntries(): entries have FK constraints to feeds, categories, and
users (feed_icons/icons kept as LEFT JOIN since icons are optional)
PostgreSQL already optimizes these identically, but INNER JOIN makes the
intent explicit.
The WHERE clause on feeds columns already eliminates NULL-extended rows,
making the LEFT JOINs logically equivalent to INNER JOINs. PostgreSQL's
planner is smart enough to recognize this and produces an identical
execution plan (verified with EXPLAIN ANALYZE), but using INNER JOIN
makes the intent explicit for humans like me reading the query.
findSubscriptionsFromWellKnownURLs iterates a fixed table of 9 well-known
feed paths against the discovered base URLs. The table was declared as a
map[string]string, which paid per-iteration hash overhead and gave
non-deterministic probe order across runs.
This commit replaces it with a fixed-size [...]struct{path, format string}.
Probes now run in declared order (more predictable behavior for users when
multiple well-known URLs respond, and easier to reason about in tests), and the
inner loop avoids the map iterator entirely.
Micro-benchmark replaying the double loop body against 2 base URLs ×
9 paths, sans HTTP I/O (medians of 5 runs):
name old ns/op new ns/op delta
KnownURLs 24,194 18,150 -25%
name old B/op new B/op delta
KnownURLs 9,688 9,072 -6.4%
name old allocs/op new allocs/op delta
KnownURLs 110 107 -3
- No need to use `make(…)` construct for empty slices, as is already the case
in the rest of internal/googlereader/handler.go
- Replace a useless condition in internal/reader/sanitizer/sanitizer.go with an
unconditional assignment.
- Remove a useless call to url.Parse in JoinBaseURLAndPath, as url.JoinPath
already performs the validation internally
parseLocalTimeDates is called once per parsed feed entry. Each call to
time.LoadLocation("America/Los_Angeles" | "America/New_York") reads and
parses the IANA tzdata file from disk or from the embedded zoneinfo.
This commit hoists the two LoadLocation calls to package-level vars so the
lookup happens once at init time and the hot path becomes a pointer load.
Benchmarked on the local-time fallback path:
before: ~24,000 ns/op, ~15 KB/op, 26 allocs/op
after: ~330 ns/op, 0 B/op, 0 allocs/op
The metrics Basic Auth check used != for username and password, which is
technically vulnerable to timing side-channel attacks. Since the metrics
credentials are typically short static config values, the timing difference is
very likely to be in the noise level, but oh well, it's a good practise to do
credential validation in constant time.
entries_feed_idx(feed_id) is covered by both the unique constraint
entries_feed_id_hash_key(feed_id, hash) and the explicit index
entries_feed_id_status_hash_idx(feed_id, status, hash), which handle
all feed_id-leading lookups including FK cascade deletes.
entries_user_status_idx(user_id, status) is a prefix of five existing
three-column indexes (entries_user_status_feed_idx,
entries_user_status_changed_idx, entries_user_status_published_idx,
entries_user_status_created_idx, entries_user_status_changed_published_idx),
all of which serve every query the two-column index could.
Saves ~14 MB per million entries.
Store WebAuthn backup eligibility and backup state with each
credential instead of overwriting BackupEligible from every login
assertion.
Use a nullable backup_eligible column to identify legacy credentials
and backfill those records on their next successful login. Also persist
the validated credential state after login, including sign count, clone
warning, and backup state.
Remove the username-based WebAuthn login flow because it allowed
username enumeration before password verification.
WebAuthn login now uses discoverable credentials only, and new
registrations require resident keys. Existing non-resident credentials
are no longer usable for first-factor login; they should only be used
in a post-password MFA flow, which Miniflux does not currently
implement.
BREAKING CHANGE: Users with existing non-resident WebAuthn credentials
must register a new passkey.
The icons table is deduplicated by hash and shared with feeds via the
feed_icons junction. When a feed is deleted, the ON DELETE CASCADE
removes its feed_icons row but leaves the icons row behind. The same
happens in StoreFeedIcon when a feed's icon is replaced. Over time
these orphaned bytea blobs accumulate and bloat the database.
Add Storage.CleanupOrphanIcons, which deletes icons rows that no
feed_icons row still references, and call it from runCleanupTasks.
The middleware (*authMiddleware).validateApiKey is registered for 14
routes in NewHandler. Its body was just `return http.HandlerFunc(func…)`,
which made the compiler consider it inlinable. As a result, the entire
losure body was duplicated at every call site, taking space in the .text
section.
Move the request-handling logic into a separate non-inlined method
serveValidated and keep validateApiKey as a thin wrapper that only
allocates the http.HandlerFunc. The 14 duplicated symbols are gone and
the stripped binary shrinks from 20,513,033 to 20,447,497 bytes, which isn't
that much, but it's still something, especially for such a simple commit.
Replace `num += string(char)` with index-based slicing of the input
string. The previous loop allocated a new string for every digit in
the duration (and again on each `+=`), giving O(n²) allocation
behavior. Slicing `after[start:i]` reuses the original string's
backing memory and only allocates once per numeric component when
passed to `strconv.Atoi`.
It also makes the code a bit more compact/simpler.
Called once per YouTube/podcast entry that exposes an ISO8601
duration during feed processing.
Drop the snake_case cred_uid, fix the missing-n typos in webAuthUser
and webAuthCredential, replace credCredential with validatedCredential,
unify uid/userID on userID, rename the shadowed url local in
newWebAuthn to baseURL, and use the full credential / credentials names
throughout instead of the abbreviated cred / creds. No behaviour change.
The web session middleware redirects unauthenticated requests to the
login page before any non-public handler runs, so request.UserID is
guaranteed non-zero in beginRegistration, finishRegistration, and
deleteCredential. Remove the dead checks to match the other WebAuthn
handlers.
While both deleteCredential either validate or pass down a uid,
saveCredential doesn't. This isn't exploitable as an authenticated attacker
would need to guess the 32 bytes handle of another one, but it doesn't hurt to
explicitly check if a user is only operating on their own user.
Account unlinking mutates state, so /oauth2/{provider}/unlink can no
longer be reached via GET. Pull the OAuth2/OIDC and WebAuthn sections
out of the settings form and render each as its own fieldset above the
form, with the unlink action as a self-contained POST form. Rename
the username/password fieldset legend to "Password Authentication" and
add matching legends for the federated and passkey sections so all
authentication methods read consistently.
The function strings.Fields will allocate every single word it's creating,
meaning that for a text of 10k words, 10k allocations will be made, only for
them to be counted an discarded. We can do much better by counting the words
ourself via a small countWords helper function, and write a test to prove that
it doesn't allocate anything.
Feed refresh endpoints mutate state and should not be reachable via
GET. Drop the GET registrations on /feeds/refresh, /feed/{id}/refresh,
/category/{id}/feeds/refresh and /category/{id}/entries/refresh, and
update the templates and the R keyboard shortcut to submit POST forms
with a CSRF token.
Logout is a state-changing action and should not be reachable via GET.
Switching to POST routes the request through the CSRF middleware so
prefetchers and cross-site GETs can no longer terminate the session.
When subscribing to podcasts or videocasts (or any feeds with enclosures), Miniflux presents audio/video controls on top, and links on bottom. But it's is not a part of the article, it is Miniflux-only UI modification. A new rewrite rule is added to expose enclosure links to the content, so that it can be accessed outsite of Miniflux, specifically on any native mobile RSS apps. That way, it is now possible to access media files to open them eg. in native media player apps.
The `sess.OAuth2State()` function returns "" when no flow has been initiated,
thus making `subtle.ConstantTimeCompare("", "")` return 1, making a callback
with state= passes this check.
This isn't an exploitable vulnerability, as PKCE is enforced for both Google
and OIDC (authorization.go:45-49, google.go:57-60) and the missing
code_verifier will fail at the IdP token exchange.
This was found as I was digging into Forgejo OAuth2's implementation, and
wondered how miniflux was faring.
Previously, every allowed attribute had to allocate memory for a slice,
concatenated in `key=…`, and a final strings.Join call. Instead of doing all of
this, a single string.Builder is used, with two simple local functions. It
doesn't significantly complexify the code, while improving the performances.
Now, I know that this isn't really a bottleneck in miniflux, but the
improvement is around 50% for the wikipedia/github benchmark (BenchmarkSanitize),
so I think it's worth it, given that the sanitizeAttributes function is call
for every attribute a feed items.
Before:
```
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/sanitizer
BenchmarkSanitizeImageHeavyNoQuery-8 6326 567676 ns/op 78305 B/op 918 allocs/op
BenchmarkSanitizeImageHeavyWithQuery-8 4186 1028312 ns/op 124353 B/op 1366 allocs/op
BenchmarkSanitize-8 50 21440566 ns/op
PASS
ok miniflux.app/v2/internal/reader/sanitizer 9.039s
```
After:
```
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/sanitizer
BenchmarkSanitizeImageHeavyNoQuery-8 7512 530973 ns/op 73186 B/op 790 allocs/op
BenchmarkSanitizeImageHeavyWithQuery-8 1173 1023078 ns/op 118209 B/op 1238 allocs/op
BenchmarkSanitize-8 92 13265599 ns/op
PASS
ok miniflux.app/v2/internal/reader/sanitizer 6.553s
```
On the main page (showing ~every unread feed item), every item
uses at least 4 icons. For 100 unread items, that's 400 icons, meaning
that the `icon` func is called 400 times.
On my local microbenchmark, using `fmt.Sprintf` uses one dynamic allocation and
takes ~275ns. Using concatenation in a dedicated function (that gets inlined)
doesn't allocate any memory, and takes ~2.1ns. This thus saves 400 short-lived
allocations and reduces the execution time by a factor of 100.
Extend OPML export to include Miniflux-specific feed settings as custom
attributes on each outline element (scraper rules, rewrite rules, URL
rewrite rules, blocklist/keeplist rules, user agent, cookie, proxy URL,
and various boolean flags).
On import, these attributes are applied when creating new feeds, allowing
a Miniflux OPML export to serve as a full backup and restore mechanism
for feed configuration. Existing feeds are not modified to preserve the
original import semantics.
Obtaining the amount of unread entries, errored feeds and if the user has
integrations enabled can be done in a single query, instead of doing it one by
one. This should reduce the amount of queries from 3 or 2 to 1, depending on
the page.
This commit is touching a significant amount of files, and the
search-and-replace, while done with love and care, would benefit from a
thorough review, to ensure that nothing was subtly broken.
Reject API "limit" query parameter and "entries_per_page" preference
values above 1000, and clamp the storage entry query builder so any
caller (REST API, Google Reader, internal UI) is bounded.
The HTML settings form now exposes the cap via the input "max"
attribute.
Per RFC 3986 §3.1, URI schemes are case-insensitive. HasValidURIScheme
previously did a literal HasPrefix check, so inputs like "HTTPS://..."
were rejected. Use strings.Cut to extract the scheme and compare each
allowlisted entry with strings.EqualFold.
safeURL wrapped any string in template.URL, defeating html/template's
URL filter and allowing javascript:/data: URIs from feed entries to
render verbatim.
untrustedURL validates the scheme via sanitizer.HasValidURIScheme,
falling back to "#" otherwise. The sanitizer allowlist is reused
because html/template's built-in filter is too narrow for feeds (only
http(s), mailto, and relative URLs).
Reject any URI that is not a same-origin relative path or an absolute
http(s) URL, preventing attacker-controlled feed entry URLs (e.g.
javascript:, data:, mailto:, scheme-relative //host/...) from being
emitted in a Location header.
4 to 1, with a meaningful ns/op win on hot feed-parse paths. personNames trades
a slightly larger fixed allocation for fewer growths and gets ~29% faster.
The micro-benchmarks used to obtain those numbers are, well, micro-benchmarks,
and thus I don't think it would make sense to add them to miniflux.
Replaces tag references (e.g. @v6) with the exact commit SHA and a
trailing version comment across all workflows. Pinning by SHA prevents
supply-chain risk from a tag being moved to a malicious commit.
Marks pull requests as stale after 60 days of inactivity and closes them
14 days later. Issues are excluded. Runs daily and can be triggered
manually via workflow_dispatch.
Previously CategoriesWithFeedCount loaded the user via UserByID just
to read CategoriesSortingOrder. Pass it in explicitly so the UI path
(which already has the user loaded) avoids a redundant lookup.
The integrations table lacked a foreign key to users, forcing
RemoveUser to delete integration rows explicitly and RemoveUserAsync
to iterate over feeds to avoid a long-running transaction.
Add a migration introducing the missing ON DELETE CASCADE foreign key
and drop both workarounds. The sole caller of RemoveUserAsync inlines
the goroutine so the fire-and-forget behaviour is visible at the call
site.
Also fix godoc comments in user.go.
The previous implementation iterated over entries and issued one DELETE
per row as a workaround to avoid a long-running transaction when
removing feeds with many entries. In practice this caused N+1
round-trips and took over 3 minutes to delete a feed with 22k entries.
Rely on the ON DELETE CASCADE on entries.feed_id (and transitively
enclosures.entry_id) and issue a single DELETE on feeds. Postgres
handles large cascaded deletes efficiently with row-level locking.
Also fix grammar, accuracy, and a missing godoc comment across the
exported functions in feed.go.
Archived entries could be re-ingested as new unread rows when a feed
re-emitted them. Replace the "removed" soft-delete status with an
entry_tombstones table keyed on (feed_id, hash); the INSERT is guarded
by WHERE NOT EXISTS so archival and refresh can no longer race.
Lift the validation that rejected DISABLE_LOCAL_AUTH=1 combined with
OAUTH2_USER_CREATION=0 or AUTH_PROXY_USER_CREATION=0. Admins can now
pre-create users and forbid auto-registration while still forcing all
logins through OAuth2 or an auth proxy.
Fixes: #3163
The flash message keys `successMessage` and `errorMessage` collided with
per-template form validation variables set by handlers (e.g. add
subscription), causing form errors to also render in the global flash
banner. Rename the flash keys to `flashSuccessMessage` and
`flashErrorMessage` to keep them distinct from form-scoped variables.
Treat user-configured proxies (feed proxy, application proxy, proxy
rotator) as trusted hops so that the private-network restriction does
not block requests routed through a proxy listening on a private
address. Direct requests and redirects still enforce the check.
When "Read articles by opening external links" is enabled, the entry
title routes to an internal handler that HTTP-redirects to the external
URL. The title link had no target="_blank", so the redirect was followed
in the current tab, ignoring "Open external links in a new tab".
Add target="_blank" to the title link in every entry list template when
both preferences are enabled.
Refs: #4241
Wrap the UI handler chain with http.CrossOriginProtection as the
outermost layer so cross-origin unsafe-method requests are rejected
via Sec-Fetch-Site/Origin checks before session lookup or token CSRF
validation runs. Stacks with the existing per-session token CSRF for
defense in depth; API handlers are unaffected.
There is no need to allocate a new HTMLMinifer every time a page needs to be
minified, as HTMLMinifer is thread-safe, we can using a singleton instead.
The memory overhead is negligible, as a minify.M struct is small.
Inspect response headers to recognize Cloudflare interstitial pages
(cf-mitigated: challenge, or 403/503 served by cloudflare with cf-ray
and an HTML body) and surface a dedicated localized error instead of a
generic HTTP failure.
Some non-conformant feeds ship the same <guid> for every item, which
caused Miniflux to collapse all of them into a single entry. Keep the
first occurrence hashed as SHA256(guid) so existing stored entries
still match, and disambiguate later collisions using the entry URL
(falling back to the item position when no URL is available).
The vars.PUBLISH_DOCKER_IMAGES gate was redundant with the existing
repository_owner check and the push condition on the build steps, so
it has been removed.
In InsertEntryForFeed, there is no need to check if an entry exists and then
get its hash. Instead, try to get its hash, and consider the ErrNoRows error as
an existence check.
When marking a feed as read from the category feeds page
(/category/n/feeds), the redirect now returns to the same category
feeds page instead of the global /feeds page.
Every list page load now executes one fewer database round-trip, eliminating a
full 3-table JOIN (`entries` + `feeds` + `categories`) that was previously done
solely for counting. The `unread_entries.go` page haw two fewer SQL queries in
the common case.
Move option combination checks from cli.go into an exported
Validate() method on configOptions, called after both config file
and environment variable parsing complete. Add new checks for
TLS, metrics auth, database pool, and scheduler interval consistency.
NewManager now dispatches on the configured provider name instead of
inferring it from the presence of a discovery endpoint. The CLI validates
that OAUTH2_OIDC_DISCOVERY_ENDPOINT is set when the OIDC provider is
selected.
Remove Get prefix from Provider interface methods, rename Profile
struct to UserProfile to avoid method name collision, fix acronym
casing (authUrl → authURL), fix receiver naming, and return
interfaces from constructors instead of unexported concrete types.
Every keypress triggers a loop over all registered shortcuts, and each
iteration calls `.split(" ")` on the combination string. With ~30 shortcuts
registered that's 30 `String.split()` allocations on every single keypress.
But since these results never change after `on()` is called, they can be
computed once there and then cached.
Instead of interleaving DOM read/write, collect all the items first, then
modify them. This has a small but noticeable performance impact, as
`classList.add` might triggers a style invalidation.
The fetchFeedCounter query should happen before the the db.Query call, so that
a single database connection can be (re)used, instead of using a second one.
Register either the OIDC provider when a discovery endpoint is
configured, or the Google provider otherwise, but never both.
Also scope the empty client secret warning to OIDC configuration.
Prevents replay of the one-time-use PKCE code verifier and CSRF state
values by clearing them from the app session immediately after the
OAuth2 callback consumes them.
The ID token from the token exchange was not being validated. Now the
provider's JWKS keys are used to verify the JWT signature, issuer,
audience, and expiry. The verified subject is also cross-checked against
the UserInfo response to detect mismatches.
Previously, when an iframe was blocked by the allowlist check, its child
elements were still rendered as content. Now blocked iframes discard
their children, matching the behavior of allowed iframes.
Return errors instead of zero values or nil from CountUsers,
CountAllFeeds, CountAllFeedsWithErrors, and CountAllEntries so the
metric collector can log failures and preserve previous gauge values.
Move JS and CSS routes to /js/{checksum}/{filename} and
/stylesheets/{checksum}/{filename}, matching the icon route pattern.
Use full filenames (e.g. "app.js", "light_serif.css") as bundle map
keys so handlers can use PathValue directly without suffix stripping.
Embed content checksums in icon URLs (e.g. /icon/<checksum>/sprite.svg)
so browsers fetch updated assets on upgrade instead of serving stale
cached versions. This matches the existing pattern used for JS and CSS
bundles.
Closes#3728
There is no need to call `[]byte(mediaURL)` twice. It might make a minor
difference performance-wise on images-heavy feed items with long media URL,
like autogenerated CDN ones.
The plural-form switch handled Arabic under ar_AR, but the supported
locale and translation file use ar_SA.
That mismatch made Arabic requests fall back to the default two-form
rule, so plural translations selected the wrong string at runtime.
Avoid scanning and allocating entire untrusted feed titles during
template rendering just to truncate them. The previous
implementation could turn large remote titles into unnecessary CPU
and memory pressure on each page render.
Iterating only until the requested rune boundary preserves
Unicode-safe truncation while keeping the work proportional to the
visible output instead of the full input.
Treat 100.64.0.0/10 as non-public in urllib.IsNonPublicIP.
This closes a gap where RFC 6598 shared address space was not
classified as non-public, which could allow outbound requests to
CGNAT addresses through code paths that rely on this helper.
Add a go:fix inline directive to client.New so Go 1.26 can
automatically rewrite callers to NewClient().
This keeps the deprecated wrapper in place while making migration
easier for library users.
- Hoist 4 similar error into a single global variable
- Hoist a bunch of fmt.Sprintf calls outside of loops
- Remove an else-return construct
- Use strconv.FormatInt instead of fmt.Sprintf
Move the private-network IP check from a pre-flight DNS lookup into
net.Dialer.Control, which runs after DNS resolution but before the TCP
connection is established. This ensures the validated IP is the one
actually connected to, closing the TOCTOU window that allowed DNS
rebinding attacks.
As a side effect, the check now also applies to redirect targets,
which the previous pre-flight approach did not cover.
Add a shared HTTP client factory that blocks connections to private
network addresses at connect time via a custom DialContext, preventing
SSRF and DNS-rebinding attacks.
A new INTEGRATION_ALLOW_PRIVATE_NETWORKS option (default: false)
controls this behavior. Integrations targeting fixed external services
(Telegram, Archive.org, Pinboard, Notion, Instapaper) skip the check.
Apply entry filters in two phases:
- Before scraping, to skip unnecessary requests.
- After scraping (when crawler runs), so rules can match fetched/original content.
Block outbound requests to private networks made by the fetcher
by default. The restriction now applies to all outgoing requests
performed by the fetcher.
Previous PR #3947 intentionally enforced this restriction only
for the media proxy and icon fetching, considering the
self-hosted nature of Miniflux.
Right after `originalFeed.CheckedNow()`, `originalFeed.ScheduleNextCheck` is
called, with its second argument being `refreshdelay`, a time.Duration set to its implicit
default value. The variable `refreshdelay` is never reassigned, until much
deeper in the function. This commit replaces it with `time.Duration(0)`, to
make it explicit that no special delay is expected, and declares it later in a
restricted scope.
Instead of parsing the HTML once in findSubscriptionsFromWebPage and another
time in findCanonicalURL, do it once in a `parseHTMLDocument` function.
This was also the opportunity to restrict the `link[rel='canonical' i`
query to `head link[rel='canonical' i`.
- Don't clean an url without any parameters
- Don't recompute the Hostname for parsedFeedURL and parsedSiteURL in
a loop
- Don't keep processing a parameter once it has been detected as a tracking
one.
- Don't create a string by concatenation on the heap to then pass it to a
strings.Builder, pass everything, one by one, to it instead.
- Lower the complexity of shouldRemoveCandidate from quadratic to linear.
There is no need to process the whole page to guess its format: if we can't
find a format indicator in the first 50 xml tokens, odds are that we won't find
it at all.
This should save some time when trying to find a feed, as this function is
called a handful of times on various pages.
Instead of operating on a slices that is built/garbage-collected on every HTML
tag, use a struct keeping track of the mandatory attributes. This commit also
replaces two `continue` with `return`, as there is no point to continue
analysing the tag if the conditions surrounding the continue aren't met.
It introduces a new configuration option `ignore_entry_updates` for feeds, allowing users to skip updating existing entries during scheduled polling.
This is useful when external services (e.g., AI summarizers) modify entry content and users want to preserve those modifications across feed syncs.
When using the 'v' shortcut in a Safari PWA on macOS, Miniflux opened
the link in something similar to an 'in-app browser' with no URL bar and
a separate window for each link. When clicking the link with the mouse,
a "normal" browser tab is opened instead.
It looks like target="_blank" on the <a> element implies the "noopener"
attribute, which causes Safari to use the main browser. For window.open,
"noopener" is not implied. This commit adds the "noreferrer" argument to
the window.open call (which implies the "noopener" option) rather than
changing the window.opener attribute manually. This option causes Safari
to use the main browser with the 'v' shortcut.
Since FindSubscriptions is reading the Body in a []byte, there is no need to
wrap it into a Reader for NewCharsetReader to ReadAll into another buffer: we
can simply pass the []byte instead. For some large webpages, this might shave
off two allocation-deallocation of a handful of megabytes when looking for
subscriptions.
- Sort blockedResourceURLSubstrings
- Add x.com to the list of blocked links
- Move isBlockedResource above ResolveToAbsoluteURLWithParsedBaseURL, to
avoid having to properly parse blocked urls.
Move hasRequiredAttributes above getExtraAttributes, as extra attributes,
as their name implies, aren't required. This allows to return a []string
instead of a ([]string, []string), as well as simplifying getExtraAttributes
of course. Note that this resulted in an ordering change of <iframe> attributes
so the testsuite had to be updated.
This improves the performances of SanitizeHTML by a bit less than 10% on local
benchmarks.
There is no need to do a lookup in allowedHTMLTagsAndAttributes in the loop,
as the tag never changes in this function. This is a minor optimization,
but since sanitizeAttributes is in the hot path of SanitizeHTML, it should be
worthwhile.
As stated in [golang.org/x/net/html's documentation](https://pkg.go.dev/golang.org/x/net):
> If your use case requires semantically well-formed HTML documents, as defined
by the WHATWG specification, the parser should be used rather than the
tokenizer.
The sanitizer is modifying the HTML document, and I'm pretty sure we don't want
to implement a WHATWG-compliant parser ourself, as there are _so many_
edge cases everywhere. Picking the parser instead of the tokenizer ensures that
miniflux has the same view of the DOM than web browsers, removing the risk of
disparities, and thus injection-related security issues. Another benefit of
this commit is to make the code way more readable/simple.
The only changes made to the testsuites are:
- Removing the trailing `/` on self-contained tags, as the spec doesn't require
them. Adding them systematically could also have been done, but as the
testsuite isn't consistent in this regard, it would have significantly
increased the size of the commit to normalize it.
- Some weird things that were wrong in the first place, like how
`<td rowspan="<b>injection</b>">text</td>` is interpreted by the browser as
`text`, and not as `<td rowspan="<b>test</b>">test</td>`.
Care has been taken to re-use as much code from the tokenizer-based sanitizer
as possible.
The new implementation follows the WebKit HTMLSrcsetParser approach. It
correctly handles various edge cases that the previous string-splitting
method could not parse.
Then there are things in blockedStack, it means that we're currently iterating
on children of a blocked tag, so there is no need to actually process them.
Most browsers allow opening links in a new window or new
tab by holding down a modifier key like shift, command,
control (depending on the OS and browser) while clicking a link.
Unconditional event.preventDefault() on click events breaks this functionality.
- Add a new IsValid method to avoid having to materialize a map just to check
if a given timezone is in it.
- Use an iterator instead of materializing a map every time the full list of
timezones is required.
- Use a slice in the template instead of a map to iterate over all timezones.
This option can be replaced with a filter rule `max-age:<duration>` instead.
Global environment variables should be reserved for the Miniflux process
configuration that are not meant to be modified by end users.
The `output` parameter seems to be optional and Miniflux will always
returns a JSON response. This change makes Miniflux more consistent with
other open source RSS readers.
The Google Reader API is not supposed to be used by web clients.
Removing CORS should not break any Google Reader client and it
reduces the attack surface.
Add an IP-based allow list to prevent spoofing of HTTP headers that
should only be set by trusted reverse proxies.
Note that `TRUSTED_REVERSE_PROXY_NETWORKS` must be configured when
`AUTH_PROXY_HEADER` is used.
The following HTTP headers are taken into consideration only when the
client is an allowed reverse proxy: `X-Forwarded-For`,
`X-Forwarded-Proto` and `X-Real-Ip`.
`document_vectors_idx` used to take 30M before this commit, now it's only 226
kB, after a full VACUUM. A handy command to check the size of indexes is `\di+
public.*`
There is no need to match on the whole title, use two groups, then merge the
whole thing together, when we can simply search-and-replace only the matching
value.
- There is no need to use two replacers instead of one. It might help keep the
input in a low CPU cache.
- Don't cast an int to float (twice) for nothing in checkTimezoneRange.
There is no need to iterate twice on the attributes of source and img tags,
when we can do it once. This shouldn't bring any noticeable performances
improvements, except maybe in some extreme cases on pages with a lot of images
with a lot of attributes.
No need to to invoke the whole Printf machinery for constant strings. While
this shouldn't have an impact on memory consumption nor allocation (as
constructing errors to return is never in a hot path), this should reduce a bit
the code size, as errors.New will be inlined to a simple struct initialization
instead of a function call.
Having the CSP built in a function instead of in the template makes it easier
to properly construct it. This was also the opportunity to switch from
default-src 'self' to default-src 'none', to deny everything that isn't
explicitly allowed, instead of allowing everything coming from 'self'.
Moreover, as Miniflux is shoving the content of feeds in the same origin as
itself, using self doesn't do much security-wise. It's much better to
systematically use a nonce-based policy, so that an attacker able to bypass the
sanitization will have to guess the nonce to gain arbitrary javascript
execution.
Adds a new content rewrite rule to strip image URL query parameters from blurred images.
This addresses issues with sites like Belgian national news that use blurry placeholder images which get replaced with high-quality versions, allowing Miniflux to fetch the original images instead of the placeholders.
- There is no need for getEncoding to return a string instead of an array of
bytes, so let's make it return a []byte instead of a string.
- There is no reason why the function used for decoder.CharsetReader
has to be defined as a lambda instead of a proper function. One might argue
the other way around, but a lambda is living on the heap, while a "real"
function doesn't.
The hstore extension was briefly used at some point by miniflux, but not
anymore. Yet it's still required to deploy miniflux, as a hstore column is
created then destroyed during the database creation/migration. This commit
refactor the migrations (scary!) to get rid of hstore, so that it doesn't need
to be installed/present when deploying/running miniflux.
This should close#3759
- use an array instead of a map for the schemes, as the overwhelming majority
of them will be either http or https, which we can place in front of the
array. This is faster than using a map.
- Simplify hasValidURIScheme by using strings.HasPrefix instead of doing
strings.IndexByte
- Simplify isBlockedResource by using a simple for loop, instead of a weird
slices.ContainsFunc+strings.Contains construct.
On my noisy system:
```
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/sanitizer
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Sanitize-8 22.19m ± 4% 21.97m ± 4% ~ (p=0.948 n=50)
```
This PR refactors the configuration parser, replacing the old parser implementation with a new, more structured approach that includes validation and improved organization.
Key changes:
- Complete rewrite of the configuration parser using a map-based structure with built-in validation
- Addition of comprehensive validator functions for configuration values
- Renamed numerous configuration getter methods for better consistency
- Look for JSON feeds in one pass instead of two
- Move conditions around to reduce the amount of comparisons
- Edit an existing test to exercise this commit's changes
Optimizes the filterValidXMLChars function by changing the loop variable type from int to uint to eliminate bound checks during compilation, resulting in a ~4% performance improvement.
- Changes loop variable i from int to uint to remove compiler-generated bound checks
- Adjusts type conversions accordingly to maintain correctness
```
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/parser
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Parse-8 40.91m ± 3% 39.30m ± 2% -3.94% (p=0.000 n=50)
```
- Implement a better/simpler polyfill for web browsers that don't supported
trusted types yet
- Use two separate policies: one to create HTML, another to create/use script
urls
- Instead of having the policy live in the top-level scope, they're now
declared at the lowest possible scope, right before they're used, making them
inaccessible outside of it. This puts their usage completely out of reach of
an attacker unable to gain some control outside of those two (small) scopes,
and thus removes the need to tighten the policies.
- Remove the now-unused tt.js file
This has been tested on Firefox (doesn't support trusted types) and on Chromium
(does support trusted types).
Replaces usage of the word "bookmark" with "star"/"starred" in order to be more
consistent with the UI and database models, and to reduce confusion with
"bookmarklet" and integration features.
This is in preparation of future work on read-it-later features.
Which are also not called "bookmarks" to prevent any further confusion.
https://github.com/orgs/miniflux/discussions/3719
Related-to: https://github.com/miniflux/v2/pull/2219
In general, duration is used as time unit representation.
At some places when int is returned, there's no documentation which unit is used.
So just convert to time.Duration ASAP.
- b.body can never be of type error, so let's remove it from the switch-case
construct.
- there is no need to use defer when the only return statement is two lines
after.
Instead of blindly compiling all the common/ templates for every view/ ones,
let's be explicit about the dependencies. This should significantly decrease
the resident memory consumption, as ParseTemplate is responsible for ~10M of
the current 11M of heap memory on my instance, so any win there is interesting.
This will also allow better factorization of templates, now that everything is
explicit. Another side-effect is that it'll make testing easier, as we now have
a comprehensive list of views/ templates affected by a change in a file in
common/
Some Miniflux clients expect a specific version format.
For example, Flux News converts the string version to an integer.
Using `Development Version` will break some clients.
The length of a tsvector (lexemes + positions) must be less than 1 megabyte.
We don't need to index the entire content, and we need to keep a buffer for the positions.
- Add `content IS NOT NULL` to avoid clearing the same entries over and over
- Create `DeleteRemovedEntriesEnclosures` and `ClearRemovedEntriesContent` that report individual counts
Don't use a slice of pointers to opml items, when we can simply use a slice of
items instead. This should reduce the amount of memory allocations and the
number of indirections the GC has to process, speedup up the import process.
Note that this doesn't introduce any additional copies, as the only time a
slice of subscription is created, the items are created and inserted inline.
For small fixed-size structures, it's better to use a slice of values, instead
of a slice of pointers to values: they're stored contiguously and thus can be
iterated on quickly by the CPU, and it does remove an indirection per object
every time the GC kicks in.
- Use `user` everywhere, instead of sometimes `loggedUser`
- Delay the instantiation of some variables: no need to perform SQL queries for
nothing.
- Remove a SQL query getting the whole user struct when only user.ID is used.
Since tdewolff/minify supports SVG minimization, let's make use of it. As we
need to keep the license in the SVG because we're nice netizens, we can at
least use SPDX identifiers instead of using it verbatim.
This does save a couple of kB.
Each batch of feeds sent to the worker pool is now guaranteed to contain unique feed URLs.
When `POLLING_LIMIT_PER_HOST` is set, an additional limit is applied to the number of concurrent requests per hostname, helping to prevent overloading a single server.
Note: Additional requests may still be made during feed refresh. For example, to fetch feed icons or when the web scraper is enabled for a particular feed.
The unread page may show outdated entries when navigating back from an article, due to Chrome's back/forward cache (bfcache) restoring the page from memory.
Reference: https://web.dev/articles/bfcache
There is no need to have templates used only used in a single file be part of
every single other ones. This should reduce a bit the resident memory
consumption of miniflux.
- Use a simple struct instead of two slices to store the data and the checksums
of resources
- Remove a superfluous call to Sprintf
- Factorise presence check and data retrieval in some maps
- Size the maps when possible
There is no need to perform a heavy-weight SQL query gathering all the
information available on a feed when we're only interested in its last check
timestamp.
- Replace a call to fmt.Sprintf with a concatenation
- Explicit declaration of return values in FetchJobs
- Initialize the size of FetchJobs return value to b.limit: when b.limit is
used, which is most of the time, this avoid resizing the slice, and when it
isn't, the size of the map is set to 0, which is equivalent to the previous
situation anyway.
- Move a call to `request.UserID(r)` to a lower scope.
- Move the seeking inside of DetectFeedFormat instead of having it everywhere
in ParseFeed
- Provide a hint for the compiler to eliminate a useless bound check in
DetectJSONFormat, otherwise it'll check that buffer[i] is valid on every
iteration of the loop. This shouldn't make a big difference, but oh well.
Apparently, postgresql isn't smart enough to realize that once a true value
value is found as part of a `SELECT true`, there is no need to scan the rest of
the table, so we have to make this explicit. We could also have used the
`SELECT EXISTS(…)` construct, but it's more verbose and I don't like it.
- The JS bundle has its own isolated scope
- There is no need to use IIFEs anymore (Immediately Invoked Function Expressions)
- Modules are executed after the HTML document is fully parsed, similar to `defer` attribute
- There is no need to use `DOMContentLoaded` anymore
- Module scripts inherently run in strict mode (no need to define `use strict` anymore)
Since Go doesn't support unions, and because the translation format is a bit
wacky with the same field having multiple types, we must resort to
introspection to switch between single-item translation (for singular), and
multi-items ones (for plurals).
Previously, introspection was done at runtime, which is not only slow, but will
also only catch typing errors while trying to use the translations. The current
approach is to use a struct with a different field per possible type, and
implement a custom unmarshaller to dispatch the translations to the right one.
This should marginally reduce the memory consumption since interface-boxing
doesn't happen anymore, speed up the translations matching, and enforce proper
typing earlier. This also allows us to remove a bunch of now-useless tests.
As stated in the documentation:
> Read calls io.ReadFull on Reader and crashes the program irrecoverably if an
error is returned. The default Reader uses operating system APIs that are
documented to never return an error on all but legacy Linux systems.
Maintaining a separate ChangeLog file is redundant and error-prone,
as it largely duplicated the Git commit history without adding meaningful context.
Release notes are still available on GitHub Releases and the Miniflux website.
- Use an iterator instead of generating a whole slice when iterating on the selection.
- Using an iterator allows to use a for-loop construct, instead of a lambda,
which is a bit clearer
- Do the filtering Find()'s selector, instead of in the loop, which doesn't
matter much now that we're using an iterator, but it makes the code a bit
more obvious/simpler, and likely reduces a bit the number of iterations.
There is no need to materialize the whole text content of the selection only to
count its number of commas. As we already have a getLengthOfTextContent
function that is pretty similar, this commit refactors it to make it more
generic, in the form of a map/fold(+).
There is no installation wizard for Windows and running the
command line binary as-is could lead to confusion for some users.
Unit tests still run on Windows, and people can still compile from
source if interested.
- Don't define the queries before possible early returns
- Check for the presence of the href attribute in the queries, instead of later
on iterating on the selection
- Add two edge-cases to the tests
- Use EachIter instead of Each, if only to avoid the lambda
There is no need to try to match the regexp over the whole input, having it
anchored is enough. If we feel extra-lenient, we might strip spaces in
front/tail, but I don't think it's necessary.
This commit also invert a condition to reduce the level of nested indentation,
and make a condition stricter.
Instead of doing some ciphers manipulation before instantiating the http.Transport
and then assigning them, instantiate http.Transport, and then in an if do the
manipulation. This makes the code a bit clearer, which is always nice when it
comes to cryptographic shenanigans.
- Surface the faulty line number when trying to parse it
- Use strings.Cut instead of strings.SplitN
- Use strings.TrimSuffix instead of an if
- Simplify parseStringList and make its code more compact
There is no need to allocate half a kilobyte of memory only check that a buffer
starts with a bunch of spaces and a `{`, 32b should be more than enough. Also,
no need to allocate it on the heap, having it on the stack works perfectly.
- Reuse isAudio and isVideo instead of re-implementing them
- In IsImage, return early is the mimetype is enough, instead of systematically
lowercasing the url as well.
- Extract the common parts of the two ProxifyAbsoluteURL implementations into a
private function, make the code smaller and clearer.
- Fix a logic error where `A && B || C` was used instead of `A && (B || C)
Instead of using an ugly (and incomplete) regex, let's use a simple for-loop to
parse ISO8601 dates, and make it explicit that we're only supporting a subset
of the spec, as we only care about youtube video durations.
There is no need to use a mutex to check the length of the proxies list,
as it's read-only during the whole lifetime of a ProxyRotator structure.
Moreover, it's a bit clearer to explicitly wrap the 2 lines of mutex-needing
operations between a Lock/Unlock instead of using a defer.
Instead of having a switch-case returning a function to be executed, it's
simpler/faster to have a single function containing a switch-case. It also
allows to group languages with identical plural form in a single
implementation, and remove the "default" guard value, as switch-case already
have a `default:` case.
- Grow the underlying buffer of SanitizeHTML's strings.Builder to 3/4 of the
raw HTML from the start, to reduce the amount of iterative allocations. This
number is a complete guesstimation, but it sounds reasonable to me.
- Add a `absoluteURLParsedBase` function to avoid parsing baseURL over and over.
When we're only interested in the length of contained Text, there is no need to
materialize it fully to then call len() on the result: we can simply iterate
over the text element and sum their length instead.
- There is no need to materialize all the content of a given Node when we can
simply compute its length directly, saving a lot of memory, on the order of
several megabytes on my instance, with peaks at a couple of dozen.
- One might object to the usage of a recursive construct, but this is a direct
port of goquery's Text method, so this change doesn't make anything worse.
- The computation of linkLength can be similarly computed, but this can go in
another commit, as it's a bit trickier, since we need to get the length of
every Node that has a `a` Node as parent, without iterating on the whole
parent chain every time.
There is no need to send the whole title and content to have them truncated on
postgresql's side when we can do this client-side. This should save some
memory on the database's side, as well as some bandwidth
if it's located on another server. And it makes the SQL queries a tad more
readable as well.
Before
```console
$ go test -bench=.
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/readability
BenchmarkExtractContent-8 34 86102474 ns/op
BenchmarkGetWeight-8 10573 103045 ns/op
PASS
ok miniflux.app/v2/internal/reader/readability 5.409s
```
After
```console
$ go test -bench=.
goos: linux
goarch: arm64
pkg: miniflux.app/v2/internal/reader/readability
BenchmarkExtractContent-8 56 83130924 ns/op
BenchmarkGetWeight-8 246541 5241 ns/op
PASS
ok miniflux.app/v2/internal/reader/readability 6.026s
```
This should make ProcessFeedEntries marginally faster, while saving
some memory.
- Use an array of strings instead of a regex, like done in ef13756b1a7a7ba30fd34174a5367381fd8b4849
- Extract the `shouldRemove` function from `removeUnlikelyCandidates`, as there
is no reason to have it there instead of being a proper standalone function.
- Improve a condition, where the goquery selection would have its `id`
attribute left unchecked if a `class` one was present, regardless of if
`class` was a candidate to removal or not.
- Add some comments
This has close to no impact for now, as our slog.Debug/Info/... are leaking
their parameters to the heap, but using proper typing instead of Any allows
to skip some reflection-based computation, making things marginally faster,
and removing the corresponding heap leak.
This change enables Miniflux to serve TLS over Unix domain sockets.
If `CERT_FILE` and `KEY_FILE` are configured, Unix socket listeners
specified via `LISTEN_ADDR` will now automatically start with TLS enabled,
using the provided certificates. This uses the existing `http.Server.ServeTLS`
method.
If no certificates are provided, Unix socket listeners will continue to
operate as plain, non-TLS sockets.
This change implements the ability to specify multiple listen addresses.
This allows the application to listen on different interfaces or ports simultaneously,
or a combination of IP addresses and Unix sockets.
Closes#3343
- Use proper variable names for `key=value` strings parts
- Explicitly assign false to the `match` boolean
- Use an explicit `len(parts) == 2` assertion to help the compiler remove
`isSliceInBounds` calls.
- Refactor identical code into a containsRegexPattern function.
- Early exit when parsing the first date fails when using the `Between`
operator, instead of trying to parse the second one.
As youtubeVideoID is assigned to getVideoIDFromYouTubeURL(entry.URL),
there is no need to call the latter again when we can simly use youtubeVideoID
instead.
There is no need to use SHA256 everywhere, especially on small inputs where we
don't care about its cryptographic properties. We're using FNV as it's the
faster available hash in go's standard library, and we're picking its "a"
version as it's slightly better avalanche characteristics, which are
relevant for small inputs.
This commit has the side-effect of invalidating all favicons saved in the
database, which is desirable to benefit from the resize process implemented in
777d0dd2, as it didn't apply retro-actively.
We're also making use of hex.EncodeToString instead of fmt.Sprintf, as it's
marginally faster.
Note that we can't change the usage of sha256 for feed.Hash as it's used to
deduplicate entries in the database.
Go 1.24 provides the helpful rand.Text() function, returning a base32-encoded
string containing at least 128 bits of randomness. We should make use of it
everywhere it makes sense to do so, if only to not having to think about much
entropy do we need for each cases, and just trust the go crypto team.
Also, rand.Read() can't fail, so no need to check its return value:
https://pkg.go.dev/crypto/rand#Read This behaviour is consistent with go's
standard library itself.
- TLS 1.2 is used as MinVersion by default
- With regard to CipherSuites, in Go 1.22 RSA key exchange based cipher suites
were removed from the default list, and in Go 1.23 3DES cipher suites were
removed as well. Ciphers for TLS1.3 aren't configurable.
- No need to specify CurveP25, as the servers will likely disable the weird
ones like CurveP384 and CurveP521. Removing the explicit specification also
enables the post-quantum X25519MLKEM768, wow!
I trust the go team to make better choices on the long term than us keeping
miniflux up to date with the latest TLS trend.
- Factorize some conditions
- Remove useless `default` case and move the return at the end of the functions
- Use strings.CutPrefix instead of strings.HasPrefix + strings.TrimPrefix
- Use switch-case constructs instead of slices.Contains, as this reduces the
complexity of the functions and allows them to be inlined, as well as helping
the compiler to optimize them, as it sucks at interprocedural optimizations.
The previous regex was using the [ABC..D]*[ABC] pattern, resulting in a lot of
backtracking. The new regex is stopping the matching at the first space or end
of text (and removes the trailing `.` should one be present).
The backtracking was taking around 50% of the CPU time spent in atom.Parse
Previously, url.Parse(baseUrl) was called on every self-closing tags, and on
most opening tags, accounting for around 15% of the CPU time spent in
processor.ProcessFeedEntries
The `sanitizer.StripTags` function is calling `html.NewTokenizer`, which is
allocating a 4096 bytes buffer on the heap, as well a running a complex state
machine to tokenize html. There is no need to do all of this for empty strings.
This commit also fixes a TrimSpace/StripTags call inversion.
A bit more than 10% of processor.ProcessFeedEntries' CPU time is spent in
urlcleaner.RemoveTrackingParameters, specifically calling url.Parse, so let's
extract this operation outside of it, and do it once before calling
urlcleaner.RemoveTrackingParameters multiple times.
Co-authored-by: Frédéric Guillot <f@miniflux.net>
Calls to urllib.AbsoluteURL take a bit less than 10% of the time spent in
parser.ParseFeed, completely parsing an url only to check if it's absolute, and
if not, to make it so.
Checking if it starts with `https://` or `http://` is usually enough to find if
an url is absolute, and if is doesn't, it's always possible to fall back to
urllib.AbsoluteURL.
This also comes with the advantage of reducing heap allocations, as most of the
time spent in urllib.AbsoluteURL is heap-related (de)allocations.
- There is no need to have groups as we're only using this regex for
`MatchString`.
- Since the only place where this regex is used is already calling
strings.ToLower, there is no need to check for `A-Z`.
Display the article's external URL directly in the single entry view.
Rationale: On mobile devices, users couldn't see where a link pointed before tapping it.
Previously, the only way to view the external URL was by hovering - an action not available on touch devices.
Instead of using bytes.Map which is returning a copy of the provided []byte,
use a custom in-place implementation, as the bytes.Map call is taking around
25% of rss.Parse
io.ReadAll is growing the underlying buffer progressively, while
io.Copy is able to allocate it in one go, which is significantly faster.
io.ReadAll is currently accounting for around 10% of the CPU time of rss.Parse
The call to fmt.Sprintf in WithFeedID accounts for more than 20% of the time
spent in GetFeed. Use strconv.Itoa instead, as it's much much faster.
Also change WithCategoryID in the same way, for consistency's sake.
Rationale: Opening links in the current tab is the default browser behavior.
Using `target="_blank"` on external links can lead to accessibility issues and override user preferences. It may also interfere with assistive technologies and expected browser behavior.
To maintain backward compatibility, this option is enabled by default (`true`), which adds `target="_blank"` to links.
- Added new routes: /liveness, /healthz, /readiness, /readyz
- These routes do not take the base path into consideration and are always available at the root of the server
Fixes an issue where upgrading from older versions of Miniflux could fail with the following PostgreSQL error:
```
[FATAL] [Migration v45] pq: index row size 2744 exceeds btree version 4 maximum 2704 for index "entries_feed_url_idx"
```
- Expected format: "tag:google.com,2005:reader/item/00000000148b9369" (hexadecimal string with prefix and padding)
- NetNewsWire uses this format: "tag:google.com,2005:reader/item/2f2" (hexadecimal string with prefix and no padding)
- Reeder uses this format: "000000000000048c" (hexadecimal string without prefix and padding)
- Liferea uses this format: "12345" (decimal string)
Navigator.share returns a promise that's executed in the background, but
unless we await it explicitly, we won't get the exceptions in the
try/catch block.
Prior to this commit, to share an entry, a user has to click on the
share link and then copy the URL they are redirected to. The danger is
that they may right-click and copy the share link without actually
clicking on it, and therefore share a link that, when authenticated,
shares the entry, rather than actually sharing the entry.
Here, we avoid this misinterpretation by making sharing into a POST
request and using a form rather than a link.
Because Miniflux runs as a confidential service, a missing client secret
is a mistake in configuration. An empty client secret appears to be
valid per RFC 6749 (and is in fact the default set by Miniflux!), so we
log a warning.
Steps to reproduce:
1. In /unread, open a feed's settings in a new tab. The feed must have unread entries in /unread.
2. In the new tab/window, delete the feed.
3. Without refreshing, mark an entry from the now-deleted feed as read.
4. Result: The total unread count in the UI header switches to NaN.
The [strings.Fields](https://pkg.go.dev/strings#Fields) considers `'\t', '\n',
'\v', '\f', '\r', ' ', U+0085 (NEL), U+00A0 (NBSP).` as spaces, so no need to
remove them beforehand.
This is a continuation of f2f60a8f73
Currently, removing a feed from `/category/{id}/feeds` redirects incorrectly to `/feeds`. This change fixes it so that
removing a feed will now correctly redirect to `/category/{id}/feeds`. Removing a feed from `/feeds` is unaffected and
will work as it does currently.
To fix this, a new UI endpoint `/category/{categoryID}/feed/{feedID}/remove` is added and a corresponding handler method
to validate and perform the removal from DB.
- Mark a method as `static`
- use `Math.sqrt` instead of `Math.pow(…, 0.5)`
- Use `Math.sign` instead of a condition on the sign
- Inline some used-once variables
- Reduce the scope of some variables
According to https://caniuse.com/?search=passive,
all browsers released after 2016 do support passive event listeners,
so no need to check for its presence.
- Use `….classList.toggle` instead of `….classList.add`/`….classList.remove` in a condition
- Replace a `function()` with a `() =>`
- Use `Math.min` instead of a handwritten condition
Miniflux can be build with `go build -tags=sqlite` to test this. Note that
while it builds, it will fail at runtime, as some of the SQL used in miniflux is
postgresql-specific.
The implementation is equivalent to
`cases.Title(language.English).String(strings.ToLower(…))`,
and this is the only place in miniflux where
"golang.org/x/text/cases" and "golang.org/x/text/language"
are (directly) used.
This reduces the binary size from 27015590 to
26686112 on my machine.
Kudos to https://gsa.zxilly.dev for making it straightforward to catch things
like this.
The coverage information isn't used anywhere in the CI, so no need to have it
for every OS. As for `-race`, there is no point in using it everywhere, one
time should be enough, especially since it's taking a lot of time on Windows.
- Use chained strings.Contains instead of a regex for
blacklistCandidatesRegexp, as this is a bit faster
- Simplify a Find.Each.Remove to Find.Remove
- Don't concatenate id and class for removeUnlikelyCandidates, as it makes no
sense to match on overlaps. It might also marginally improve performances, as
regex now have to run on two strings separately, instead of both.
- Add a small benchmark
Some websites are using images of O(10kB) when not )O(100kB) for their
favicons. As miniflux only displays them with a 16x16 resolution, let's do our
best to resize them before storing them in the database. This should make
miniflux consume less bandwidth when serving pages, for the joy of mobile users
on a small data plan.
Of course, images that already are 16x16 aren't resized.
No need for brittle regex when matching plain strings or domain names.
This should save some negligible amount of heap memory as well as
tremendously speeding up the matching.
- Replace a completely overkill regex
- Use `.Remove()` instead of a hand-rolled loop
- Use a strings.Builder instead of a bytes.NewBufferString
- Replace a call to Fprintf with string concatenation, as the latter are much
faster
- Remove a superfluous cast
- Delay some computations
- Add some tests
As mentioned in goquery's documentation (https://pkg.go.dev/github.com/PuerkitoBio/goquery#Single):
> By default, Selection.Find and other functions that accept a selector string
to select nodes will use all matches corresponding to that selector. By using
the Matcher returned by Single, at most the first match will be selected.
>
> The one using Single is optimized to be potentially much faster on large documents.
In internal/reader/handler/handler.go:RefreshFeed, there is a call to
store.UserByID pretty early, which is only used for
originalFeed.WithTranslatedErrorMessage(localizedError.Translate(user.Language)
Its only other usage is in processor.ProcessFeedEntries(store, originalFeed,
user, forceRefresh), which is pretty late in RefreshFeed, and only called if
there are new items in the feed. It makes sense to only fetch the user's
language if the error localization function is used.
Calls to `store.UserByID` take around 10% of the CPU time of RefreshFeed in my
profiling.
This commit also makes `processor.ProcessFeedEntries` take a `userID` instead
of a `user`, to make the code a bit more concise.
This should close#2984
While doing some profiling for #2900, I noticed that
`miniflux.app/v2/internal/locale.LoadCatalogMessages` is responsible for more
than 10% of the consumed memory. As most miniflux instances won't have enough
diverse users to use all the available translations at the same time, it
makes sense to load them on demand.
The overhead is a single function call and a check in a map, per call to
translation-related functions.
- Use string concatenation instead of `Sprintf`, as this is much faster, and the
call to `Sprintf` is responsible for 30% of the CPU time of the function
- Anchor the youtube regex, to allow it to bail early, as this also account for
another 30% of the CPU time. It might be worth chaining calls to `TrimPrefix`
and check if the string has been trimmed instead of using a regex, to speed
things up even more, but this needs to be benchmarked properly.
It was added in 2022 by #1513, to support blog.laravel.com, which has
since switched to HTML. The Atom 0.3/1.0, RSS 1.0/2.0, RDF, and JSON formats
don't support markdown in their spec, and any website serving it there should
be considered as buggy and fixed.
This shaves off 2MB from the miniflux binary, which is quite steep for a
feature that nobody is/should be using, and remove a dependency which is always
a good thing.
- Use `token.String()` instead of `html.EscapeString(token.Data)`
- Refactor conditions to highlight their similitude, enabling further
refactoring
This refactoring brings forth at least one bug: `tagStack` is never emptied.
The `isAnchor` function's first parameter was always `a`, instead of being
passed `tagName`. As this function is a single line and was only called in a
single place, it can be inlined.
- Use `[^"]` instead of `.`, to help the regex engine to determine boundaries,
instead of having it bruteforce its way to find them
- Use `+` instead of `*`, as empty rules don't make sense
This function takes around 1.5% of the total CPU time on my instance, and most
of it is spent in `mapassign_faststr` to initialize the `map`. This is replaced
with a switch-case construct, that should be both significantly faster as well
as pretty dull in term of memory consumption.
- Pre-allocate a slice
- Inline a local variable
- Remove a superfluous call to `strings.TrimSpace`
- Simplify some conditions via a switch-case construct
The `pg_timezone_names` view was added in 8.2.
It should be equivalent to the function query.
See: https://pgpedia.info/p/pg_timezone_names.html
This small change allows `miniflux` to run on postgres-compatible
databases like CockroachDB, which don't have this function.
When you download/save proxified media, the original filename is lost. That
information could be retained by passing a header `Content-Disposition: inline;
filename="ORIGNAL_FILENAME.EXT"` when serving the media file. The requested URL
would still be obfuscated, but if the client downloads the file it'll use that
original filename.
Since go-webauthn v0.11.0, the backup eligibility flag is strictly validated, but Miniflux does not store this flag.
This workaround to set the flag based on the parsed response, and avoid "BackupEligible flag inconsistency detected during login validation" error.
See https://github.com/go-webauthn/webauthn/pull/240
When a server returns a 304 response with a strong validator, any other
stored fields must be updated if they are also present in the response.
This behaviour is described in RFC9111, sections 3.2 and 4.3.4.
Shared entry does not link to any user and therefore should not display
any saved progression. Curiously, the progression of a user (the one that shared ?)
was still integrated in the page. This does not make sens regarding the sharing
feature itself. It is also a leak of user personal information onto a public page.
I simply removed the data from the template when the user object is not present.
I tested the change on "regular" entry page, ensuring the save progression feature
still works, and on shared page checking if any error happened in the JavaScript console.
Everything seems in order.
This document outlines how to contribute effectively to Miniflux.
## Philosophy
Miniflux follows a **minimalist philosophy**. The feature set is intentionally kept limited to avoid bloatware. Before contributing, please understand that:
- **Improving existing features takes priority over adding new ones**
- **Quality over quantity** - well-implemented, focused features are preferred
- **Simplicity is key** - complex solutions are discouraged in favor of simple, maintainable code
## Before You Start
### Feature Requests
Before implementing a new feature:
- Check if it aligns with Miniflux's philosophy
- Consider if the feature could be implemented differently to maintain simplicity
- Remember that developing software takes significant time, and this is a volunteer-driven project
- If you need a specific feature, the best approach is to contribute it yourself
### Bug Reports
When reporting bugs:
- Search existing issues first to avoid duplicates
- Provide clear reproduction steps
- Include relevant system information (OS, browser, Miniflux version)
- Include error messages, screenshots, and logs when applicable
- **New bugs or regressions** - reduces software quality
- **Unnecessary dependencies** - conflicts with minimalist approach
- **Performance degradation** - slows down the software
- **Poor-quality code** - hard to maintain
- **Dependent PRs** - creates review complexity
- **Radical UI changes** - disrupts user experience
- **Conflicts with philosophy** - doesn't align with minimalist approach
### Pull Request Template
When creating a pull request, please include:
- **Description:** What does this PR do?
- **Motivation:** Why is this change needed?
- **Testing:** How was this tested?
- **Breaking Changes:** Are there any breaking changes?
- **Related Issues:** Link to any related issues
## Code Style
- Follow Go conventions and best practices
- Use `gofmt` to format your Go code, and `jshint` for JavaScript
- Write clear, descriptive variable and function names
- Include comments for complex logic
- Keep functions small and focused
## Testing
### Unit Tests
- Write unit tests for new functions and methods
- Ensure tests are fast and don't require external dependencies
- Aim for good test coverage
### Integration Tests
- Add integration tests for new API endpoints
- Tests run against a real PostgreSQL database
- Ensure tests clean up after themselves
## Communication
- **Discussions:** Use GitHub Discussions for general questions and community interaction
- **Issues:** Use GitHub issues for bug reports and feature requests
- **Pull Requests:** Use PR comments for code-specific discussions
- **Philosophy Questions:** Refer to the FAQ for common questions about project direction
## Questions?
- Check the [FAQ](https://miniflux.app/faq.html) for common questions
- Review the [development documentation](https://miniflux.app/docs/development.html) and [internationalization guide](https://miniflux.app/docs/i18n.html)
- Look at existing issues and pull requests for examples
- Retrieves original links when feeds are sourced from FeedBurner.
- Opens external links with attributes `rel="noopener noreferrer" referrerpolicy="no-referrer"` for improved security.
- Implements the HTTP header `Referrer-Policy: no-referrer` to prevent referrer leakage.
- Provides a media proxy to avoid tracking and resolve mixed content warnings when using HTTPS.
- Plays YouTube videos via the privacy-focused domain `youtube-nocookie.com`.
- Supports alternative YouTube video players such as [Invidious](https://invidio.us).
- Blocks external JavaScript to prevent tracking and enhance security.
- Sanitizes external content before rendering it.
- Enforces a [Content Security](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) and a [Trusted Types Policy](https://developer.mozilla.org/en-US/docs/Web/API/Trusted_Types_API) to only application JavaScript and blocks inline scripts and styles.
### Bot Protection Bypass Mechanisms
- Optionally disable HTTP/2 to mitigate fingerprinting.
- Allows configuration of a custom user agent.
- Supports adding custom cookies for specific use cases.
- Enables the use of proxies for enhanced privacy or bypassing restrictions.
### Content Manipulation
- Fetches the original article and extracts only the relevant content using a local Readability parser.
- Allows custom scraper rules based on <abbr title="Cascading Style Sheets">CSS</abbr> selectors.
- Supports custom rewriting rules for content manipulation.
- Provides a regex filter to include or exclude articles based on specific patterns.
- Optionally permits self-signed or invalid certificates (disabled by default).
- Scrapes YouTube's website to retrieve video duration as read time or uses the YouTube API (disabled by default).
### User Interface
- Optimized stylesheet for readability.
- Responsive design that adapts seamlessly to desktop, tablet, and mobile devices.
- Minimalistic and distraction-free user interface.
- No requirement to download an app from Apple App Store or Google Play Store.
- Can be added directly to the home screen for quick access.
- Supports a wide range of keyboard shortcuts for efficient navigation.
- Optional touch gesture support for navigation on mobile devices.
- Custom stylesheets and JavaScript to personalize the user interface to your preferences.
- Themes:
- Light (Sans-Serif)
- Light (Serif)
- Dark (Sans-Serif)
- Dark (Serif)
- System (Sans-Serif) – Automatically switches between Dark and Light themes based on system preferences.
- Bookmarklet for subscribing to websites directly from any web browser.
- Webhooks for real-time notifications or custom integrations.
- Compatibility with existing mobile applications using the Fever or Google Reader API.
- REST API with client libraries available in [Go](https://github.com/miniflux/v2/tree/main/client) and [Python](https://github.com/miniflux/python-client).
printErrorAndExit(errors.New("DISABLE_LOCAL_AUTH is enabled but neither OAUTH2_PROVIDER nor AUTH_PROXY_HEADER is not set. Please enable at least one authentication source"))
returnerrors.New("DISABLE_LOCAL_AUTH is enabled but neither OAUTH2_PROVIDER nor AUTH_PROXY_HEADER is set. Please enable at least one authentication source")
This document describes the Fever-compatible API implemented by the `internal/fever` package in this repository.
## Endpoint
- Path: `BASE_URL/fever/`
- Methods: not restricted by the router; read requests are typically sent as `GET`, write requests should be sent as `POST`
- Response format: JSON only
- Reported API version: `3`
## Authentication
Fever authentication is enabled per user from the Miniflux integrations page.
-`Fever Username` and `Fever Password` are configured in Miniflux
- Miniflux stores the Fever token as the MD5 hash of `username:password`
- Clients authenticate by sending that token as the `api_key` parameter
- Token lookup is case-insensitive
Example:
```text
api_key = md5("fever_username:fever_password")
```
Example shell command:
```bash
printf'%s''fever_username:fever_password'| md5sum
```
Authentication failure does not return HTTP 401. The middleware returns HTTP 200 with:
```json
{
"api_version":3,
"auth":0
}
```
On successful authentication, every response includes:
-`api_version`: always `3`
-`auth`: always `1`
-`last_refreshed_on_time`: current server Unix timestamp at response time
## Dispatch Rules
The handler selects the first matching operation in this order:
1.`groups`
2.`feeds`
3.`favicons`
4.`unread_item_ids`
5.`saved_item_ids`
6.`items`
7.`mark=item`
8.`mark=feed`
9.`mark=group`
If no selector is provided, the server returns the base authenticated response only.
For read operations, the selector must be present in the query string. For write operations, `mark`, `as`, `id`, and `before` are read from request form values, so they may come from the query string or a form body.
## Read Operations
### `?groups`
Returns:
-`groups`: list of categories
-`feeds_groups`: mapping of category IDs to feed IDs
Response shape:
```json
{
"api_version":3,
"auth":1,
"last_refreshed_on_time":1710000000,
"groups":[
{
"id":1,
"title":"All"
}
],
"feeds_groups":[
{
"group_id":1,
"feed_ids":"10,11"
}
]
}
```
Notes:
-`groups` are Miniflux categories
-`feeds_groups.feed_ids` is a comma-separated string
- categories with no feeds are returned in `groups` but have no `feeds_groups` entry
### `?feeds`
Returns:
-`feeds`: list of feeds
-`feeds_groups`: mapping of category IDs to feed IDs
Feed fields:
-`id`
-`favicon_id`
-`title`
-`url`
-`site_url`
-`is_spark`
-`last_updated_on_time`
Notes:
-`favicon_id` is `0` when the feed has no icon
-`is_spark` is always `0` in this implementation
-`last_updated_on_time` is the feed check time as a Unix timestamp
### `?favicons`
Returns:
-`favicons`: list of favicon objects
Favicon fields:
-`id`
-`data`
Notes:
-`data` is a data URL such as `image/png;base64,...`
### `?unread_item_ids`
Returns:
-`unread_item_ids`: comma-separated list of unread entry IDs
Response shape:
```json
{
"api_version":3,
"auth":1,
"last_refreshed_on_time":1710000000,
"unread_item_ids":"100,101,102"
}
```
### `?saved_item_ids`
Returns:
-`saved_item_ids`: comma-separated list of starred entry IDs
### `?items`
Returns:
-`items`: list of entries
-`total_items`: total number of non-removed entries for the user
Item fields:
-`id`
-`feed_id`
-`title`
-`author`
-`html`
-`url`
-`is_saved`
-`is_read`
-`created_on_time`
The implementation always excludes entries whose status is `removed`.
#### Pagination and filtering
The handler applies a fixed limit of 50 items.
Supported parameters:
-`since_id`: when greater than `0`, returns entries with `id > since_id`, ordered by `id ASC`
-`max_id`: when equal to `0`, returns the most recent entries ordered by `id DESC`; when greater than `0`, returns entries with `id < max_id`, ordered by `id DESC`
-`with_ids`: comma-separated list of entry IDs to fetch
Selector precedence inside `?items` is:
1.`since_id`
2.`max_id`
3.`with_ids`
4. no item filter
Notes:
-`with_ids` does not enforce the 50-ID maximum mentioned in older Fever documentation
- invalid `with_ids` members are parsed as `0` and do not match normal entries
- when `items` is requested without `since_id`, `max_id`, or `with_ids`, the code applies no explicit `ORDER BY`, so result ordering is not guaranteed by SQL
-`html` is returned after Miniflux content rewriting and may include media-proxy-rewritten URLs
Example:
```json
{
"api_version":3,
"auth":1,
"last_refreshed_on_time":1710000000,
"total_items":245,
"items":[
{
"id":100,
"feed_id":10,
"title":"Example entry",
"author":"Author",
"html":"<p>Content</p>",
"url":"https://example.org/post",
"is_saved":0,
"is_read":1,
"created_on_time":1709990000
}
]
}
```
## Write Operations
Normal successful write operations return the base authenticated response:
```json
{
"api_version":3,
"auth":1,
"last_refreshed_on_time":1710000000
}
```
### `mark=item`
Parameters:
-`mark=item`
-`id=<entry_id>`
-`as=read|unread|saved|unsaved`
Behavior:
-`as=read`: marks the entry as read
-`as=unread`: marks the entry as unread
-`as=saved`: toggles the starred flag
-`as=unsaved`: toggles the starred flag
Important:
-`saved` and `unsaved` both call the same toggle operation
- sending `as=saved` twice will save, then unsave
- sending `as=unsaved` twice will unsave, then save
- if `id <= 0`, the handler returns without writing a response body
- if the entry does not exist or is already removed, the server returns the base response without an error
### `mark=feed`
Parameters:
-`mark=feed`
-`as=read`
-`id=<feed_id>`
-`before=<unix_timestamp>`
Behavior:
- marks unread entries in the feed as read when `published_at < before`
- the update runs asynchronously in a goroutine after the response is returned
Notes:
- if `id <= 0`, the handler returns without writing a response body
- if `before` is missing or invalid, it is treated as Unix time `0`, which usually means nothing is marked as read
### `mark=group`
Parameters:
-`mark=group`
-`as=read`
-`id=<group_id>`
-`before=<unix_timestamp>`
Behavior:
-`id=0`: marks all unread entries as read, ignoring `before`
-`id>0`: marks unread entries in the matching category as read when `published_at < before`
- the update runs asynchronously in a goroutine after the response is returned
Notes:
- group IDs map to Miniflux category IDs
- if `id < 0`, the handler returns without writing a response body
- if `before` is missing or invalid for `id>0`, it is treated as Unix time `0`, which usually means nothing is marked as read
## Error Handling
Authentication failures:
- HTTP status: `200`
- body: `{"api_version":3,"auth":0}`
Internal errors:
- HTTP status: `500`
- body:
```json
{
"error_message":"..."
}
```
## Differences From Generic Fever Documentation
This implementation is Fever-compatible, but it does not match every detail of historical Fever API docs.
- Responses are always JSON; `api=xml` is mentioned in code comments but is not implemented
-`api_version` is `3`
-`last_refreshed_on_time` is set to the current response time, not the timestamp of the most recently refreshed feed
- the `Kindling` and `Sparks` super groups are not returned
-`feeds[].is_spark` is always `0`
- item ordering without explicit pagination parameters is unspecified
-`as=saved` and `as=unsaved` toggle the saved flag instead of setting it absolutely
This document describes the Google Reader compatible API implemented by the `internal/googlereader` package in this repository.
Miniflux implements a compatibility subset intended for existing Google Reader clients. It is not a full reimplementation of the historical Google Reader API, and several behaviors are intentionally narrower or implementation-specific.
-`BASE_URL` includes the Miniflux root URL and any configured `BasePath`
- Response format:
-`ClientLogin`: plain text by default, JSON when `output=json`
- most API reads: JSON
- most API writes: plain text `OK`
## Enabling the API
Google Reader compatibility is configured per user from the Miniflux integrations page.
-`Google Reader API` must be enabled
-`Google Reader Username` must be unique across all Miniflux users
-`Google Reader Password` is stored as a bcrypt hash
The Google Reader username and password are separate integration credentials. They are not the Miniflux account password.
## Authentication
### `POST /accounts/ClientLogin`
This endpoint exchanges the configured Google Reader username and password for an auth token.
Form parameters:
-`Email`: Google Reader username
-`Passwd`: Google Reader password
-`output`: optional, set to `json` for a JSON response
Successful responses:
- default: plain text
- with `output=json`: JSON
Example plain-text response:
```text
SID=readeruser/0123456789abcdef...
LSID=readeruser/0123456789abcdef...
Auth=readeruser/0123456789abcdef...
```
Example JSON response:
```json
{
"SID":"readeruser/0123456789abcdef...",
"LSID":"readeruser/0123456789abcdef...",
"Auth":"readeruser/0123456789abcdef..."
}
```
On authentication failure, `ClientLogin` returns HTTP `401` with the normal JSON error body:
```json
{
"error_message":"access unauthorized"
}
```
### Auth token format
The token format is:
```text
<googlereader_username>/<hex_digest>
```
The digest is generated server-side from:
- the Google Reader username
- the stored bcrypt hash of the Google Reader password
Specifically, the code computes an HMAC-SHA256 digest of an empty message using the key:
```text
googlereader_username + bcrypt_hash
```
Because the bcrypt hash is only known to the server, clients should not try to precompute the token. Use `ClientLogin` or `GET /reader/api/0/token`.
### Authenticating API calls
Miniflux uses different auth mechanisms for `GET` and `POST` requests:
-`GET` requests must send the header `Authorization: GoogleLogin auth=<token>`
-`POST` requests are authenticated with `T=<token>` read from the parsed form values
Notes:
- the auth scheme must be exactly `GoogleLogin`
- the auth field name must be exactly lowercase `auth`
- for `POST`, `T` may come from the URL query or the form body because the server reads merged form values
-`POST` requests do not accept the token from the `Authorization` header
-`GET` requests do not accept the token from the query string
### `GET /reader/api/0/token`
This endpoint requires normal `GET` authentication and returns the same token as plain text.
Many Google Reader clients use this as the edit token for subsequent write requests. In Miniflux, the edit token and auth token are the same value.
### Authentication failure on `/reader/api/0/*`
When API authentication fails under `/reader/api/0`, Miniflux returns:
- HTTP `401`
- header `X-Reader-Google-Bad-Token: true`
- content type `text/plain; charset=utf-8`
- body `Unauthorized`
This is different from `ClientLogin`, which returns a JSON `401`.
## Identifier formats
### Stream IDs
The implementation recognizes these stream forms:
- built-in streams:
-`user/-/state/com.google/read`
-`user/-/state/com.google/starred`
-`user/-/state/com.google/reading-list`
-`user/-/state/com.google/kept-unread`
-`user/-/state/com.google/broadcast`
-`user/-/state/com.google/broadcast-friends`
-`user/-/state/com.google/like`
- user-specific equivalents:
-`user/<user_id>/state/com.google/...`
- label streams:
-`user/-/label/<name>`
-`user/<user_id>/label/<name>`
- feed streams:
-`feed/<value>`
Important feed stream difference:
- read APIs usually emit `feed/<numeric_feed_id>`
-`subscription/edit` with `ac=subscribe` expects `feed/<absolute_feed_url>`
-`subscription/edit` with `ac=edit` or `ac=unsubscribe` expects `feed/<numeric_feed_id>`
So `feed/<...>` is not a single stable identifier format across all endpoints.
### Item IDs
`edit-tag` and `stream/items/contents` accept repeated `i` parameters in all of these formats:
- long Google Reader form: `tag:google.com,2005:reader/item/00000000148b9369`
- short prefixed hexadecimal form: `tag:google.com,2005:reader/item/2f2`
- bare 16-character hexadecimal form: `000000000000048c`
- decimal entry ID: `12345`
Responses use different forms depending on endpoint:
-`stream/items/ids` returns decimal IDs as strings
-`stream/items/contents` returns long-form Google Reader item IDs
## Common response conventions
JSON errors use this shape:
```json
{
"error_message":"..."
}
```
Plain-text success responses from write endpoints are usually:
```text
OK
```
## POST parameter parsing
Most `POST` handlers call `ParseForm()` and read from `r.Form`, so parameters may be supplied either in the query string or in a standard form body.
Important exception:
-`POST /reader/api/0/edit-tag` reads `a` and `r` from `r.PostForm`, so those tag lists must come from the request body
Because `GET` auth comes only from the `Authorization` header, query parameters never authenticate `GET` requests even when other parameters are read from the query string.
## Endpoint reference
### `GET /reader/api/0/user-info`
Returns JSON only. No `output=json` parameter is required.
Response fields:
-`userId`: Miniflux user ID as a string
-`userName`: Miniflux username
-`userProfileId`: same value as `userId`
-`userEmail`: same value as `userName`
Example:
```json
{
"userId":"1",
"userName":"demo",
"userProfileId":"1",
"userEmail":"demo"
}
```
### `GET /reader/api/0/tag/list?output=json`
Returns the starred state and user labels.
Notes:
-`output=json` is required
- only labels and the starred state are returned
- built-in states such as `read` and `reading-list` are not listed here
t.Errorf("Expected %q for RemoteAddr %q, got %q",scenario.expected,scenario.ip,result)
}
}
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Kelly Norton