feat(filter): expand CEL filter surface with startsWith/endsWith, matches(), and all()

Let users write three more CEL constructs in the filter field, each compiled to
SQL across SQLite/MySQL/Postgres:

- Scalar startsWith()/endsWith() on content/filename/mime_type (case-insensitive)
- matches() regex: PG ~, MySQL/SQLite REGEXP (Go-backed SQLite fn), validated at
  compile time via cel.ValidateRegexLiterals()
- all() comprehension over tags via per-element subqueries, non-empty required

Also: contains() now escapes LIKE metacharacters (%, _, \); cross-dialect render
tests plus behavioral tests; cel-go bumped to v0.28.1; new operators surfaced in
the frontend shortcut guide.
This commit is contained in:
boojack
2026-06-15 23:22:28 +08:00
parent 817561df8f
commit f0e4a5624f
16 changed files with 1835 additions and 34 deletions
@@ -0,0 +1,887 @@
# CEL Filter Surface Expansion — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let users write three more CEL constructs in the `filter` field — scalar `startsWith()`/`endsWith()` (case-insensitive), `matches(regex)`, and `all()` over tags (non-empty) — each compiled to SQL across SQLite/MySQL/Postgres.
**Architecture:** The `internal/filter` engine parses CEL with `cel-go`, walks the AST into a dialect-agnostic IR (`ir.go`), and renders dialect SQL (`render.go`). cel-go never evaluates — every feature must become a SQL `WHERE` fragment. We add IR nodes + parser recognition + per-dialect rendering, and register a Go-backed `REGEXP` function for SQLite (which has no built-in one).
**Tech Stack:** Go, `github.com/google/cel-go v0.28.0`, `modernc.org/sqlite` (pure-Go), `github.com/stretchr/testify/require`.
**Spec:** `docs/superpowers/specs/2026-06-15-cel-filter-surface-expansion-design.md`
---
## File structure
| File | Change | Responsibility |
|------|--------|----------------|
| `internal/filter/ir.go` | Modify | Replace `ContainsCondition` with `TextMatchCondition`; add `RegexCondition`; add `ComprehensionAll` kind |
| `internal/filter/parser.go` | Modify | Recognize top-level `contains`/`startsWith`/`endsWith`/`matches`; accept `all()` comprehension |
| `internal/filter/render.go` | Modify | Render text-match (LIKE), regex, and `all()` per-element subqueries; shared `foldedLike`/`likePattern`/`escapeLikeLiteral` helpers |
| `internal/filter/schema.go` | Modify | Add `cel.ValidateRegexLiterals()` validator; enable text matching on attachment `mime_type` |
| `internal/filter/engine_test.go` | Modify | Compile-level accept/reject unit tests |
| `store/db/sqlite/functions.go` | Modify | Register a Go-backed `regexp(pattern, value)` scalar function with a compiled-pattern cache |
| `store/db/sqlite/sqlite.go` | Modify | Call `ensureRegexpRegistered()` in `NewDB` |
| `store/test/memo_filter_test.go` | Modify | Behavioral tests for the new memo filters |
| `store/test/attachment_filter_test.go` | Modify | Behavioral tests for `filename`/`mime_type` |
| `internal/filter/README.md` | Modify | Document new syntax + regex cross-dialect caveat |
**Key design choices locked in:**
- The existing `Field.SupportsContains` flag is **reused** as the gate for *all* text-matching ops (`contains`/`startsWith`/`endsWith`/`matches`) — no rename, lower risk. We just enable it on `mime_type`.
- New scalar `startsWith`/`endsWith`/`contains` are **case-insensitive** (reuse the existing `memos_unicode_lower` / `ILIKE` machinery). `matches()` and `==` are case-sensitive.
- `all()` over a memo with zero tags does **not** match (non-empty guard).
- LIKE patterns escape `%` `_` `\`; only SQLite needs an explicit `ESCAPE '\'` clause (Postgres/MySQL default the escape char to backslash, and patterns are passed as bound parameters so no SQL-literal backslash hazard).
**Suggested task order** lands the two cheap features (text-match refactor, scalar prefix/suffix, regex) before the heavy `all()` work, giving a natural stop point. `all()` (Task 5) is the largest piece and could be deferred to a follow-up if needed.
---
### Task 1: Refactor `ContainsCondition` → `TextMatchCondition` (+ LIKE escaping)
Foundation refactor. No new user-facing behavior except that LIKE metacharacters in `contains()` values are now treated literally. Existing tests must stay green.
**Files:**
- Modify: `internal/filter/ir.go` (replace `ContainsCondition`)
- Modify: `internal/filter/parser.go` (`buildContainsCondition` → shared builder)
- Modify: `internal/filter/render.go` (`renderContainsCondition``renderTextMatch` + helpers)
- [ ] **Step 1: Add a failing escaping test** in `internal/filter/engine_test.go`
```go
func TestCompileContainsEscapesLikeWildcards(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
stmt, err := engine.CompileToStatement(context.Background(), `content.contains("50%_off")`, RenderOptions{Dialect: DialectSQLite})
require.NoError(t, err)
// The % and _ in the value must be escaped so they are matched literally,
// and SQLite needs an explicit ESCAPE clause.
require.Contains(t, stmt.SQL, `ESCAPE '\'`)
require.Equal(t, []any{`%50\%\_off%`}, stmt.Args)
}
```
- [ ] **Step 2: Run it to verify it fails**
Run: `go test ./internal/filter/ -run TestCompileContainsEscapesLikeWildcards -v`
Expected: FAIL (current renderer emits `%50%_off%` with no `ESCAPE`).
- [ ] **Step 3: Replace `ContainsCondition` in `internal/filter/ir.go`**
Delete the `ContainsCondition` struct + its `isCondition()` (lines ~76-82) and add:
```go
// TextMatchMode enumerates LIKE-based string match modes.
type TextMatchMode string
const (
TextMatchContains TextMatchMode = "contains"
TextMatchPrefix TextMatchMode = "prefix"
TextMatchSuffix TextMatchMode = "suffix"
)
// TextMatchCondition models a case-insensitive LIKE match on a scalar string field
// (content.contains/startsWith/endsWith).
type TextMatchCondition struct {
Field string
Mode TextMatchMode
Value string
}
func (*TextMatchCondition) isCondition() {}
```
- [ ] **Step 4: Update the parser in `internal/filter/parser.go`**
In `buildCallCondition`, replace the `case "contains":` line with:
```go
case "contains":
return buildTextMatchCondition(call, schema, TextMatchContains)
```
Delete `buildContainsCondition` (lines ~196-227) and add:
```go
func buildTextMatchCondition(call *exprv1.Expr_Call, schema Schema, mode TextMatchMode) (Condition, error) {
if call.Target == nil {
return nil, errors.New("text match requires a target")
}
targetName, err := getIdentName(call.Target)
if err != nil {
return nil, err
}
field, ok := schema.Field(targetName)
if !ok {
return nil, errors.Errorf("unknown identifier %q", targetName)
}
if !field.SupportsContains {
return nil, errors.Errorf("identifier %q does not support text matching", targetName)
}
if len(call.Args) != 1 {
return nil, errors.New("text match expects exactly one argument")
}
value, err := getConstValue(call.Args[0])
if err != nil {
return nil, errors.Wrap(err, "text match only supports literal arguments")
}
str, ok := value.(string)
if !ok {
return nil, errors.New("text match argument must be a string")
}
return &TextMatchCondition{Field: targetName, Mode: mode, Value: str}, nil
}
```
- [ ] **Step 5: Update the renderer in `internal/filter/render.go`**
In `renderCondition`, replace `case *ContainsCondition:` / `return r.renderContainsCondition(c)` with:
```go
case *TextMatchCondition:
return r.renderTextMatch(c)
```
Delete `renderContainsCondition` (lines ~449-469) and add:
```go
func (r *renderer) renderTextMatch(cond *TextMatchCondition) (renderResult, error) {
field, ok := r.schema.Field(cond.Field)
if !ok {
return renderResult{}, errors.Errorf("unknown field %q", cond.Field)
}
column := field.columnExpr(r.dialect)
pattern := likePattern(cond.Mode, cond.Value)
return renderResult{sql: r.foldedLike(column, pattern)}, nil
}
// foldedLike renders a case-insensitive LIKE comparison of colExpr against a
// (already metacharacter-escaped) pattern, using each dialect's case-folding.
func (r *renderer) foldedLike(colExpr, pattern string) string {
switch r.dialect {
case DialectSQLite:
// memos_unicode_lower gives Unicode-aware folding; ESCAPE '\' is required
// because SQLite has no default LIKE escape character.
return fmt.Sprintf(`memos_unicode_lower(%s) LIKE memos_unicode_lower(%s) ESCAPE '\'`, colExpr, r.addArg(pattern))
case DialectPostgres:
// ILIKE is case-insensitive; backslash is the default escape character.
return fmt.Sprintf("%s ILIKE %s", colExpr, r.addArg(pattern))
default: // MySQL: default collation is case-insensitive; backslash is the default escape.
return fmt.Sprintf("%s LIKE %s", colExpr, r.addArg(pattern))
}
}
// likePattern escapes LIKE metacharacters in value and wraps it for the mode.
func likePattern(mode TextMatchMode, value string) string {
escaped := escapeLikeLiteral(value)
switch mode {
case TextMatchPrefix:
return escaped + "%"
case TextMatchSuffix:
return "%" + escaped
default:
return "%" + escaped + "%"
}
}
// escapeLikeLiteral escapes the LIKE metacharacters \, %, and _ so user input
// is matched literally. Backslash is the escape character on all three dialects.
func escapeLikeLiteral(s string) string {
return strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(s)
}
```
- [ ] **Step 6: Run the new test + existing suites to verify green**
Run: `go test ./internal/filter/ -v`
Expected: PASS (including `TestCompileContainsEscapesLikeWildcards`).
Run: `go test ./store/test/ -run TestMemoFilterContent -v`
Expected: PASS (existing `contains` behavioral tests, including special-characters/unicode, still pass).
- [ ] **Step 7: Commit**
```bash
git add internal/filter/ir.go internal/filter/parser.go internal/filter/render.go internal/filter/engine_test.go
git commit -m "refactor(filter): unify string matching into TextMatchCondition with LIKE escaping
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
### Task 2: Scalar `startsWith()` / `endsWith()`
Wire the new prefix/suffix modes through the parser and enable text matching on attachment `mime_type`. Rendering already exists from Task 1.
**Files:**
- Modify: `internal/filter/parser.go` (add `startsWith`/`endsWith` cases)
- Modify: `internal/filter/schema.go` (enable `SupportsContains` on `mime_type`)
- Modify: `store/test/memo_filter_test.go`, `store/test/attachment_filter_test.go` (behavioral tests)
- Modify: `internal/filter/engine_test.go` (reject on unsupported field)
- [ ] **Step 1: Add failing behavioral tests** in `store/test/memo_filter_test.go`
```go
func TestMemoFilterContentStartsWith(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-todo", tc.User.ID).Content("TODO: buy milk"))
tc.CreateMemo(NewMemoBuilder("memo-done", tc.User.ID).Content("Done with milk"))
// Prefix match, case-insensitive (consistent with contains()).
memos := tc.ListWithFilter(`content.startsWith("todo")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-todo", memos[0].UID)
memos = tc.ListWithFilter(`content.startsWith("nope")`)
require.Len(t, memos, 0)
}
func TestMemoFilterContentEndsWith(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-md", tc.User.ID).Content("notes.md"))
tc.CreateMemo(NewMemoBuilder("memo-txt", tc.User.ID).Content("notes.txt"))
memos := tc.ListWithFilter(`content.endsWith(".md")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-md", memos[0].UID)
}
```
And in `store/test/attachment_filter_test.go`:
```go
func TestAttachmentFilterFilenameStartsWith(t *testing.T) {
t.Parallel()
tc := NewAttachmentFilterTestContextWithUser(t)
defer tc.Close()
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("invoice-2026.pdf").MimeType("application/pdf"))
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("photo.png").MimeType("image/png"))
got := tc.ListWithFilter(`filename.startsWith("invoice")`)
require.Len(t, got, 1)
require.Equal(t, "invoice-2026.pdf", got[0].Filename)
// mime_type prefix matching (newly enabled).
got = tc.ListWithFilter(`mime_type.startsWith("image/")`)
require.Len(t, got, 1)
require.Equal(t, "photo.png", got[0].Filename)
}
```
- [ ] **Step 2: Run to verify they fail**
Run: `go test ./store/test/ -run 'TestMemoFilterContentStartsWith|TestMemoFilterContentEndsWith|TestAttachmentFilterFilenameStartsWith' -v`
Expected: FAIL — `startsWith` hits `buildCallCondition`'s default branch ("unsupported call expression"), and `mime_type` is not yet text-matchable.
- [ ] **Step 3: Add parser cases** in `internal/filter/parser.go` `buildCallCondition`
Immediately after the `case "contains":` line, add:
```go
case "startsWith":
return buildTextMatchCondition(call, schema, TextMatchPrefix)
case "endsWith":
return buildTextMatchCondition(call, schema, TextMatchSuffix)
```
- [ ] **Step 4: Enable text matching on `mime_type`** in `internal/filter/schema.go`
In `NewAttachmentSchema`, add `SupportsContains: true` to the `mime_type` field entry:
```go
"mime_type": {
Name: "mime_type",
Kind: FieldKindScalar,
Type: FieldTypeString,
Column: Column{Table: "attachment", Name: "type"},
SupportsContains: true,
Expressions: map[DialectName]string{},
},
```
- [ ] **Step 5: Add a compile-reject unit test** in `internal/filter/engine_test.go`
```go
func TestCompileRejectsStartsWithOnUnsupportedField(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
_, err = engine.Compile(context.Background(), `visibility.startsWith("P")`)
require.Error(t, err)
require.Contains(t, err.Error(), "does not support text matching")
}
```
- [ ] **Step 6: Run tests to verify green**
Run: `go test ./internal/filter/ ./store/test/ -run 'StartsWith|EndsWith|TextMatch' -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add internal/filter/parser.go internal/filter/schema.go internal/filter/engine_test.go store/test/memo_filter_test.go store/test/attachment_filter_test.go
git commit -m "feat(filter): support startsWith()/endsWith() on scalar string fields
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
### Task 3: Register a SQLite `REGEXP` function
`modernc.org/sqlite` has no built-in `REGEXP`. SQLite desugars `X REGEXP Y` to `regexp(Y, X)`, so register a 2-arg `regexp(pattern, value)` scalar function backed by Go's `regexp`, mirroring `ensureUnicodeLowerRegistered`.
**Files:**
- Modify: `store/db/sqlite/functions.go`
- Modify: `store/db/sqlite/sqlite.go`
- Test: `store/db/sqlite/functions_test.go` (create)
- [ ] **Step 1: Write a failing test** — create `store/db/sqlite/functions_test.go`
```go
package sqlite
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestRegexpFunctionMatches(t *testing.T) {
require.NoError(t, ensureRegexpRegistered())
re, err := compileRegexp(`^v\d+$`)
require.NoError(t, err)
require.True(t, re.MatchString("v12"))
require.False(t, re.MatchString("version"))
// Caching returns the same compiled instance.
re2, err := compileRegexp(`^v\d+$`)
require.NoError(t, err)
require.Same(t, re, re2)
_, err = compileRegexp(`(`)
require.Error(t, err)
}
```
- [ ] **Step 2: Run to verify it fails**
Run: `go test ./store/db/sqlite/ -run TestRegexpFunctionMatches -v`
Expected: FAIL — `ensureRegexpRegistered`/`compileRegexp` undefined.
- [ ] **Step 3: Implement in `store/db/sqlite/functions.go`**
Add `"errors"` and `"regexp"` to the imports, then append:
```go
var (
registerRegexpOnce sync.Once
registerRegexpErr error
// regexpCache memoizes compiled patterns; keys are pattern strings.
regexpCache sync.Map
)
// ensureRegexpRegistered registers a Go-backed `regexp(pattern, value)` scalar
// function so SQLite's `value REGEXP pattern` operator works (modernc.org/sqlite
// has no built-in implementation). Patterns use Go's RE2 syntax. Registered once
// globally; safe to call multiple times.
func ensureRegexpRegistered() error {
registerRegexpOnce.Do(func() {
registerRegexpErr = msqlite.RegisterScalarFunction("regexp", 2, func(_ *msqlite.FunctionContext, args []driver.Value) (driver.Value, error) {
if len(args) != 2 || args[0] == nil || args[1] == nil {
return int64(0), nil
}
pattern, ok := args[0].(string)
if !ok {
return nil, errors.New("regexp pattern must be a string")
}
var value string
switch v := args[1].(type) {
case string:
value = v
case []byte:
value = string(v)
default:
return int64(0), nil
}
re, err := compileRegexp(pattern)
if err != nil {
return nil, err
}
if re.MatchString(value) {
return int64(1), nil
}
return int64(0), nil
})
})
return registerRegexpErr
}
// compileRegexp compiles and caches a RE2 pattern.
func compileRegexp(pattern string) (*regexp.Regexp, error) {
if cached, ok := regexpCache.Load(pattern); ok {
return cached.(*regexp.Regexp), nil
}
re, err := regexp.Compile(pattern)
if err != nil {
return nil, err
}
regexpCache.Store(pattern, re)
return re, nil
}
```
- [ ] **Step 4: Wire into `NewDB`** in `store/db/sqlite/sqlite.go`
Right after the `ensureUnicodeLowerRegistered()` block, add:
```go
if err := ensureRegexpRegistered(); err != nil {
return nil, errors.Wrap(err, "failed to register sqlite regexp function")
}
```
- [ ] **Step 5: Run to verify green**
Run: `go test ./store/db/sqlite/ -run TestRegexpFunctionMatches -v`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add store/db/sqlite/functions.go store/db/sqlite/sqlite.go store/db/sqlite/functions_test.go
git commit -m "feat(sqlite): register Go-backed REGEXP function
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
### Task 4: `matches(regex)` on string fields
Add the IR node, parser recognition, per-dialect rendering, and the compile-time regex validator.
**Files:**
- Modify: `internal/filter/ir.go` (add `RegexCondition`)
- Modify: `internal/filter/parser.go` (`matches` case + builder)
- Modify: `internal/filter/render.go` (`renderRegex`)
- Modify: `internal/filter/schema.go` (add `cel.ValidateRegexLiterals()` to both schemas)
- Modify: `internal/filter/engine_test.go`, `store/test/memo_filter_test.go`
- [ ] **Step 1: Add failing tests** — compile-level in `internal/filter/engine_test.go`:
```go
func TestCompileRejectsMalformedRegex(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
_, err = engine.Compile(context.Background(), `content.matches("(")`)
require.Error(t, err)
}
func TestCompileMatchesRendersRegexOperator(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
stmt, err := engine.CompileToStatement(context.Background(), `content.matches("v[0-9]+")`, RenderOptions{Dialect: DialectPostgres})
require.NoError(t, err)
require.Contains(t, stmt.SQL, "~")
require.Equal(t, []any{"v[0-9]+"}, stmt.Args)
}
```
And behavioral in `store/test/memo_filter_test.go` (runs against SQLite by default, exercising the registered `REGEXP` function):
```go
func TestMemoFilterContentMatches(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-v1", tc.User.ID).Content("release v12 shipped"))
tc.CreateMemo(NewMemoBuilder("memo-plain", tc.User.ID).Content("no version here"))
memos := tc.ListWithFilter(`content.matches("v[0-9]+")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-v1", memos[0].UID)
memos = tc.ListWithFilter(`content.matches("^xyz")`)
require.Len(t, memos, 0)
}
```
- [ ] **Step 2: Run to verify they fail**
Run: `go test ./internal/filter/ -run 'Malformed|MatchesRenders' -v && go test ./store/test/ -run TestMemoFilterContentMatches -v`
Expected: FAIL — `matches` is unhandled and no regex validator is configured.
- [ ] **Step 3: Add the IR node** in `internal/filter/ir.go`
```go
// RegexCondition models field.matches("pattern") on a string field.
type RegexCondition struct {
Field string
Pattern string
}
func (*RegexCondition) isCondition() {}
```
- [ ] **Step 4: Add parser support** in `internal/filter/parser.go`
In `buildCallCondition`, after the `case "endsWith":` block, add:
```go
case "matches":
return buildMatchesCondition(call, schema)
```
Then add the builder:
```go
func buildMatchesCondition(call *exprv1.Expr_Call, schema Schema) (Condition, error) {
if call.Target == nil {
return nil, errors.New("matches requires a target")
}
targetName, err := getIdentName(call.Target)
if err != nil {
return nil, err
}
field, ok := schema.Field(targetName)
if !ok {
return nil, errors.Errorf("unknown identifier %q", targetName)
}
if !field.SupportsContains {
return nil, errors.Errorf("identifier %q does not support matches()", targetName)
}
if len(call.Args) != 1 {
return nil, errors.New("matches expects exactly one argument")
}
value, err := getConstValue(call.Args[0])
if err != nil {
return nil, errors.Wrap(err, "matches only supports literal arguments")
}
pattern, ok := value.(string)
if !ok {
return nil, errors.New("matches argument must be a string")
}
return &RegexCondition{Field: targetName, Pattern: pattern}, nil
}
```
- [ ] **Step 5: Add the renderer** in `internal/filter/render.go`
In `renderCondition`, after the `case *TextMatchCondition:` arm, add:
```go
case *RegexCondition:
return r.renderRegex(c)
```
Then add:
```go
func (r *renderer) renderRegex(cond *RegexCondition) (renderResult, error) {
field, ok := r.schema.Field(cond.Field)
if !ok {
return renderResult{}, errors.Errorf("unknown field %q", cond.Field)
}
column := field.columnExpr(r.dialect)
switch r.dialect {
case DialectPostgres:
// POSIX regex match operator.
return renderResult{sql: fmt.Sprintf("%s ~ %s", column, r.addArg(cond.Pattern))}, nil
case DialectMySQL, DialectSQLite:
// MySQL has a native REGEXP operator; SQLite uses the registered regexp() function.
return renderResult{sql: fmt.Sprintf("%s REGEXP %s", column, r.addArg(cond.Pattern))}, nil
default:
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
}
}
```
- [ ] **Step 6: Add the regex validator** in `internal/filter/schema.go`
Add the `cel` import line already present. In **both** `NewSchema` and `NewAttachmentSchema`, append the validator to the `envOptions` slice (e.g. after `nowFunction`):
```go
cel.ASTValidators(cel.ValidateRegexLiterals()),
```
- [ ] **Step 7: Run tests to verify green**
Run: `go test ./internal/filter/ -run 'Malformed|MatchesRenders' -v && go test ./store/test/ -run TestMemoFilterContentMatches -v`
Expected: PASS.
- [ ] **Step 8: Commit**
```bash
git add internal/filter/ir.go internal/filter/parser.go internal/filter/render.go internal/filter/schema.go internal/filter/engine_test.go store/test/memo_filter_test.go
git commit -m "feat(filter): support matches() regex on string fields
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
### Task 5: `all()` comprehension on tags (non-empty)
The heaviest task. `exists()` matches against the *serialized* JSON array and cannot express "every element matches", so `all()` needs real per-element iteration via `json_each` / `jsonb_array_elements_text` / `JSON_TABLE`, plus a non-empty guard.
**Files:**
- Modify: `internal/filter/ir.go` (add `ComprehensionAll`)
- Modify: `internal/filter/parser.go` (accept `all()` in `detectComprehensionKind`)
- Modify: `internal/filter/render.go` (`renderTagAll` + element predicate SQL; branch in `renderListComprehension`)
- Modify: `store/test/memo_filter_test.go`
- [ ] **Step 1: Add failing behavioral tests** in `store/test/memo_filter_test.go`
```go
func TestMemoFilterTagsAll(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-all-work", tc.User.ID).Content("all work").Tags("work/a", "work/b"))
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed").Tags("work/a", "home"))
tc.CreateMemo(NewMemoBuilder("memo-untagged", tc.User.ID).Content("untagged"))
// Every tag starts with "work/": only the all-work memo qualifies.
memos := tc.ListWithFilter(`tags.all(t, t.startsWith("work/"))`)
require.Len(t, memos, 1)
require.Equal(t, "memo-all-work", memos[0].UID)
// Untagged memos must NOT match (non-empty guard, decision B).
require.NotContains(t, uids(memos), "memo-untagged")
}
func TestMemoFilterTagsAllEquals(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-only-x", tc.User.ID).Content("only x").Tags("x", "x"))
tc.CreateMemo(NewMemoBuilder("memo-x-and-y", tc.User.ID).Content("x and y").Tags("x", "y"))
memos := tc.ListWithFilter(`tags.all(t, t == "x")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-only-x", memos[0].UID)
}
```
Add this helper near the top of `store/test/memo_filter_test.go` (after the imports) if not already present:
```go
func uids(memos []*store.Memo) []string {
out := make([]string, 0, len(memos))
for _, m := range memos {
out = append(out, m.UID)
}
return out
}
```
- [ ] **Step 2: Run to verify they fail**
Run: `go test ./store/test/ -run 'TestMemoFilterTagsAll' -v`
Expected: FAIL — `detectComprehensionKind` returns "all() comprehension is not supported".
- [ ] **Step 3: Add the IR kind** in `internal/filter/ir.go`
In the `ComprehensionKind` const block, add `ComprehensionAll`:
```go
const (
ComprehensionExists ComprehensionKind = "exists"
ComprehensionAll ComprehensionKind = "all"
)
```
- [ ] **Step 4: Accept `all()` in the parser** in `internal/filter/parser.go`
In `detectComprehensionKind`, replace the `all()` rejection block:
```go
// all() starts with true and uses AND (&&) - not supported
if accuInit.GetBoolValue() {
if step := comp.LoopStep.GetCallExpr(); step != nil && step.Function == "_&&_" {
return "", errors.New("all() comprehension is not supported; use exists() instead")
}
}
```
with:
```go
// all() starts with true and uses AND (&&) in the loop step.
if accuInit.GetBoolValue() {
if step := comp.LoopStep.GetCallExpr(); step != nil && step.Function == "_&&_" {
return ComprehensionAll, nil
}
}
```
- [ ] **Step 5: Branch and render in `internal/filter/render.go`**
At the top of `renderListComprehension`, right after the `field.Kind != FieldKindJSONList` guard, add:
```go
if cond.Kind == ComprehensionAll {
return r.renderTagAll(field, cond.Predicate)
}
```
Then add the new render path + element-predicate helper:
```go
// renderTagAll renders tags.all(t, <pred>): the array is non-empty AND no element
// fails the predicate. Element predicates use plain CEL semantics (case-insensitive
// for startsWith/endsWith/contains, case-sensitive for ==), evaluated per element.
func (r *renderer) renderTagAll(field Field, pred PredicateExpr) (renderResult, error) {
arrayExpr := jsonArrayExpr(r.dialect, field)
elemCond, err := r.elementPredicateSQL(pred)
if err != nil {
return renderResult{}, err
}
switch r.dialect {
case DialectSQLite:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND %s != '[]'", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM json_each(%s) WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
case DialectMySQL:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND JSON_LENGTH(%s) > 0", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM JSON_TABLE(%s, '$[*]' COLUMNS (value VARCHAR(512) PATH '$')) AS elem WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
case DialectPostgres:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND jsonb_array_length(%s) > 0", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(%s) AS elem(value) WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
default:
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
}
}
// elementPredicateSQL builds the per-element SQL condition for an all() predicate.
// The iterated element is exposed as the unqualified column `value` on all dialects
// (json_each.value / JSON_TABLE column / elem(value)).
func (r *renderer) elementPredicateSQL(pred PredicateExpr) (string, error) {
switch p := pred.(type) {
case *EqualsPredicate:
return fmt.Sprintf("value = %s", r.addArg(p.Value)), nil
case *StartsWithPredicate:
return r.foldedLike("value", likePattern(TextMatchPrefix, p.Prefix)), nil
case *EndsWithPredicate:
return r.foldedLike("value", likePattern(TextMatchSuffix, p.Suffix)), nil
case *ContainsPredicate:
return r.foldedLike("value", likePattern(TextMatchContains, p.Substring)), nil
default:
return "", errors.Errorf("unsupported predicate %T in all()", pred)
}
}
```
> Note: `foldedLike`, `likePattern`, and `escapeLikeLiteral` were added in Task 1; reuse them as-is.
- [ ] **Step 6: Run tests to verify green**
Run: `go test ./store/test/ -run 'TestMemoFilterTagsAll' -v`
Expected: PASS.
Run: `go test ./store/test/ -run 'TestMemoFilterTagsExists' -v`
Expected: PASS (exists() rendering untouched).
- [ ] **Step 7: Commit**
```bash
git add internal/filter/ir.go internal/filter/parser.go internal/filter/render.go store/test/memo_filter_test.go
git commit -m "feat(filter): support all() comprehension over tags
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
### Task 6: Docs + full verification
**Files:**
- Modify: `internal/filter/README.md`
- [ ] **Step 1: Document the new syntax** — append to the "SQL Generation Notes" section of `internal/filter/README.md`:
```markdown
- **String Matching** — `content.contains(x)`, `content.startsWith(x)`, and
`content.endsWith(x)` render as case-insensitive `LIKE`/`ILIKE` with LIKE
metacharacters (`%`, `_`, `\`) escaped. Available on scalar string fields whose
schema sets `SupportsContains` (memo `content`; attachment `filename`,
`mime_type`).
- **Regex** — `field.matches("pattern")` renders to `~` (Postgres) or `REGEXP`
(MySQL/SQLite). SQLite uses a Go-backed `regexp` function registered in
`store/db/sqlite/functions.go`. Patterns are validated at compile time against
Go's RE2 via `cel.ValidateRegexLiterals()`. **Caveat:** regex *syntax* differs
per engine (Go RE2 on SQLite, POSIX ERE on Postgres, ICU on MySQL 8.0+), so
engine-specific patterns may not be portable.
- **Tag `all()`** — `tags.all(t, <pred>)` matches only non-empty tag sets where
every element satisfies the predicate, via per-element iteration
(`json_each` / `jsonb_array_elements_text` / `JSON_TABLE`).
```
- [ ] **Step 2: Run the full engine + store suite (SQLite)**
Run: `go test ./internal/filter/... ./store/...`
Expected: PASS.
- [ ] **Step 3: Vet and lint**
Run: `go vet ./internal/filter/... ./store/db/sqlite/...`
Expected: no output.
Run: `golangci-lint run internal/filter/... store/db/sqlite/...` (if available; skip if the binary is absent).
Expected: no findings.
- [ ] **Step 4: Cross-dialect verification (if Docker/CI DSNs available)**
Run MySQL and Postgres suites to confirm the `all()` subqueries and regex operators render correctly:
```bash
DRIVER=mysql go test ./store/test/ -run 'TagsAll|Matches|StartsWith|EndsWith'
DRIVER=postgres go test ./store/test/ -run 'TagsAll|Matches|StartsWith|EndsWith'
```
Expected: PASS. (These require the project's standard test DB setup; if unavailable locally, rely on CI which runs all three drivers.)
- [ ] **Step 5: Commit**
```bash
git add internal/filter/README.md
git commit -m "docs(filter): document string matching, regex, and tag all() support
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>"
```
---
## Self-review notes
- **Spec coverage:** ① scalar `startsWith`/`endsWith` → Task 2; ② `all()` non-empty → Task 5; ④ `matches()` + SQLite REGEXP fn + `ValidateRegexLiterals` → Tasks 3-4; the LIKE-escaping fix → Task 1; docs/caveat → Task 6. `lowerAscii`/`upperAscii` correctly omitted (dropped in spec). Hardening/native-AST migration correctly deferred to the follow-up spec.
- **Type consistency:** `TextMatchCondition`/`TextMatchMode`/`likePattern`/`foldedLike`/`escapeLikeLiteral` (Task 1) are reused by Tasks 2 and 5; `RegexCondition`/`renderRegex` (Task 4) and `ensureRegexpRegistered`/`compileRegexp` (Task 3) names match across their call sites; `ComprehensionAll` (Task 5) matches its parser and render references.
- **Element reference:** the unqualified `value` column is produced by `json_each` (SQLite), the `JSON_TABLE(... COLUMNS (value ...))` (MySQL), and `elem(value)` (Postgres), so `elementPredicateSQL` is dialect-agnostic.
```
@@ -0,0 +1,257 @@
# Design: Expand the CEL filter surface
- **Date:** 2026-06-15
- **Status:** Approved (design); ready for implementation planning
- **Area:** `internal/filter` (memo & attachment filter engine)
- **Follow-up spec:** CEL engine hardening + native-AST migration (separate, sequenced after this)
## Summary
memos lets API clients pass a CEL expression in the `filter` field of list
requests. The `internal/filter` engine uses `cel-go` purely as a **parse +
type-check frontend**, then walks the AST and translates it into a SQL `WHERE`
fragment for the active dialect (SQLite / MySQL / Postgres). cel-go never
evaluates anything.
This spec adds three new CEL constructs that users can write, each with a SQL
translation across all three dialects:
1. `startsWith()` / `endsWith()` on scalar string fields (case-insensitive).
2. `all()` comprehension on tag lists (matches only non-empty tag sets).
3. `matches(regex)` on string fields.
## Goals
- Expose the three constructs above through the existing
parse → IR → render pipeline.
- Keep parity across SQLite, MySQL, and Postgres, with golden tests for each.
- Preserve the engine's invariant: only schema-declared fields and explicitly
supported operations are accepted; everything else is rejected with a clear
error.
## Non-goals
- **Value-producing CEL features with no SQL form** are explicitly out of scope:
optional types (`?.`, `optional.of`), `map()` / `filter()` transforms, the
math extension, string-manipulation extensions (`replace`, `split`,
`substring`, `format`), and two-variable comprehensions. There is nothing to
push into a `WHERE` clause for these.
- **`lowerAscii()` / `upperAscii()`** — dropped. `contains()` is already
case-insensitive on all dialects, and the new `startsWith`/`endsWith` are
case-insensitive too (see decisions), so explicit case-folding adds little.
Revisit only if users ask.
- **Parser hardening and the native-AST proto migration** are a separate
follow-up spec. The one exception that rides along here is
`cel.ValidateRegexLiterals()`, which feature ③ requires for safety.
## Background: how the engine works today
Pipeline (see `internal/filter/README.md`):
1. **Parse**`env.Compile(filter)` parses and type-checks against the
memo/attachment environment declared in `schema.go`; the AST is converted via
`cel.AstToParsedExpr()`.
2. **Normalize**`parser.go` walks the CEL `Expr` and builds a
dialect-agnostic IR (`ir.go`): logical ops, comparisons, `IN`, `contains()`,
and `exists()` comprehensions over tag lists.
3. **Render**`render.go` walks the IR and emits dialect-specific SQL plus
placeholder args.
Two existing facts that shaped this design:
- **`contains()` is already case-insensitive** on all three dialects
(`render.go` `renderContainsCondition`): SQLite uses the custom
`memos_unicode_lower` function, Postgres uses `ILIKE`, MySQL relies on its
default case-insensitive collation.
- **Custom SQLite scalar functions are already registered**
(`store/db/sqlite/functions.go`, `ensureUnicodeLowerRegistered` via
`modernc.org/sqlite`'s `RegisterScalarFunction`, invoked from
`store/db/sqlite/sqlite.go`). The new `REGEXP` function follows this exact
pattern.
cel-go version: `v0.28.0` (latest is `v0.28.1`, a patch with nothing relevant to
memos). No version bump is required for this work.
## Resolved decisions
| # | Decision | Choice |
|---|----------|--------|
| A | Case-sensitivity of new scalar `startsWith`/`endsWith` | **Case-insensitive**, consistent with existing `contains()`. `==` stays case-sensitive (exact match). |
| B | `all()` over a memo with zero tags | **Require non-empty**: an untagged memo does NOT match an `all()` filter. (Diverges from strict CEL vacuous-truth, but matches search-box intuition.) |
| C | Keep `lowerAscii()` / `upperAscii()`? | **Drop** from this spec. |
## Detailed design
### ① `startsWith()` / `endsWith()` on scalar string fields
**Surface.** Allow `field.startsWith("x")` and `field.endsWith("x")` as
top-level boolean calls for scalar string fields. Today these functions are only
recognized *inside* tag comprehensions (`parser.go` `extractPredicate`).
Applicable fields: memo `content`; attachment `filename`, `mime_type`.
`creator` is intentionally **excluded**: it is an identity field with `==`/`!=`
semantics whose column is wrapped as `'users/' || username`, so prefix/suffix
matching there would match against the `users/` prefix and surprise users.
**Schema.** Generalize the per-field text-matching capability. Today `Field` has
`SupportsContains bool`. Replace/extend with a capability that also covers
prefix/suffix matching (e.g. a `SupportsTextMatch bool`, or reuse
`SupportsContains` to gate all three LIKE-based ops). Fields that already set
`SupportsContains: true` gain prefix/suffix support.
**Parser.** In `buildCallCondition`, recognize `startsWith` / `endsWith` calls
whose target is a scalar string field and whose single argument is a string
literal. Reject non-literal arguments and fields without the capability.
**IR.** Generalize `ContainsCondition` into a single node:
```go
type TextMatchMode string
const (
TextMatchContains TextMatchMode = "contains"
TextMatchPrefix TextMatchMode = "prefix"
TextMatchSuffix TextMatchMode = "suffix"
)
type TextMatchCondition struct {
Field string
Mode TextMatchMode
Value string
}
```
`contains()` migrates to `TextMatchCondition{Mode: TextMatchContains}`.
**Render.** Build a `LIKE` pattern from the (escaped) literal:
- prefix → `value%`
- suffix → `%value`
- contains → `%value%`
Reuse the existing case-insensitive rendering already used by `contains()`:
SQLite `memos_unicode_lower(col) LIKE memos_unicode_lower(?)`, Postgres
`col ILIKE $n`, MySQL `col LIKE ?`.
**LIKE-escaping fix.** The current `contains()` renderer interpolates the raw
value into the pattern without escaping `%`, `_`, or `\`. This means a search
for `50%` behaves as a wildcard. The new shared path will escape these
metacharacters (and emit `ESCAPE '\'` where required by the dialect). This
closes a small latent wildcard-injection inconsistency and applies uniformly to
contains/prefix/suffix.
### ② `all()` comprehension on tag lists
**Surface.** Allow `tags.all(t, <pred>)` where `<pred>` is one of the predicates
already supported for `exists()`: `t == "x"`, `t.startsWith("x")`,
`t.endsWith("x")`, `t.contains("x")`.
**Parser.** `detectComprehensionKind` currently accepts only `exists()` and
explicitly rejects `all()`. Add a `ComprehensionAll` kind (accumulator inits to
`true`, loop step uses `_&&_`). Reuse the existing predicate extraction.
**IR.** Add `ComprehensionAll` to the `ComprehensionKind` enum; the existing
`ListComprehensionCondition` already carries `Kind`.
**Render — proper per-element semantics.** The existing `exists()`
implementation matches the *serialized* JSON array text with `LIKE`, which works
for "at least one element matches a substring" but **cannot** express "every
element matches." `all()` therefore needs real per-element iteration. Decision B
(require non-empty) means: array is non-empty **AND** no element fails the
predicate.
- **SQLite:**
```sql
(<array> IS NOT NULL AND <array> != '[]'
AND NOT EXISTS (SELECT 1 FROM json_each(<array>)
WHERE NOT (<predicate on json_each.value>)))
```
- **Postgres:**
```sql
(<array> IS NOT NULL AND jsonb_array_length(<array>) > 0
AND NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(<array>) AS e(value)
WHERE NOT (<predicate on e.value>)))
```
- **MySQL:**
```sql
(<array> IS NOT NULL AND JSON_LENGTH(<array>) > 0
AND NOT EXISTS (SELECT 1 FROM JSON_TABLE(<array>, '$[*]'
COLUMNS (value VARCHAR(512) PATH '$')) AS j
WHERE NOT (<predicate on j.value>)))
```
The per-element predicate reuses LIKE/`=` against the element `value`
(case-insensitive for `startsWith`/`endsWith`/`contains`, consistent with ①).
Hierarchical-tag prefix behavior should match the existing `exists()` rendering
(a prefix matches the exact tag or a `tag/...` child).
> Note: this introduces correlated subqueries against the same `memo.payload`
> column the outer query already reads; confirm the generated SQL composes with
> the surrounding `WHERE` and placeholder offsets in `helpers.AppendConditions`.
### ④ `matches(regex)` on string fields
**Surface.** Allow `field.matches("pattern")` for the same free-text fields as ①
(`content`, `filename`, `mime_type`; `creator` excluded), literal pattern only.
**Env / validation.** Add `cel.ValidateRegexLiterals()` to the env options in
`schema.go` so malformed patterns fail at compile time with a clear message
(validated against Go's RE2).
**Parser / IR.** Recognize `matches` calls; add:
```go
type RegexCondition struct {
Field string
Pattern string
}
```
Reject non-literal patterns and fields without text-match capability.
**Render.**
- **Postgres:** `col ~ $n`
- **MySQL:** `col REGEXP ?`
- **SQLite:** `col REGEXP ?`. SQLite desugars `X REGEXP Y` to the function call
`regexp(Y, X)`, so register a 2-arg scalar function named `regexp(pattern,
value)` returning 1/0, backed by Go's `regexp` package, following the
`ensureUnicodeLowerRegistered` pattern in `store/db/sqlite/functions.go`.
Compile patterns lazily with a small cache (or rely on RE2 compile per call;
decide during implementation based on measured cost).
**Documented caveats** (engine differences are inherent, not bugs):
- Regex *syntax* differs per engine: SQLite uses Go RE2; Postgres uses POSIX
ERE; MySQL 8.0+ uses ICU. Portable patterns work everywhere; engine-specific
constructs may not. Document this in `internal/filter/README.md`.
- ReDoS risk is low: RE2 (SQLite path) is linear-time; Postgres/MySQL POSIX
engines do not catastrophically backtrack. `ValidateRegexLiterals()` rejects
patterns that don't compile under RE2 as a first-line guard.
## Testing strategy
For each feature, add golden tests in
`store/db/{sqlite,mysql,postgres}/memo_filter_test.go` (and the attachment
filter tests where applicable):
- **Happy path:** assert the exact SQL fragment and args per dialect.
- **Error paths:** non-literal argument, unsupported field, malformed regex,
unsupported predicate inside `all()`.
- **`all()` empty-set:** confirm an untagged memo does not match (decision B).
- **LIKE escaping:** confirm `%`, `_`, `\` in `contains`/`startsWith`/`endsWith`
values are treated literally.
Run `go test ./...` (engine unit tests plus all three dialect suites). The
`contains()``TextMatchCondition` refactor must keep existing golden outputs
unchanged except for the intentional escaping fix.
## Rollout / sequencing
This is the first of two specs. The second (already agreed) covers engine
hardening: tightened parser limits (`ParserExpressionSizeLimit`,
`ParserRecursionLimit`, `ParserErrorRecoveryLimit`), the
`ValidateComprehensionNestingLimit` / `ValidateHomogeneousAggregateLiterals`
validators, and migrating `parser.go` off the deprecated
`genproto/.../expr/v1alpha1` proto to the native `common/ast` API. Building this
surface-expansion spec first is acceptable; the hardening migration is a pure
refactor that the golden tests written here will help protect.
+1 -1
View File
@@ -10,7 +10,7 @@ require (
github.com/aws/aws-sdk-go-v2/credentials v1.19.15 github.com/aws/aws-sdk-go-v2/credentials v1.19.15
github.com/aws/aws-sdk-go-v2/service/s3 v1.100.0 github.com/aws/aws-sdk-go-v2/service/s3 v1.100.0
github.com/go-sql-driver/mysql v1.9.3 github.com/go-sql-driver/mysql v1.9.3
github.com/google/cel-go v0.28.0 github.com/google/cel-go v0.28.1
github.com/google/jsonschema-go v0.4.3 github.com/google/jsonschema-go v0.4.3
github.com/google/uuid v1.6.0 github.com/google/uuid v1.6.0
github.com/gorilla/feeds v1.2.0 github.com/gorilla/feeds v1.2.0
+2 -2
View File
@@ -111,8 +111,8 @@ github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63Y
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/cel-go v0.28.0 h1:KjSWstCpz/MN5t4a8gnGJNIYUsJRpdi/r97xWDphIQc= github.com/google/cel-go v0.28.1 h1:YWIwi77J4xIsYUwAF/iIuS6haffzIHS8yWI8glSbLWM=
github.com/google/cel-go v0.28.0/go.mod h1:X0bD6iVNR8pkROSOoHVdgTkzmRcosof7WQqCD6wcMc8= github.com/google/cel-go v0.28.1/go.mod h1:X0bD6iVNR8pkROSOoHVdgTkzmRcosof7WQqCD6wcMc8=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0= github.com/google/jsonschema-go v0.4.3 h1:/DBOLZTfDow7pe2GmaJNhltueGTtDKICi8V8p+DQPd0=
+14
View File
@@ -53,6 +53,20 @@ stmt, _ := engine.CompileToStatement(ctx, `has_task_list && visibility == "PUBLI
Postgres uses `@>`. Postgres uses `@>`.
- **Boolean Flags** — Fields such as `has_task_list` render as `IS TRUE` equality - **Boolean Flags** — Fields such as `has_task_list` render as `IS TRUE` equality
checks, or comparisons against `CAST('true' AS JSON)` depending on the dialect. checks, or comparisons against `CAST('true' AS JSON)` depending on the dialect.
- **String Matching** — `content.contains(x)`, `content.startsWith(x)`, and
`content.endsWith(x)` render as case-insensitive `LIKE`/`ILIKE` with LIKE
metacharacters (`%`, `_`, `\`) escaped. Available on scalar string fields whose
schema sets `SupportsContains` (memo `content`; attachment `filename`,
`mime_type`).
- **Regex** — `field.matches("pattern")` renders to `~` (Postgres) or `REGEXP`
(MySQL/SQLite). SQLite uses a Go-backed `regexp` function registered in
`store/db/sqlite/functions.go`. Patterns are validated at compile time against
Go's RE2 via `cel.ValidateRegexLiterals()`. **Caveat:** regex *syntax* differs
per engine (Go RE2 on SQLite, POSIX ERE on Postgres, ICU on MySQL 8.0+), so
engine-specific patterns may not be portable.
- **Tag `all()`** — `tags.all(t, <pred>)` matches only non-empty tag sets where
every element satisfies the predicate, via per-element iteration
(`json_each` / `jsonb_array_elements_text` / `JSON_TABLE`).
## Typical Integration ## Typical Integration
+158
View File
@@ -37,3 +37,161 @@ func TestCompileRejectsNonBooleanTopLevelConstant(t *testing.T) {
_, err = engine.Compile(context.Background(), `1`) _, err = engine.Compile(context.Background(), `1`)
require.EqualError(t, err, "filter must evaluate to a boolean value") require.EqualError(t, err, "filter must evaluate to a boolean value")
} }
func TestCompileRejectsMalformedRegex(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
_, err = engine.Compile(context.Background(), `content.matches("(")`)
require.Error(t, err)
}
func TestCompileMatchesRendersRegexOperator(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
stmt, err := engine.CompileToStatement(context.Background(), `content.matches("v[0-9]+")`, RenderOptions{Dialect: DialectPostgres})
require.NoError(t, err)
require.Contains(t, stmt.SQL, "~")
require.Equal(t, []any{"v[0-9]+"}, stmt.Args)
}
func TestCompileRejectsStartsWithOnUnsupportedField(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
_, err = engine.Compile(context.Background(), `visibility.startsWith("P")`)
require.Error(t, err)
require.Contains(t, err.Error(), "does not support text matching")
}
func TestCompileContainsEscapesLikeWildcards(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
stmt, err := engine.CompileToStatement(context.Background(), `content.contains("50%_off")`, RenderOptions{Dialect: DialectSQLite})
require.NoError(t, err)
// The % and _ in the value must be escaped so they are matched literally,
// and SQLite needs an explicit ESCAPE clause.
require.Contains(t, stmt.SQL, `ESCAPE '\'`)
require.Equal(t, []any{`%50\%\_off%`}, stmt.Args)
}
// =============================================================================
// Cross-dialect rendering tests (no DB required; complements the SQLite-only
// behavioral tests in store/test by asserting MySQL/Postgres SQL generation).
// =============================================================================
func TestRenderStartsWithPerDialect(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
cases := []struct {
dialect DialectName
fragments []string
}{
{DialectSQLite, []string{"memos_unicode_lower(", "`memo`.`content`", `ESCAPE '\'`}},
{DialectPostgres, []string{"memo.content ILIKE $1"}},
{DialectMySQL, []string{"`memo`.`content` LIKE ?"}},
}
for _, tc := range cases {
stmt, err := engine.CompileToStatement(context.Background(), `content.startsWith("TODO")`, RenderOptions{Dialect: tc.dialect})
require.NoError(t, err, tc.dialect)
for _, frag := range tc.fragments {
require.Contains(t, stmt.SQL, frag, "dialect %s", tc.dialect)
}
require.Equal(t, []any{"TODO%"}, stmt.Args, "dialect %s", tc.dialect)
}
}
func TestRenderEndsWithPerDialect(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
for _, dialect := range []DialectName{DialectSQLite, DialectPostgres, DialectMySQL} {
stmt, err := engine.CompileToStatement(context.Background(), `content.endsWith(".md")`, RenderOptions{Dialect: dialect})
require.NoError(t, err, dialect)
require.Equal(t, []any{"%.md"}, stmt.Args, "dialect %s", dialect)
}
}
func TestRenderMatchesPerDialect(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
cases := []struct {
dialect DialectName
fragment string
}{
{DialectSQLite, "`memo`.`content` REGEXP ?"},
{DialectMySQL, "`memo`.`content` REGEXP ?"},
{DialectPostgres, "memo.content ~ $1"},
}
for _, tc := range cases {
stmt, err := engine.CompileToStatement(context.Background(), `content.matches("v[0-9]+")`, RenderOptions{Dialect: tc.dialect})
require.NoError(t, err, tc.dialect)
require.Contains(t, stmt.SQL, tc.fragment, "dialect %s", tc.dialect)
require.Equal(t, []any{"v[0-9]+"}, stmt.Args, "dialect %s", tc.dialect)
}
}
func TestRenderTagsAllPerDialect(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
cases := []struct {
dialect DialectName
fragments []string
}{
{DialectSQLite, []string{"NOT EXISTS", "json_each(", "!= '[]'", "memos_unicode_lower(value)"}},
{DialectPostgres, []string{"NOT EXISTS", "jsonb_array_elements_text(", "jsonb_array_length(", "value ILIKE"}},
{DialectMySQL, []string{"NOT EXISTS", "JSON_TABLE(", "JSON_LENGTH(", "value LIKE"}},
}
for _, tc := range cases {
stmt, err := engine.CompileToStatement(context.Background(), `tags.all(t, t.startsWith("work/"))`, RenderOptions{Dialect: tc.dialect})
require.NoError(t, err, tc.dialect)
for _, frag := range tc.fragments {
require.Contains(t, stmt.SQL, frag, "dialect %s", tc.dialect)
}
require.Equal(t, []any{"work/%"}, stmt.Args, "dialect %s", tc.dialect)
}
}
func TestRenderTextMatchEscaping(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
// Both % and _ in the value must be escaped so they match literally.
stmt, err := engine.CompileToStatement(context.Background(), `content.contains("a%b_c")`, RenderOptions{Dialect: DialectSQLite})
require.NoError(t, err)
require.Equal(t, []any{`%a\%b\_c%`}, stmt.Args)
}
func TestRenderAllRejectsUnsupportedPredicate(t *testing.T) {
t.Parallel()
engine, err := NewEngine(NewSchema())
require.NoError(t, err)
// size() is not a valid per-element predicate inside all().
_, err = engine.CompileToStatement(context.Background(), `tags.all(t, size(t) > 2)`, RenderOptions{Dialect: DialectSQLite})
require.Error(t, err)
}
+23 -3
View File
@@ -73,13 +73,32 @@ type ElementInCondition struct {
func (*ElementInCondition) isCondition() {} func (*ElementInCondition) isCondition() {}
// ContainsCondition models the <field>.contains(<value>) call. // TextMatchMode enumerates LIKE-based string match modes.
type ContainsCondition struct { type TextMatchMode string
const (
TextMatchContains TextMatchMode = "contains"
TextMatchPrefix TextMatchMode = "prefix"
TextMatchSuffix TextMatchMode = "suffix"
)
// TextMatchCondition models a case-insensitive LIKE match on a scalar string field
// (content.contains/startsWith/endsWith).
type TextMatchCondition struct {
Field string Field string
Mode TextMatchMode
Value string Value string
} }
func (*ContainsCondition) isCondition() {} func (*TextMatchCondition) isCondition() {}
// RegexCondition models field.matches("pattern") on a string field.
type RegexCondition struct {
Field string
Pattern string
}
func (*RegexCondition) isCondition() {}
// ConstantCondition captures a literal boolean outcome. // ConstantCondition captures a literal boolean outcome.
type ConstantCondition struct { type ConstantCondition struct {
@@ -130,6 +149,7 @@ type ComprehensionKind string
const ( const (
ComprehensionExists ComprehensionKind = "exists" ComprehensionExists ComprehensionKind = "exists"
ComprehensionAll ComprehensionKind = "all"
) )
// PredicateExpr represents predicates used in comprehensions. // PredicateExpr represents predicates used in comprehensions.
+50 -10
View File
@@ -87,7 +87,13 @@ func buildCallCondition(call *exprv1.Expr_Call, schema Schema) (Condition, error
case "@in": case "@in":
return buildInCondition(call, schema) return buildInCondition(call, schema)
case "contains": case "contains":
return buildContainsCondition(call, schema) return buildTextMatchCondition(call, schema, TextMatchContains)
case "startsWith":
return buildTextMatchCondition(call, schema, TextMatchPrefix)
case "endsWith":
return buildTextMatchCondition(call, schema, TextMatchSuffix)
case "matches":
return buildMatchesCondition(call, schema)
default: default:
val, ok, err := evaluateBool(call) val, ok, err := evaluateBool(call)
if err != nil { if err != nil {
@@ -193,9 +199,9 @@ func buildInCondition(call *exprv1.Expr_Call, schema Schema) (Condition, error)
return nil, errors.New("invalid use of in operator") return nil, errors.New("invalid use of in operator")
} }
func buildContainsCondition(call *exprv1.Expr_Call, schema Schema) (Condition, error) { func buildTextMatchCondition(call *exprv1.Expr_Call, schema Schema, mode TextMatchMode) (Condition, error) {
if call.Target == nil { if call.Target == nil {
return nil, errors.New("contains requires a target") return nil, errors.New("text match requires a target")
} }
targetName, err := getIdentName(call.Target) targetName, err := getIdentName(call.Target)
if err != nil { if err != nil {
@@ -207,25 +213,59 @@ func buildContainsCondition(call *exprv1.Expr_Call, schema Schema) (Condition, e
return nil, errors.Errorf("unknown identifier %q", targetName) return nil, errors.Errorf("unknown identifier %q", targetName)
} }
if !field.SupportsContains { if !field.SupportsContains {
return nil, errors.Errorf("identifier %q does not support contains()", targetName) return nil, errors.Errorf("identifier %q does not support text matching", targetName)
} }
if len(call.Args) != 1 { if len(call.Args) != 1 {
return nil, errors.New("contains expects exactly one argument") return nil, errors.New("text match expects exactly one argument")
} }
value, err := getConstValue(call.Args[0]) value, err := getConstValue(call.Args[0])
if err != nil { if err != nil {
return nil, errors.Wrap(err, "contains only supports literal arguments") return nil, errors.Wrap(err, "text match only supports literal arguments")
} }
str, ok := value.(string) str, ok := value.(string)
if !ok { if !ok {
return nil, errors.New("contains argument must be a string") return nil, errors.New("text match argument must be a string")
} }
return &ContainsCondition{ return &TextMatchCondition{
Field: targetName, Field: targetName,
Mode: mode,
Value: str, Value: str,
}, nil }, nil
} }
func buildMatchesCondition(call *exprv1.Expr_Call, schema Schema) (Condition, error) {
if call.Target == nil {
return nil, errors.New("matches requires a target")
}
targetName, err := getIdentName(call.Target)
if err != nil {
return nil, err
}
field, ok := schema.Field(targetName)
if !ok {
return nil, errors.Errorf("unknown identifier %q", targetName)
}
if !field.SupportsContains {
return nil, errors.Errorf("identifier %q does not support matches()", targetName)
}
if len(call.Args) != 1 {
return nil, errors.New("matches expects exactly one argument")
}
value, err := getConstValue(call.Args[0])
if err != nil {
return nil, errors.Wrap(err, "matches only supports literal arguments")
}
pattern, ok := value.(string)
if !ok {
return nil, errors.New("matches argument must be a string")
}
return &RegexCondition{
Field: targetName,
Pattern: pattern,
}, nil
}
func buildValueExpr(expr *exprv1.Expr, schema Schema) (ValueExpr, error) { func buildValueExpr(expr *exprv1.Expr, schema Schema) (ValueExpr, error) {
if identName, err := getIdentName(expr); err == nil { if identName, err := getIdentName(expr); err == nil {
if _, ok := schema.Field(identName); !ok { if _, ok := schema.Field(identName); !ok {
@@ -466,10 +506,10 @@ func detectComprehensionKind(comp *exprv1.Expr_Comprehension) (ComprehensionKind
} }
} }
// all() starts with true and uses AND (&&) - not supported // all() starts with true and uses AND (&&) in the loop step.
if accuInit.GetBoolValue() { if accuInit.GetBoolValue() {
if step := comp.LoopStep.GetCallExpr(); step != nil && step.Function == "_&&_" { if step := comp.LoopStep.GetCallExpr(); step != nil && step.Function == "_&&_" {
return "", errors.New("all() comprehension is not supported; use exists() instead") return ComprehensionAll, nil
} }
} }
+105 -13
View File
@@ -72,8 +72,10 @@ func (r *renderer) renderCondition(cond Condition) (renderResult, error) {
return r.renderInCondition(c) return r.renderInCondition(c)
case *ElementInCondition: case *ElementInCondition:
return r.renderElementInCondition(c) return r.renderElementInCondition(c)
case *ContainsCondition: case *TextMatchCondition:
return r.renderContainsCondition(c) return r.renderTextMatch(c)
case *RegexCondition:
return r.renderRegex(c)
case *ListComprehensionCondition: case *ListComprehensionCondition:
return r.renderListComprehension(c) return r.renderListComprehension(c)
case *ConstantCondition: case *ConstantCondition:
@@ -446,28 +448,69 @@ func (r *renderer) renderScalarInCondition(field Field, values []ValueExpr) (ren
}, nil }, nil
} }
func (r *renderer) renderContainsCondition(cond *ContainsCondition) (renderResult, error) { func (r *renderer) renderTextMatch(cond *TextMatchCondition) (renderResult, error) {
field, ok := r.schema.Field(cond.Field) field, ok := r.schema.Field(cond.Field)
if !ok { if !ok {
return renderResult{}, errors.Errorf("unknown field %q", cond.Field) return renderResult{}, errors.Errorf("unknown field %q", cond.Field)
} }
column := field.columnExpr(r.dialect) column := field.columnExpr(r.dialect)
arg := fmt.Sprintf("%%%s%%", cond.Value) pattern := likePattern(cond.Mode, cond.Value)
return renderResult{sql: r.foldedLike(column, pattern)}, nil
}
func (r *renderer) renderRegex(cond *RegexCondition) (renderResult, error) {
field, ok := r.schema.Field(cond.Field)
if !ok {
return renderResult{}, errors.Errorf("unknown field %q", cond.Field)
}
column := field.columnExpr(r.dialect)
switch r.dialect {
case DialectPostgres:
// POSIX regex match operator.
return renderResult{sql: fmt.Sprintf("%s ~ %s", column, r.addArg(cond.Pattern))}, nil
case DialectMySQL, DialectSQLite:
// MySQL has a native REGEXP operator; SQLite uses the registered regexp() function.
return renderResult{sql: fmt.Sprintf("%s REGEXP %s", column, r.addArg(cond.Pattern))}, nil
default:
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
}
}
// foldedLike renders a case-insensitive LIKE comparison of colExpr against a
// (already metacharacter-escaped) pattern, using each dialect's case-folding.
func (r *renderer) foldedLike(colExpr, pattern string) string {
switch r.dialect { switch r.dialect {
case DialectSQLite: case DialectSQLite:
// Use custom Unicode-aware case folding function for case-insensitive comparison. // memos_unicode_lower gives Unicode-aware folding; ESCAPE '\' is required
// This overcomes SQLite's ASCII-only LOWER() limitation. // because SQLite has no default LIKE escape character.
sql := fmt.Sprintf("memos_unicode_lower(%s) LIKE memos_unicode_lower(%s)", column, r.addArg(arg)) return fmt.Sprintf(`memos_unicode_lower(%s) LIKE memos_unicode_lower(%s) ESCAPE '\'`, colExpr, r.addArg(pattern))
return renderResult{sql: sql}, nil
case DialectPostgres: case DialectPostgres:
sql := fmt.Sprintf("%s ILIKE %s", column, r.addArg(arg)) // ILIKE is case-insensitive; backslash is the default escape character.
return renderResult{sql: sql}, nil return fmt.Sprintf("%s ILIKE %s", colExpr, r.addArg(pattern))
default: default: // MySQL: default collation is case-insensitive; backslash is the default escape.
sql := fmt.Sprintf("%s LIKE %s", column, r.addArg(arg)) return fmt.Sprintf("%s LIKE %s", colExpr, r.addArg(pattern))
return renderResult{sql: sql}, nil
} }
} }
// likePattern escapes LIKE metacharacters in value and wraps it for the mode.
func likePattern(mode TextMatchMode, value string) string {
escaped := escapeLikeLiteral(value)
switch mode {
case TextMatchPrefix:
return escaped + "%"
case TextMatchSuffix:
return "%" + escaped
default:
return "%" + escaped + "%"
}
}
// escapeLikeLiteral escapes the LIKE metacharacters \, %, and _ so user input
// is matched literally. Backslash is the escape character on all three dialects.
func escapeLikeLiteral(s string) string {
return strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(s)
}
func (r *renderer) renderListComprehension(cond *ListComprehensionCondition) (renderResult, error) { func (r *renderer) renderListComprehension(cond *ListComprehensionCondition) (renderResult, error) {
field, ok := r.schema.Field(cond.Field) field, ok := r.schema.Field(cond.Field)
if !ok { if !ok {
@@ -478,6 +521,10 @@ func (r *renderer) renderListComprehension(cond *ListComprehensionCondition) (re
return renderResult{}, errors.Errorf("field %q is not a JSON list", cond.Field) return renderResult{}, errors.Errorf("field %q is not a JSON list", cond.Field)
} }
if cond.Kind == ComprehensionAll {
return r.renderTagAll(field, cond.Predicate)
}
// Render based on predicate type // Render based on predicate type
switch pred := cond.Predicate.(type) { switch pred := cond.Predicate.(type) {
case *EqualsPredicate: case *EqualsPredicate:
@@ -493,6 +540,51 @@ func (r *renderer) renderListComprehension(cond *ListComprehensionCondition) (re
} }
} }
// renderTagAll renders tags.all(t, <pred>): the array is non-empty AND no element
// fails the predicate. Element predicates use plain CEL semantics (case-insensitive
// for startsWith/endsWith/contains, case-sensitive for ==), evaluated per element.
func (r *renderer) renderTagAll(field Field, pred PredicateExpr) (renderResult, error) {
arrayExpr := jsonArrayExpr(r.dialect, field)
elemCond, err := r.elementPredicateSQL(pred)
if err != nil {
return renderResult{}, err
}
switch r.dialect {
case DialectSQLite:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND %s != '[]'", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM json_each(%s) WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
case DialectMySQL:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND JSON_LENGTH(%s) > 0", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM JSON_TABLE(%s, '$[*]' COLUMNS (value VARCHAR(512) PATH '$')) AS elem WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
case DialectPostgres:
nonEmpty := fmt.Sprintf("%s IS NOT NULL AND jsonb_array_length(%s) > 0", arrayExpr, arrayExpr)
sub := fmt.Sprintf("NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(%s) AS elem(value) WHERE NOT (%s))", arrayExpr, elemCond)
return renderResult{sql: fmt.Sprintf("(%s AND %s)", nonEmpty, sub)}, nil
default:
return renderResult{}, errors.Errorf("unsupported dialect %s", r.dialect)
}
}
// elementPredicateSQL builds the per-element SQL condition for an all() predicate.
// The iterated element is exposed as the unqualified column `value` on all dialects
// (json_each.value / JSON_TABLE column / elem(value)).
func (r *renderer) elementPredicateSQL(pred PredicateExpr) (string, error) {
switch p := pred.(type) {
case *EqualsPredicate:
return fmt.Sprintf("value = %s", r.addArg(p.Value)), nil
case *StartsWithPredicate:
return r.foldedLike("value", likePattern(TextMatchPrefix, p.Prefix)), nil
case *EndsWithPredicate:
return r.foldedLike("value", likePattern(TextMatchSuffix, p.Suffix)), nil
case *ContainsPredicate:
return r.foldedLike("value", likePattern(TextMatchContains, p.Substring)), nil
default:
return "", errors.Errorf("unsupported predicate %T in all()", pred)
}
}
// renderTagEquals generates SQL for tags.exists(t, t == "value"). // renderTagEquals generates SQL for tags.exists(t, t == "value").
func (r *renderer) renderTagEquals(field Field, value string, _ ComprehensionKind) (renderResult, error) { func (r *renderer) renderTagEquals(field Field, value string, _ ComprehensionKind) (renderResult, error) {
arrayExpr := jsonArrayExpr(r.dialect, field) arrayExpr := jsonArrayExpr(r.dialect, field)
+8 -5
View File
@@ -256,6 +256,7 @@ func NewSchema() Schema {
cel.Variable("has_code", cel.BoolType), cel.Variable("has_code", cel.BoolType),
cel.Variable("has_incomplete_tasks", cel.BoolType), cel.Variable("has_incomplete_tasks", cel.BoolType),
nowFunction, nowFunction,
cel.ASTValidators(cel.ValidateRegexLiterals()),
} }
return Schema{ return Schema{
@@ -277,11 +278,12 @@ func NewAttachmentSchema() Schema {
Expressions: map[DialectName]string{}, Expressions: map[DialectName]string{},
}, },
"mime_type": { "mime_type": {
Name: "mime_type", Name: "mime_type",
Kind: FieldKindScalar, Kind: FieldKindScalar,
Type: FieldTypeString, Type: FieldTypeString,
Column: Column{Table: "attachment", Name: "type"}, Column: Column{Table: "attachment", Name: "type"},
Expressions: map[DialectName]string{}, SupportsContains: true,
Expressions: map[DialectName]string{},
}, },
"create_time": { "create_time": {
Name: "create_time", Name: "create_time",
@@ -315,6 +317,7 @@ func NewAttachmentSchema() Schema {
cel.Variable("create_time", cel.IntType), cel.Variable("create_time", cel.IntType),
cel.Variable("memo_id", cel.AnyType), cel.Variable("memo_id", cel.AnyType),
nowFunction, nowFunction,
cel.ASTValidators(cel.ValidateRegexLiterals()),
} }
return Schema{ return Schema{
+60
View File
@@ -5,6 +5,8 @@ package sqlite
import ( import (
"database/sql/driver" "database/sql/driver"
"errors"
"regexp"
"sync" "sync"
"golang.org/x/text/cases" "golang.org/x/text/cases"
@@ -42,3 +44,61 @@ func ensureUnicodeLowerRegistered() error {
}) })
return registerUnicodeLowerErr return registerUnicodeLowerErr
} }
var (
registerRegexpOnce sync.Once
registerRegexpErr error
// regexpCache memoizes compiled patterns; keys are pattern strings.
regexpCache sync.Map
)
// ensureRegexpRegistered registers a Go-backed `regexp(pattern, value)` scalar
// function so SQLite's `value REGEXP pattern` operator works (modernc.org/sqlite
// has no built-in implementation). Patterns use Go's RE2 syntax. Registered once
// globally; safe to call multiple times.
func ensureRegexpRegistered() error {
registerRegexpOnce.Do(func() {
registerRegexpErr = msqlite.RegisterScalarFunction("regexp", 2, func(_ *msqlite.FunctionContext, args []driver.Value) (driver.Value, error) {
if len(args) != 2 || args[0] == nil || args[1] == nil {
return int64(0), nil
}
pattern, ok := args[0].(string)
if !ok {
return nil, errors.New("regexp pattern must be a string")
}
var value string
switch v := args[1].(type) {
case string:
value = v
case []byte:
value = string(v)
default:
return int64(0), nil
}
re, err := compileRegexp(pattern)
if err != nil {
return nil, err
}
if re.MatchString(value) {
return int64(1), nil
}
return int64(0), nil
})
})
return registerRegexpErr
}
// compileRegexp compiles and caches a RE2 pattern.
func compileRegexp(pattern string) (*regexp.Regexp, error) {
if cached, ok := regexpCache.Load(pattern); ok {
if re, ok := cached.(*regexp.Regexp); ok {
return re, nil
}
}
re, err := regexp.Compile(pattern)
if err != nil {
return nil, err
}
regexpCache.Store(pattern, re)
return re, nil
}
+24
View File
@@ -0,0 +1,24 @@
package sqlite
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestRegexpFunctionMatches(t *testing.T) {
require.NoError(t, ensureRegexpRegistered())
re, err := compileRegexp(`^v\d+$`)
require.NoError(t, err)
require.True(t, re.MatchString("v12"))
require.False(t, re.MatchString("version"))
// Caching returns the same compiled instance.
re2, err := compileRegexp(`^v\d+$`)
require.NoError(t, err)
require.Same(t, re, re2)
_, err = compileRegexp(`(`)
require.Error(t, err)
}
+4
View File
@@ -31,6 +31,10 @@ func NewDB(profile *profile.Profile) (store.Driver, error) {
return nil, errors.Wrap(err, "failed to register sqlite unicode lower function") return nil, errors.Wrap(err, "failed to register sqlite unicode lower function")
} }
if err := ensureRegexpRegistered(); err != nil {
return nil, errors.Wrap(err, "failed to register sqlite regexp function")
}
// Connect to the database with some sane settings: // Connect to the database with some sane settings:
// - No shared-cache: it's obsolete; WAL journal mode is a better solution. // - No shared-cache: it's obsolete; WAL journal mode is a better solution.
// - No foreign key constraints: it's currently disabled by default, but it's a // - No foreign key constraints: it's currently disabled by default, but it's a
+36
View File
@@ -35,6 +35,42 @@ func TestAttachmentFilterFilenameContains(t *testing.T) {
require.Len(t, attachments, 0) require.Len(t, attachments, 0)
} }
func TestAttachmentFilterFilenameEndsWith(t *testing.T) {
t.Parallel()
tc := NewAttachmentFilterTestContext(t)
defer tc.Close()
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("report.pdf").MimeType("application/pdf"))
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("photo.png").MimeType("image/png"))
got := tc.ListWithFilter(`filename.endsWith(".pdf")`)
require.Len(t, got, 1)
require.Equal(t, "report.pdf", got[0].Filename)
// matches() on mime_type, anchored.
got = tc.ListWithFilter(`mime_type.matches("^image/")`)
require.Len(t, got, 1)
require.Equal(t, "photo.png", got[0].Filename)
}
func TestAttachmentFilterFilenameStartsWith(t *testing.T) {
t.Parallel()
tc := NewAttachmentFilterTestContext(t)
defer tc.Close()
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("invoice-2026.pdf").MimeType("application/pdf"))
tc.CreateAttachment(NewAttachmentBuilder(tc.CreatorID).Filename("photo.png").MimeType("image/png"))
got := tc.ListWithFilter(`filename.startsWith("invoice")`)
require.Len(t, got, 1)
require.Equal(t, "invoice-2026.pdf", got[0].Filename)
// mime_type prefix matching (newly enabled).
got = tc.ListWithFilter(`mime_type.startsWith("image/")`)
require.Len(t, got, 1)
require.Equal(t, "photo.png", got[0].Filename)
}
func TestAttachmentFilterFilenameSpecialCharacters(t *testing.T) { func TestAttachmentFilterFilenameSpecialCharacters(t *testing.T) {
t.Parallel() t.Parallel()
tc := NewAttachmentFilterTestContext(t) tc := NewAttachmentFilterTestContext(t)
+184
View File
@@ -10,6 +10,14 @@ import (
"github.com/usememos/memos/store" "github.com/usememos/memos/store"
) )
func uids(memos []*store.Memo) []string {
out := make([]string, 0, len(memos))
for _, m := range memos {
out = append(out, m.UID)
}
return out
}
// ============================================================================= // =============================================================================
// Content Field Tests // Content Field Tests
// Schema: content (string, supports contains) // Schema: content (string, supports contains)
@@ -94,6 +102,182 @@ func TestMemoFilterContentCaseSensitivity(t *testing.T) {
} }
} }
func TestMemoFilterTagsAll(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-all-work", tc.User.ID).Content("all work").Tags("work/a", "work/b"))
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed").Tags("work/a", "home"))
tc.CreateMemo(NewMemoBuilder("memo-untagged", tc.User.ID).Content("untagged"))
// Every tag starts with "work/": only the all-work memo qualifies.
memos := tc.ListWithFilter(`tags.all(t, t.startsWith("work/"))`)
require.Len(t, memos, 1)
require.Equal(t, "memo-all-work", memos[0].UID)
// Untagged memos must NOT match (non-empty guard, decision B).
require.NotContains(t, uids(memos), "memo-untagged")
}
func TestMemoFilterTagsAllEquals(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-only-x", tc.User.ID).Content("only x").Tags("x", "x"))
tc.CreateMemo(NewMemoBuilder("memo-x-and-y", tc.User.ID).Content("x and y").Tags("x", "y"))
memos := tc.ListWithFilter(`tags.all(t, t == "x")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-only-x", memos[0].UID)
}
func TestMemoFilterContentStartsWithEscaping(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-pct", tc.User.ID).Content("100% complete"))
tc.CreateMemo(NewMemoBuilder("memo-plain", tc.User.ID).Content("100 things to do"))
// The % must be treated literally, not as a LIKE wildcard.
memos := tc.ListWithFilter(`content.startsWith("100%")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-pct", memos[0].UID)
}
func TestMemoFilterStartsWithCombinedAndNegated(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-pub", tc.User.ID).Content("Hello public").Visibility(store.Public))
tc.CreateMemo(NewMemoBuilder("memo-priv", tc.User.ID).Content("Hello private").Visibility(store.Private))
tc.CreateMemo(NewMemoBuilder("memo-bye", tc.User.ID).Content("Goodbye public").Visibility(store.Public))
memos := tc.ListWithFilter(`content.startsWith("Hello") && visibility == "PUBLIC"`)
require.Len(t, memos, 1)
require.Equal(t, "memo-pub", memos[0].UID)
memos = tc.ListWithFilter(`!content.startsWith("Hello")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-bye", memos[0].UID)
}
func TestMemoFilterContentMatchesAdvanced(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-release", tc.User.ID).Content("release v2024 notes"))
tc.CreateMemo(NewMemoBuilder("memo-draft", tc.User.ID).Content("draft document"))
tc.CreateMemo(NewMemoBuilder("memo-animals", tc.User.ID).Content("cat and dog"))
// Anchor: starts with "release".
memos := tc.ListWithFilter(`content.matches("^release")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-release", memos[0].UID)
// Character class + quantifier: a 4-digit run (portable across RE2/POSIX/ICU).
memos = tc.ListWithFilter(`content.matches("[0-9]{4}")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-release", memos[0].UID)
// Alternation.
memos = tc.ListWithFilter(`content.matches("cat|mouse")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-animals", memos[0].UID)
// No match.
memos = tc.ListWithFilter(`content.matches("^zzz")`)
require.Len(t, memos, 0)
}
func TestMemoFilterTagsAllMorePredicates(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-all-done", tc.User.ID).Tags("a/done", "b/done"))
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Tags("a/done", "b/todo"))
tc.CreateMemo(NewMemoBuilder("memo-single", tc.User.ID).Tags("solo/done"))
// endsWith: every tag ends with "done".
memos := tc.ListWithFilter(`tags.all(t, t.endsWith("done"))`)
require.ElementsMatch(t, []string{"memo-all-done", "memo-single"}, uids(memos))
// contains: every tag contains "/".
memos = tc.ListWithFilter(`tags.all(t, t.contains("/"))`)
require.ElementsMatch(t, []string{"memo-all-done", "memo-mixed", "memo-single"}, uids(memos))
}
func TestMemoFilterTagsAllNegatedAndCombined(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-work", tc.User.ID).Content("work memo").Tags("work/a", "work/b"))
tc.CreateMemo(NewMemoBuilder("memo-mixed", tc.User.ID).Content("mixed memo").Tags("work/a", "home"))
tc.CreateMemo(NewMemoBuilder("memo-untagged", tc.User.ID).Content("untagged memo"))
// Negation: NOT all-work. The untagged memo's all() is false (non-empty guard),
// so !all() is true and it is included.
memos := tc.ListWithFilter(`!tags.all(t, t.startsWith("work/"))`)
require.ElementsMatch(t, []string{"memo-mixed", "memo-untagged"}, uids(memos))
// Combined with a content filter.
memos = tc.ListWithFilter(`tags.all(t, t.startsWith("work/")) && content.contains("work")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-work", memos[0].UID)
}
func TestMemoFilterContentStartsWith(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-todo", tc.User.ID).Content("TODO: buy milk"))
tc.CreateMemo(NewMemoBuilder("memo-done", tc.User.ID).Content("Done with milk"))
// Prefix match, case-insensitive (consistent with contains()).
memos := tc.ListWithFilter(`content.startsWith("todo")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-todo", memos[0].UID)
memos = tc.ListWithFilter(`content.startsWith("nope")`)
require.Len(t, memos, 0)
}
func TestMemoFilterContentEndsWith(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-md", tc.User.ID).Content("notes.md"))
tc.CreateMemo(NewMemoBuilder("memo-txt", tc.User.ID).Content("notes.txt"))
memos := tc.ListWithFilter(`content.endsWith(".md")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-md", memos[0].UID)
}
func TestMemoFilterContentMatches(t *testing.T) {
t.Parallel()
tc := NewMemoFilterTestContext(t)
defer tc.Close()
tc.CreateMemo(NewMemoBuilder("memo-v1", tc.User.ID).Content("release v12 shipped"))
tc.CreateMemo(NewMemoBuilder("memo-plain", tc.User.ID).Content("no version here"))
memos := tc.ListWithFilter(`content.matches("v[0-9]+")`)
require.Len(t, memos, 1)
require.Equal(t, "memo-v1", memos[0].UID)
memos = tc.ListWithFilter(`content.matches("^xyz")`)
require.Len(t, memos, 0)
}
// ============================================================================= // =============================================================================
// Visibility Field Tests // Visibility Field Tests
// Schema: visibility (string, ==, !=) // Schema: visibility (string, ==, !=)
+22
View File
@@ -85,14 +85,36 @@ const shortcutExamples = [
description: "Search text inside memo content.", description: "Search text inside memo content.",
icon: SearchIcon, icon: SearchIcon,
}, },
{
title: "Starts with",
filter: 'content.startsWith("TODO")',
description: "Memos whose content begins with text (also endsWith).",
icon: SearchIcon,
},
{
title: "Regex match",
filter: 'content.matches("v[0-9]+")',
description: "Match content with a regular expression.",
icon: FilterIcon,
},
{
title: "All tags match",
filter: 'tags.all(t, t.startsWith("work/"))',
description: "Every tag must satisfy the predicate (tagged memos only).",
icon: TagsIcon,
},
]; ];
const filterFields = [ const filterFields = [
"content.contains(...)", "content.contains(...)",
"content.startsWith(...)",
"content.endsWith(...)",
"content.matches(...)",
"visibility", "visibility",
"pinned", "pinned",
"tag in [...]", "tag in [...]",
"tags.exists(...)", "tags.exists(...)",
"tags.all(...)",
"has_task_list", "has_task_list",
"has_incomplete_tasks", "has_incomplete_tasks",
"has_link", "has_link",