memos/internal/filter/README.md

Memo Filter Engine

This package houses the memo-only filter engine that turns standard CEL syntax into SQL fragments for the subset of expressions supported by the memo schema. The engine follows a three phase pipeline inspired by systems such as Calcite or Prisma:

1. Parsing – CEL expressions are parsed with cel-go and validated against the memo-specific environment declared in schema.go. Only fields that exist in the schema can surface in the filter, and non-standard legacy coercions are rejected.
2. Normalization – the raw CEL AST is converted into an intermediate representation (IR) defined in ir.go. The IR is a dialect-agnostic tree of conditions (logical operators, comparisons, list membership, etc.). This step enforces schema rules (e.g. operator compatibility, type checks).
3. Rendering – the renderer in render.go walks the IR and produces a SQL fragment plus placeholder arguments tailored to a target dialect (sqlite, mysql, or postgres). Dialect differences such as JSON access, boolean semantics, placeholders, and LIKE vs ILIKE are encapsulated in renderer helpers.

The entry point is filter.DefaultEngine() from engine.go. It lazily constructs an Engine configured with the memo schema and exposes:

engine, _ := filter.DefaultEngine()
stmt, _ := engine.CompileToStatement(ctx, `has_task_list && visibility == "PUBLIC"`, filter.RenderOptions{
	Dialect: filter.DialectPostgres,
})
// stmt.SQL  -> "((memo.payload->'property'->>'hasTaskList')::boolean IS TRUE AND memo.visibility = $1)"
// stmt.Args -> ["PUBLIC"]

Core Files

File	Responsibility
schema.go	Declares memo fields, their types, backing columns, CEL environment options
ir.go	IR node definitions used across the pipeline
parser.go	Converts CEL Expr into IR while applying schema validation
render.go	Translates IR into SQL, handling dialect-specific behavior
engine.go	Glue between the phases; exposes Compile, CompileToStatement, and DefaultEngine
helpers.go	Convenience helpers for store integration (appending conditions)

SQL Generation Notes

• Placeholders — ? is used for SQLite/MySQL, $n for Postgres. The renderer tracks offsets to compose queries with pre-existing arguments.
• JSON Fields — Memo metadata lives in memo.payload. The renderer handles JSON_EXTRACT/json_extract/->/->> variations and boolean coercion.
• Tag Operations — tag in [...] and "tag" in tags become JSON array predicates. SQLite uses LIKE patterns, MySQL uses JSON_CONTAINS, and Postgres uses @>.
• Boolean Flags — Fields such as has_task_list render as IS TRUE equality checks, or comparisons against CAST('true' AS JSON) depending on the dialect.
• String Matching — content.contains(x), content.startsWith(x), and content.endsWith(x) render as case-insensitive LIKE/ILIKE with LIKE metacharacters (%, _, \) escaped. Available on scalar string fields whose schema sets SupportsContains (memo content; attachment filename, mime_type).
• Regex — field.matches("pattern") renders to ~ (Postgres) or REGEXP (MySQL/SQLite). SQLite uses a Go-backed regexp function registered in store/db/sqlite/functions.go. Patterns are validated at compile time against Go's RE2 via cel.ValidateRegexLiterals(). Caveat: regex syntax differs per engine (Go RE2 on SQLite, POSIX ERE on Postgres, ICU on MySQL 8.0+), so engine-specific patterns may not be portable.
• Tag all() — tags.all(t, <pred>) matches only non-empty tag sets where every element satisfies the predicate, via per-element iteration (json_each / jsonb_array_elements_text / JSON_TABLE).

Typical Integration

1. Fetch the engine with filter.DefaultEngine().
2. Call CompileToStatement using the appropriate dialect enum.
3. Append the emitted SQL fragment/args to the existing WHERE clause.
4. Execute the resulting query through the store driver.

The helpers.AppendConditions helper encapsulates steps 2–3 when a driver needs to process an array of filters.