feat(config): provision settings from secret files
- Load IdPs and supported instance-setting groups as runtime overlays from /etc/secrets. - Reject API mutations of deployment-managed resources and serialize authentication safety checks across database drivers. - Preserve upgrade compatibility, demo SSO policy, stable IdP ordering, and driver-specific transaction retries.
This commit is contained in:
boojack
@@ -67,6 +67,11 @@ var (
|
||||
slog.Error("failed to migrate", "error", err)
|
||||
return
|
||||
}
|
||||
if err := storeInstance.LoadDeploymentConfiguration(ctx); err != nil {
|
||||
cancel()
|
||||
slog.Error("failed to load deployment configuration", "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
s, err := server.NewServer(ctx, instanceProfile, storeInstance)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,497 @@
|
||||
# Configuration Provisioning
|
||||
|
||||
Status: Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
Memos should follow Mastodon's deployment-configuration model: configuration supplied by the deployment is loaded directly into each server process and
|
||||
remains authoritative for that process lifetime. It is not imported into the database and is not tracked as database-owned application state.
|
||||
|
||||
The first supported file-backed configuration resources are:
|
||||
|
||||
- OAuth2 identity providers.
|
||||
- Instance settings for general policy, storage, memo behavior, notifications, and AI providers.
|
||||
|
||||
Memos scans `/etc/secrets` after database migration and demo seeding, validates every matching file, builds one immutable configuration snapshot, and
|
||||
publishes that snapshot before HTTP or background services start. Applying a changed file requires a process restart.
|
||||
|
||||
Every resource file contains exactly one existing `memos.store` protobuf message encoded as protobuf JSON. No resource envelope, state file, ownership
|
||||
table, or second persistent copy of a secret is introduced. The process necessarily holds decoded secrets in its private runtime snapshot.
|
||||
|
||||
## Design model
|
||||
|
||||
Mastodon reads external authentication and other deployment configuration from environment variables or a dotenv file during process initialization. It
|
||||
does not copy that configuration into an administrator-editable database resource or maintain Terraform-style ownership state.
|
||||
|
||||
Memos should use the same lifecycle while adapting the input format to its existing generated store messages:
|
||||
|
||||
- Mounted JSON files replace a large collection of environment variables.
|
||||
- File-backed resources exist in the effective runtime configuration.
|
||||
- Stored resources continue to exist in the database but are shadowed when a file declares the same stable key.
|
||||
- UI and API mutations cannot change an actively file-backed resource.
|
||||
- Removing a file and restarting removes the runtime override; it does not delete or modify the stored resource.
|
||||
|
||||
This is deployment configuration, not resource reconciliation. Terms such as adoption, import, unmanage, drift, prune, and Terraform state do not apply.
|
||||
|
||||
## Goals
|
||||
|
||||
- Accept secrets through mounted files without committing them to seed SQL or command-line arguments.
|
||||
- Keep each file equal to one generated store protobuf message.
|
||||
- Load and validate the complete file set before exposing any of it.
|
||||
- Make deployment configuration authoritative for the lifetime of the process.
|
||||
- Preserve database-backed UI configuration for keys not supplied by files.
|
||||
- Prevent API writes from appearing to change an effective file-backed resource.
|
||||
- Keep secret values out of logs, API responses, caches that expose values, and additional persistence.
|
||||
- Preserve the administrator password sign-in path when password sign-in is disabled for regular users.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Persist file contents or file ownership metadata in the database.
|
||||
- Add a `provisioning_resource` table or provisioning columns to existing tables.
|
||||
- Reconcile database state to match a desired resource graph.
|
||||
- Delete database resources when files disappear.
|
||||
- Support multiple configuration sources with precedence rules in the first version.
|
||||
- Write UI changes back into mounted files.
|
||||
- Watch files or reload configuration without restarting in the first version.
|
||||
- Support partial field ownership within an instance-setting group.
|
||||
|
||||
## Terminology
|
||||
|
||||
**Stored configuration**
|
||||
: Configuration stored in the existing `idp` and `system_setting` database tables.
|
||||
|
||||
**Deployment configuration**
|
||||
: Configuration decoded from matching files during process startup.
|
||||
|
||||
**Effective configuration**
|
||||
: The configuration used by APIs, authentication, and background services. Deployment configuration shadows stored configuration with the same stable key.
|
||||
|
||||
**Stable key**
|
||||
: The identity-provider UID or instance-setting key used to merge deployment and stored configuration.
|
||||
|
||||
## File discovery
|
||||
|
||||
Memos scans direct children of `/etc/secrets`. The directory may contain unrelated platform secrets; only supported filename patterns are read. Memos does
|
||||
not recurse into subdirectories and does not create, modify, or delete anything in the directory.
|
||||
|
||||
| Filename pattern | Protobuf message | Stable key |
|
||||
| --- | --- | --- |
|
||||
| `memos-idp-<label>.json` | `memos.store.IdentityProvider` | `uid` |
|
||||
| `memos-instance-setting-<label>.json` | `memos.store.InstanceSetting` | `key` |
|
||||
|
||||
`<label>` uses lowercase kebab case and must match `[a-z0-9]+(?:-[a-z0-9]+)*`. Matching is case-sensitive and the extension is lowercase `.json`.
|
||||
For upgrade compatibility, identity-provider filenames accepted by the original bootstrap (`memos-idp-*.json`) continue to load when the label is not
|
||||
lowercase kebab case, but startup logs a deprecation warning. New files should always use the canonical convention.
|
||||
|
||||
Recommended labels mirror the resource key for operator readability:
|
||||
|
||||
| Resource | Canonical filename |
|
||||
| --- | --- |
|
||||
| Identity provider with UID `primary-sso` | `memos-idp-primary-sso.json` |
|
||||
| `GENERAL` | `memos-instance-setting-general.json` |
|
||||
| `STORAGE` | `memos-instance-setting-storage.json` |
|
||||
| `MEMO_RELATED` | `memos-instance-setting-memo-related.json` |
|
||||
| `NOTIFICATION` | `memos-instance-setting-notification.json` |
|
||||
| `AI` | `memos-instance-setting-ai.json` |
|
||||
|
||||
The filename label remains descriptive rather than authoritative. The `uid` or `key` inside the message is the resource identity, so renaming a file does
|
||||
not change account links or effective resource identity. Files are read in lexical order only to produce deterministic diagnostics; ordering has no
|
||||
configuration semantics.
|
||||
|
||||
Each matching file:
|
||||
|
||||
- Must be a valid protobuf JSON representation of the expected message.
|
||||
- Must not contain unknown fields.
|
||||
- Must not exceed 1 MiB.
|
||||
- Must contain exactly one resource.
|
||||
- May contain plaintext secrets because the containing directory is treated as sensitive.
|
||||
- May be a regular file or a platform-managed symlink that resolves to a regular file, as used by Kubernetes Secret volumes.
|
||||
|
||||
A missing directory or a readable directory without matching files is a normal no-op. An unreadable directory or matching file is a startup error. Startup
|
||||
logs include matched counts by resource type so a misspelled filename is visible without logging file contents. A direct child beginning with `memos-` but
|
||||
not matching a supported pattern produces a warning; unrelated filenames are silently ignored.
|
||||
|
||||
### SSO-only deployments
|
||||
|
||||
An SSO-only deployment mounts both an identity-provider file and `memos-instance-setting-general.json` with `disallowPasswordAuth` enabled. These resources
|
||||
are validated and published together during startup. The public demo seed does not contain authentication policy, so both files must be mounted to enable
|
||||
SSO-only behavior. Keep `disallowUserRegistration` disabled when first-time SSO users should be created automatically.
|
||||
|
||||
## Identity-provider files
|
||||
|
||||
An identity-provider file contains exactly one `memos.store.IdentityProvider`. The database-generated `id` must be omitted. `uid` is required and is the
|
||||
stable key.
|
||||
|
||||
Example `/etc/secrets/memos-idp-primary-sso.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"uid": "primary-sso",
|
||||
"name": "Company SSO",
|
||||
"type": "OAUTH2",
|
||||
"identifierFilter": "",
|
||||
"config": {
|
||||
"oauth2Config": {
|
||||
"clientId": "client-id",
|
||||
"clientSecret": "client-secret",
|
||||
"authUrl": "https://idp.example.com/oauth/authorize",
|
||||
"tokenUrl": "https://idp.example.com/oauth/token",
|
||||
"userInfoUrl": "https://idp.example.com/oauth/userinfo",
|
||||
"scopes": ["openid", "profile", "email"],
|
||||
"fieldMapping": {
|
||||
"identifier": "sub",
|
||||
"displayName": "name",
|
||||
"email": "email",
|
||||
"avatarUrl": "picture"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Initial validation supports only OAuth2 providers and requires:
|
||||
|
||||
- A valid, nonempty UID and display name.
|
||||
- Client ID and client secret.
|
||||
- Authorization, token, and user-info URLs.
|
||||
- At least one scope, with no empty scope entries.
|
||||
- A field-mapping object with a nonempty identifier field.
|
||||
|
||||
Duplicate UIDs across files are rejected.
|
||||
|
||||
User identity links already use the provider UID as their stable provider value. A file-backed provider therefore does not need a database-generated IdP ID
|
||||
to preserve account links or complete SSO sign-in.
|
||||
|
||||
## Instance-setting files
|
||||
|
||||
An instance-setting file contains exactly one `memos.store.InstanceSetting`. `key` is required and is the stable key. The populated `oneof` must match the
|
||||
key.
|
||||
|
||||
Example `/etc/secrets/memos-instance-setting-general.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "GENERAL",
|
||||
"generalSetting": {
|
||||
"disallowUserRegistration": false,
|
||||
"disallowPasswordAuth": true,
|
||||
"additionalScript": "",
|
||||
"additionalStyle": "",
|
||||
"weekStartDayOffset": 1,
|
||||
"disallowChangeUsername": false,
|
||||
"disallowChangeNickname": false,
|
||||
"customProfile": {
|
||||
"title": "Company Memos",
|
||||
"description": "Internal notes",
|
||||
"logoUrl": "https://example.com/logo.png"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Example `/etc/secrets/memos-instance-setting-notification.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "NOTIFICATION",
|
||||
"notificationSetting": {
|
||||
"email": {
|
||||
"enabled": true,
|
||||
"smtpHost": "smtp.example.com",
|
||||
"smtpPort": 587,
|
||||
"smtpUsername": "memos",
|
||||
"smtpPassword": "smtp-secret",
|
||||
"fromEmail": "memos@example.com",
|
||||
"fromName": "Memos",
|
||||
"replyTo": "support@example.com",
|
||||
"useTls": true,
|
||||
"useSsl": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Supported keys:
|
||||
|
||||
| Key | Deployment use |
|
||||
| --- | --- |
|
||||
| `GENERAL` | Registration, authentication, branding, scripts, styles, and user-profile policy |
|
||||
| `STORAGE` | Attachment storage type, limits, paths, and S3 credentials |
|
||||
| `MEMO_RELATED` | Memo limits, editing behavior, and reactions |
|
||||
| `NOTIFICATION` | SMTP transport and credentials |
|
||||
| `AI` | AI providers, API keys, and transcription defaults |
|
||||
|
||||
Rejected keys:
|
||||
|
||||
- `BASIC` contains the instance secret key and database schema version. Replacing the secret key invalidates sessions, while replacing the schema version
|
||||
interferes with database migration state.
|
||||
- `TAGS` is retained for backward compatibility; active tag metadata is stored per user.
|
||||
|
||||
Only one file may declare a given setting key.
|
||||
|
||||
### Complete-group replacement
|
||||
|
||||
An instance-setting group is the smallest deployment-configured unit. A file replaces the complete effective group. A scalar omitted from protobuf JSON is
|
||||
stored in the decoded message as its protobuf default; omission does not preserve a field from the database value.
|
||||
|
||||
Some existing setting getters apply application defaults after decoding zero values. For example, STORAGE defaults to local storage, a 30 MiB upload limit,
|
||||
and `assets/{timestamp}_{uuid}_{filename}` when the corresponding decoded fields are unspecified. The effective behavior is therefore the decoded file plus
|
||||
the same read-time defaults used for database-backed configuration.
|
||||
|
||||
Empty secret fields in a file mean empty values; they never mean "preserve the database secret." Credential-preservation behavior used by UI updates does
|
||||
not apply to deployment configuration.
|
||||
|
||||
### AI normalization
|
||||
|
||||
AI deployment configuration uses deterministic, self-contained normalization rather than the UI update path:
|
||||
|
||||
- Every provider requires an explicit stable `id`; the loader never generates one.
|
||||
- Every provider requires a title, a supported provider type, and an API key.
|
||||
- An empty OpenAI endpoint becomes `https://api.openai.com/v1`.
|
||||
- An empty Gemini endpoint becomes `https://generativelanguage.googleapis.com/v1beta`.
|
||||
- Duplicate provider IDs are rejected.
|
||||
- A transcription provider ID must reference a provider in the same effective AI setting.
|
||||
- Model, language, and prompt use the same length limits as API-managed settings.
|
||||
- No provider, API key, or transcription value is copied from the shadowed database setting.
|
||||
|
||||
## Configuration format compatibility
|
||||
|
||||
Using store protobuf JSON makes the selected messages a supported deployment-configuration interface even though the messages remain internal to the
|
||||
application. For every provisionable message, Memos must preserve:
|
||||
|
||||
- Existing protobuf JSON field names.
|
||||
- Existing enum names and meanings.
|
||||
- Stable resource-key and `oneof` mappings.
|
||||
- Previously valid omissions for fields that have defaults.
|
||||
|
||||
New optional fields and enum values may be added. A provisionable field may be deprecated, but its existing JSON spelling must continue to decode for the
|
||||
supported upgrade window. Field names must not be reused with a different meaning. New validation should not invalidate an existing safe configuration
|
||||
without an upgrade note and a documented replacement.
|
||||
|
||||
Unknown fields remain startup errors because this catches misspellings and configuration written for a newer, incompatible Memos version. Compatibility
|
||||
tests should keep representative JSON fixtures from earlier releases and decode them with the current loader.
|
||||
|
||||
## Runtime configuration snapshot
|
||||
|
||||
The loader builds an immutable snapshot containing maps keyed by provider UID and setting key. It does not mutate the database while loading. The `Store`
|
||||
owns this snapshot so all existing consumers resolve configuration through one boundary.
|
||||
|
||||
Startup follows this sequence:
|
||||
|
||||
```text
|
||||
Initialize or migrate database
|
||||
-> apply demo seed when enabled
|
||||
-> read all matching deployment-configuration files
|
||||
-> decode and validate every resource
|
||||
-> validate affected cross-resource invariants
|
||||
-> publish one immutable runtime snapshot
|
||||
-> construct HTTP and background services
|
||||
-> accept requests
|
||||
```
|
||||
|
||||
If any matching file is invalid, no snapshot is published and startup fails. Atomicity comes from publishing the snapshot only after complete validation;
|
||||
no cross-database transaction abstraction is required because deployment configuration performs no database writes.
|
||||
|
||||
The snapshot is loaded once. Files changed after startup have no effect until the process restarts.
|
||||
|
||||
### Immutability and copy semantics
|
||||
|
||||
Generated protobuf messages are mutable pointers, so immutability must be enforced rather than assumed:
|
||||
|
||||
- Canonical snapshot messages remain private to the `Store`.
|
||||
- Effective getters return deep clones, using `proto.Clone`, rather than canonical pointers.
|
||||
- Read-time defaults and redaction are applied only to clones.
|
||||
- Canonical snapshot messages are never inserted into the existing instance-setting TTL cache.
|
||||
- Callers cannot obtain a mutable map or message owned by the snapshot.
|
||||
|
||||
This prevents one request, background runner, defaulting helper, or redaction path from changing configuration observed by another goroutine.
|
||||
|
||||
### Effective and stored access
|
||||
|
||||
The store facade has an explicit separation between effective reads and stored-resource access:
|
||||
|
||||
- Normal list/get operations used by authentication, APIs, and background services return effective configuration.
|
||||
- Internal raw list/get operations read only the database and are used by migration, snapshot planning, and permitted mutation paths.
|
||||
- Mutation services check the snapshot source before loading a raw database row.
|
||||
- A file-backed IdP has no database ID and must never be passed to a driver update or delete operation.
|
||||
- The loader reads stored configuration through raw access before publishing the snapshot, avoiding recursive effective resolution.
|
||||
|
||||
## Effective configuration resolution
|
||||
|
||||
### Identity providers
|
||||
|
||||
List and get operations return the union of stored and file-backed providers by UID:
|
||||
|
||||
- A file-backed provider shadows a stored provider with the same UID.
|
||||
- Stored providers with other UIDs remain available.
|
||||
- Authentication resolves the same effective provider collection.
|
||||
- Removing the file and restarting reveals any stored provider that had been shadowed; it does not restore values from the file.
|
||||
- Stored providers retain their database insertion order, and a file-backed provider that shadows one occupies the same position. Providers that exist only
|
||||
in deployment configuration are appended in UID order, keeping existing API and sign-in ordering stable while remaining deterministic.
|
||||
|
||||
Operators migrating an existing stored provider to a file should keep the same UID so existing user-identity links continue to work. They should remove or
|
||||
update the shadowed stored provider before later removing the file if they do not want the old database configuration to reappear.
|
||||
|
||||
### Instance settings
|
||||
|
||||
Every effective instance-setting getter checks the runtime snapshot before its database cache:
|
||||
|
||||
- A file-backed group completely shadows the `system_setting` row with the same key.
|
||||
- Other setting groups continue to use stored values and existing application defaults.
|
||||
- The runtime snapshot must never be overwritten by a cached database value.
|
||||
- Removing a file and restarting returns the group to its stored database value.
|
||||
|
||||
The demo seed writes `MEMO_RELATED` but does not write `GENERAL`. Loading deployment configuration after seeding supplies the complete effective General
|
||||
settings without embedding deployment authentication policy in demo data.
|
||||
|
||||
## Validation and authentication safety
|
||||
|
||||
All file-local validation runs before snapshot publication. Relationships between file-backed resources are validated against the resulting effective
|
||||
configuration when the desired files affect that relationship.
|
||||
|
||||
At minimum, validation rejects:
|
||||
|
||||
- A file-backed `GENERAL` setting that disables password authentication for regular users when the resulting effective configuration has no identity
|
||||
provider.
|
||||
- An instance-setting key whose populated `oneof` does not match the key.
|
||||
- S3 storage without the required endpoint, bucket, region, or credentials.
|
||||
- Enabled email delivery without the required SMTP host, port, or sender.
|
||||
- Duplicate AI provider IDs.
|
||||
- Transcription referencing an AI provider ID absent from the effective AI setting.
|
||||
- Duplicate stable keys across files.
|
||||
|
||||
An unrelated file must not turn an existing database condition into a new startup failure. For example, a STORAGE-only file does not fail startup merely
|
||||
because the database already disables regular-user password sign-in while containing no IdP; Memos logs that existing condition as a warning. A file that
|
||||
configures GENERAL or an IdP evaluates the authentication invariant against the resulting effective state.
|
||||
|
||||
The administrator password path remains available regardless of `disallowPasswordAuth`. Runtime mutations reject transitions from a safe authentication
|
||||
state to one where password sign-in is disabled for regular users without an effective IdP. An unrelated edit may preserve an already-existing legacy
|
||||
violation so an upgrade does not make the complete `GENERAL` group uneditable; the administrator can resolve that state by enabling password sign-in or
|
||||
configuring an IdP. Deleting the last effective IdP from a previously safe state remains rejected.
|
||||
|
||||
The validation and database mutation must be one serializable store operation. In particular, updating `GENERAL` and deleting an IdP cannot use separate
|
||||
check-then-write calls, because concurrent requests could each validate an old safe state and together produce an unsafe state. The narrow runtime mutation
|
||||
operation:
|
||||
|
||||
1. Starts a serializable database transaction.
|
||||
2. Reads the stored `GENERAL` setting and stored IdPs inside that transaction.
|
||||
3. Combines them with the immutable file snapshot and the proposed mutation.
|
||||
4. Validates the resulting effective authentication state.
|
||||
5. Applies the stored-resource mutation and commits.
|
||||
6. Retries serialization conflicts a bounded number of times.
|
||||
|
||||
This transaction is required for runtime authentication-policy safety, not for loading deployment files. It adds no table or schema migration. Each
|
||||
database driver must provide equivalent transaction semantics for this narrow operation.
|
||||
|
||||
## API behavior
|
||||
|
||||
The API operates on effective resources for reads and stored resources for permitted writes.
|
||||
|
||||
Mutation behavior:
|
||||
|
||||
- Creating a stored IdP with a UID reserved by a file-backed provider returns `codes.FailedPrecondition`.
|
||||
- Updating or deleting a file-backed IdP returns `codes.FailedPrecondition`.
|
||||
- Updating a file-backed instance-setting group returns `codes.FailedPrecondition`.
|
||||
- The instance-setting guard runs before validation or future field-mask application, so every masked update to a file-backed group is rejected.
|
||||
- Mutations of unshadowed stored configuration continue normally, subject to authentication safety invariants.
|
||||
- API responses continue to redact client secrets, SMTP passwords, S3 secrets, and AI API keys.
|
||||
|
||||
No API operation writes to the mounted files. Test operations that do not change stored configuration, such as testing the effective SMTP configuration,
|
||||
remain available.
|
||||
|
||||
## Frontend behavior
|
||||
|
||||
The frontend does not receive configuration-source metadata. It presents the normal mutation controls and reports the API's `FailedPrecondition` error when
|
||||
an administrator attempts to create, update, or delete a deployment-managed resource. The API remains the sole authority for mutation enforcement.
|
||||
|
||||
## Security
|
||||
|
||||
- Treat `/etc/secrets` and every matching file as sensitive plaintext.
|
||||
- Recommend owner-only or application-group-readable filesystem permissions.
|
||||
- Never log file contents, decoded messages, before/after values, or secret fields.
|
||||
- Redact secrets from validation errors and startup summaries.
|
||||
- Do not persist deployment secrets in `idp`, `system_setting`, a state file, or ownership metadata.
|
||||
- Fail startup on an invalid or unreadable matching file rather than publishing partial configuration.
|
||||
- Keep the immutable snapshot process-local and expose only redacted API representations.
|
||||
|
||||
## Multiple server replicas
|
||||
|
||||
Every replica independently loads deployment configuration at startup, as Mastodon processes independently load environment configuration. All replicas in
|
||||
one deployment must mount identical files.
|
||||
|
||||
A rolling deployment can temporarily run old and new configuration generations at the same time. Memos does not attempt distributed reconciliation or
|
||||
cache invalidation for this process-local configuration. Deployments changing authentication or storage configuration should use a rollout strategy that
|
||||
does not route traffic to replicas with different file generations, and readiness must be reported only after the new snapshot validates successfully.
|
||||
|
||||
Because file-backed settings bypass the database setting cache, a replica cannot replace a deployment value with a stale cached database value.
|
||||
|
||||
## Database and migration impact
|
||||
|
||||
This design requires no database schema changes and no migrations:
|
||||
|
||||
- File-backed IdPs are not inserted into `idp`.
|
||||
- File-backed settings are not inserted into `system_setting`.
|
||||
- Existing user-identity links remain database-backed and continue to reference provider UIDs.
|
||||
- Existing stored configuration remains untouched beneath runtime overrides.
|
||||
|
||||
### Transition from the database-writing bootstrap
|
||||
|
||||
Versions with the original `memos-idp-*.json` bootstrap copied file-backed IdPs, including client secrets, into the `idp` table during migration. The new
|
||||
loader cannot reliably distinguish those rows from providers created through the UI, so it must not delete or scrub them automatically.
|
||||
|
||||
When a file shadows a stored provider with the same UID, startup logs a secret-free warning that a stored copy remains. Operators who previously used the
|
||||
database-writing bootstrap should clean up explicitly:
|
||||
|
||||
1. Back up the database and retain administrator password access.
|
||||
2. Temporarily remove the IdP file and restart Memos so the stored provider is no longer shadowed.
|
||||
3. Delete or update the stored provider through the administrator UI or API, or perform equivalent offline database maintenance.
|
||||
4. Restore the file and restart Memos.
|
||||
|
||||
Until that cleanup is complete, the old stored provider and secret remain in the database and can reappear if the file is removed. The no-persistence
|
||||
guarantee applies to the new loader; it does not claim to erase secrets written by earlier versions.
|
||||
|
||||
## Implementation
|
||||
|
||||
The implementation:
|
||||
|
||||
1. Replaces the database-writing IdP bootstrap with a typed deployment-configuration loader.
|
||||
2. Decodes and validates `memos-instance-setting-*.json` resources.
|
||||
3. Loads configuration after migration and demo seeding but before service construction.
|
||||
4. Publishes an immutable provider/settings snapshot owned by the store facade, with clone-on-read semantics.
|
||||
5. Resolves file-backed values before database values and caches for IdP authentication and instance settings.
|
||||
6. Uses explicit raw database reads for snapshot planning and permitted mutation paths.
|
||||
7. Validates affected startup state and uses a narrow serializable transaction for runtime authentication mutations.
|
||||
8. Enforces deployment ownership through API mutation guards and returns `codes.FailedPrecondition` for rejected writes.
|
||||
|
||||
## Test strategy
|
||||
|
||||
The implementation requires tests for:
|
||||
|
||||
- Every canonical filename pattern, legacy identity-provider filename compatibility, and supported message type.
|
||||
- Unknown fields, oversized files, unreadable files, invalid protobuf JSON, and invalid enum values.
|
||||
- Duplicate provider UIDs and setting keys.
|
||||
- Rejection of `BASIC`, `TAGS`, and key/`oneof` mismatches.
|
||||
- Complete validation before snapshot publication.
|
||||
- Store-proto JSON compatibility fixtures from earlier releases.
|
||||
- Effective merging and shadowing by provider UID and setting key.
|
||||
- Stable stored ordering and deterministic placement of deployment-only identity providers.
|
||||
- Removal of a file taking effect after constructing a new process snapshot without deleting stored configuration.
|
||||
- Existing user-identity links working with a file-backed provider of the same UID.
|
||||
- File-backed settings bypassing database cache entries.
|
||||
- Clone-on-read behavior and race tests proving snapshot messages cannot be mutated by callers.
|
||||
- Authentication safety for startup and runtime IdP deletion.
|
||||
- Concurrent `GENERAL` updates and IdP deletion preserving the runtime authentication invariant across database drivers.
|
||||
- An unrelated deployment file not failing because of untouched pre-existing authentication state.
|
||||
- Storage read-time defaults, SMTP validation, and deterministic AI normalization without database secret preservation.
|
||||
- Missing-directory behavior and matched-file count logging.
|
||||
- Mutation guards, including requests carrying field masks.
|
||||
- A stored provider shadowed by a file producing a secret-free legacy-bootstrap warning.
|
||||
- Secret redaction in errors, logs, and API responses.
|
||||
|
||||
## Research references
|
||||
|
||||
- [Mastodon environment configuration](https://docs.joinmastodon.org/admin/config/)
|
||||
- [Mastodon OmniAuth initialization](https://github.com/mastodon/mastodon/blob/main/config/initializers/3_omniauth.rb)
|
||||
- [GitLab OpenID Connect configuration](https://docs.gitlab.com/administration/auth/oidc/)
|
||||
- [Keycloak startup import](https://www.keycloak.org/server/importExport)
|
||||
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
|
||||
@@ -1,42 +0,0 @@
|
||||
# Identity Provider Bootstrap
|
||||
|
||||
Memos automatically reconciles OAuth2 identity providers from JSON files before the server starts. This is useful when a deployment platform provides configuration as mounted secret files.
|
||||
|
||||
By default, Memos scans `/etc/secrets` for files named `memos-idp-*.json`:
|
||||
|
||||
```bash
|
||||
memos
|
||||
```
|
||||
|
||||
For example, `/etc/secrets/memos-idp-primary.json` can contain:
|
||||
|
||||
```json
|
||||
{
|
||||
"uid": "primary-sso",
|
||||
"name": "Company SSO",
|
||||
"type": "OAUTH2",
|
||||
"identifierFilter": "",
|
||||
"config": {
|
||||
"oauth2Config": {
|
||||
"clientId": "client-id",
|
||||
"clientSecret": "client-secret",
|
||||
"authUrl": "https://idp.example.com/oauth/authorize",
|
||||
"tokenUrl": "https://idp.example.com/oauth/token",
|
||||
"userInfoUrl": "https://idp.example.com/oauth/userinfo",
|
||||
"scopes": ["profile", "email"],
|
||||
"fieldMapping": {
|
||||
"identifier": "sub",
|
||||
"displayName": "name",
|
||||
"email": "email",
|
||||
"avatarUrl": "picture"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each file contains exactly one `memos.store.IdentityProvider` encoded as protobuf JSON. The database-generated numeric `id` must be omitted; `uid` is the provider's stable identifier.
|
||||
|
||||
After database migrations and demo seeding, Memos reads all matching files in filename order and validates every provider before writing anything. Each provider is then created or updated by its stable `uid`; providers omitted from the files are left unchanged. Reapplying the files is safe and updates credentials on restart, which supports secret rotation. Duplicate provider UIDs across files are rejected.
|
||||
|
||||
A missing directory or a directory without matching files is a normal no-op. If a matching file is unreadable, oversized, or invalid, Memos does not start. Files with other names are ignored so the directory can safely contain unrelated secrets. Keep files containing `clientSecret` outside source control and provide them through the deployment platform's secret-management facility.
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
"google.golang.org/protobuf/types/known/emptypb"
|
||||
@@ -29,6 +30,9 @@ func (s *APIV1Service) CreateIdentityProvider(ctx context.Context, request *v1pb
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.Store.IsIdentityProviderDeploymentConfigured(idpUID) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "identity provider %q is configured by the deployment", idpUID)
|
||||
}
|
||||
|
||||
storeIdp := convertIdentityProviderToStore(request.IdentityProvider)
|
||||
storeIdp.Uid = idpUID
|
||||
@@ -93,9 +97,12 @@ func (s *APIV1Service) UpdateIdentityProvider(ctx context.Context, request *v1pb
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "invalid identity provider name: %v", err)
|
||||
}
|
||||
if s.Store.IsIdentityProviderDeploymentConfigured(uid) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "identity provider %q is configured by the deployment", uid)
|
||||
}
|
||||
|
||||
// Look up the IdP by UID to get the internal ID for update.
|
||||
existing, err := s.Store.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: &uid})
|
||||
existing, err := s.Store.GetStoredIdentityProvider(ctx, &store.FindIdentityProvider{UID: &uid})
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Internal, "failed to get identity provider, error: %+v", err)
|
||||
}
|
||||
@@ -152,9 +159,12 @@ func (s *APIV1Service) DeleteIdentityProvider(ctx context.Context, request *v1pb
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "invalid identity provider name: %v", err)
|
||||
}
|
||||
if s.Store.IsIdentityProviderDeploymentConfigured(uid) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "identity provider %q is configured by the deployment", uid)
|
||||
}
|
||||
|
||||
// Look up the IdP by UID to get the internal ID for deletion.
|
||||
identityProvider, err := s.Store.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: &uid})
|
||||
identityProvider, err := s.Store.GetStoredIdentityProvider(ctx, &store.FindIdentityProvider{UID: &uid})
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Internal, "failed to check identity provider existence: %v", err)
|
||||
}
|
||||
@@ -162,7 +172,10 @@ func (s *APIV1Service) DeleteIdentityProvider(ctx context.Context, request *v1pb
|
||||
return nil, status.Errorf(codes.NotFound, "identity provider not found")
|
||||
}
|
||||
|
||||
if err := s.Store.DeleteIdentityProvider(ctx, &store.DeleteIdentityProvider{ID: identityProvider.Id}); err != nil {
|
||||
if err := s.Store.DeleteIdentityProviderSafely(ctx, &store.DeleteIdentityProvider{ID: identityProvider.Id}); err != nil {
|
||||
if errors.Is(err, store.ErrUnsafeAuthenticationConfiguration) {
|
||||
return nil, status.Error(codes.FailedPrecondition, err.Error())
|
||||
}
|
||||
return nil, status.Errorf(codes.Internal, "failed to delete identity provider, error: %+v", err)
|
||||
}
|
||||
return &emptypb.Empty{}, nil
|
||||
|
||||
@@ -99,21 +99,36 @@ func (s *APIV1Service) getInstanceSettingByName(ctx context.Context, name string
|
||||
|
||||
instanceSettingKey := storepb.InstanceSettingKey(storepb.InstanceSettingKey_value[instanceSettingKeyString])
|
||||
// Get instance setting from store with default value.
|
||||
var instanceSetting *storepb.InstanceSetting
|
||||
switch instanceSettingKey {
|
||||
case storepb.InstanceSettingKey_BASIC:
|
||||
_, err = s.Store.GetInstanceBasicSetting(ctx)
|
||||
var setting *storepb.InstanceBasicSetting
|
||||
setting, err = s.Store.GetInstanceBasicSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_BasicSetting{BasicSetting: setting}}
|
||||
case storepb.InstanceSettingKey_GENERAL:
|
||||
_, err = s.Store.GetInstanceGeneralSetting(ctx)
|
||||
var setting *storepb.InstanceGeneralSetting
|
||||
setting, err = s.Store.GetInstanceGeneralSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: setting}}
|
||||
case storepb.InstanceSettingKey_MEMO_RELATED:
|
||||
_, err = s.Store.GetInstanceMemoRelatedSetting(ctx)
|
||||
var setting *storepb.InstanceMemoRelatedSetting
|
||||
setting, err = s.Store.GetInstanceMemoRelatedSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_MemoRelatedSetting{MemoRelatedSetting: setting}}
|
||||
case storepb.InstanceSettingKey_STORAGE:
|
||||
_, err = s.Store.GetInstanceStorageSetting(ctx)
|
||||
var setting *storepb.InstanceStorageSetting
|
||||
setting, err = s.Store.GetInstanceStorageSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_StorageSetting{StorageSetting: setting}}
|
||||
case storepb.InstanceSettingKey_TAGS:
|
||||
_, err = s.Store.GetInstanceTagsSetting(ctx)
|
||||
var setting *storepb.InstanceTagsSetting
|
||||
setting, err = s.Store.GetInstanceTagsSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_TagsSetting{TagsSetting: setting}}
|
||||
case storepb.InstanceSettingKey_NOTIFICATION:
|
||||
_, err = s.Store.GetInstanceNotificationSetting(ctx)
|
||||
var setting *storepb.InstanceNotificationSetting
|
||||
setting, err = s.Store.GetInstanceNotificationSetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_NotificationSetting{NotificationSetting: setting}}
|
||||
case storepb.InstanceSettingKey_AI:
|
||||
_, err = s.Store.GetInstanceAISetting(ctx)
|
||||
var setting *storepb.InstanceAISetting
|
||||
setting, err = s.Store.GetInstanceAISetting(ctx)
|
||||
instanceSetting = &storepb.InstanceSetting{Key: instanceSettingKey, Value: &storepb.InstanceSetting_AiSetting{AiSetting: setting}}
|
||||
default:
|
||||
return nil, status.Errorf(codes.InvalidArgument, "unsupported instance setting key: %v", instanceSettingKey)
|
||||
}
|
||||
@@ -121,16 +136,6 @@ func (s *APIV1Service) getInstanceSettingByName(ctx context.Context, name string
|
||||
return nil, status.Errorf(codes.Internal, "failed to get instance setting: %v", err)
|
||||
}
|
||||
|
||||
instanceSetting, err := s.Store.GetInstanceSetting(ctx, &store.FindInstanceSetting{
|
||||
Name: instanceSettingKey.String(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.Internal, "failed to get instance setting: %v", err)
|
||||
}
|
||||
if instanceSetting == nil {
|
||||
return nil, status.Errorf(codes.NotFound, "instance setting not found")
|
||||
}
|
||||
|
||||
// Storage and notification settings contain credentials; restrict to admins only.
|
||||
if instanceSetting.Key == storepb.InstanceSettingKey_STORAGE ||
|
||||
instanceSetting.Key == storepb.InstanceSettingKey_NOTIFICATION {
|
||||
@@ -183,6 +188,17 @@ func (s *APIV1Service) UpdateInstanceSetting(ctx context.Context, request *v1pb.
|
||||
if user.Role != store.RoleAdmin {
|
||||
return nil, status.Errorf(codes.PermissionDenied, "permission denied")
|
||||
}
|
||||
if request.Setting == nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "instance setting is required")
|
||||
}
|
||||
settingKeyString, err := ExtractInstanceSettingKeyFromName(request.Setting.Name)
|
||||
if err != nil {
|
||||
return nil, status.Errorf(codes.InvalidArgument, "invalid instance setting name: %v", err)
|
||||
}
|
||||
settingKey := storepb.InstanceSettingKey(storepb.InstanceSettingKey_value[settingKeyString])
|
||||
if s.Store.IsInstanceSettingDeploymentConfigured(settingKey) {
|
||||
return nil, status.Errorf(codes.FailedPrecondition, "instance setting %q is configured by the deployment", settingKeyString)
|
||||
}
|
||||
|
||||
// TODO: Apply update_mask if specified
|
||||
_ = request.UpdateMask
|
||||
@@ -221,8 +237,16 @@ func (s *APIV1Service) UpdateInstanceSetting(ctx context.Context, request *v1pb.
|
||||
// No credential preservation needed for other setting types.
|
||||
}
|
||||
|
||||
instanceSetting, err := s.Store.UpsertInstanceSetting(ctx, updateSetting)
|
||||
var instanceSetting *storepb.InstanceSetting
|
||||
if updateSetting.Key == storepb.InstanceSettingKey_GENERAL {
|
||||
instanceSetting, err = s.Store.UpsertInstanceGeneralSettingSafely(ctx, updateSetting)
|
||||
} else {
|
||||
instanceSetting, err = s.Store.UpsertInstanceSetting(ctx, updateSetting)
|
||||
}
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrUnsafeAuthenticationConfiguration) {
|
||||
return nil, status.Error(codes.FailedPrecondition, err.Error())
|
||||
}
|
||||
return nil, status.Errorf(codes.Internal, "failed to upsert instance setting: %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/types/known/fieldmaskpb"
|
||||
|
||||
v1pb "github.com/usememos/memos/proto/gen/api/v1"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
)
|
||||
|
||||
func TestDeploymentConfiguredResourcesRejectMutations(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
ts := NewTestService(t)
|
||||
defer ts.Cleanup()
|
||||
admin, err := ts.CreateHostUser(ctx, "admin")
|
||||
require.NoError(t, err)
|
||||
adminCtx := ts.CreateUserContext(ctx, admin.ID)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeDeploymentProto(t, filepath.Join(dir, "memos-idp-primary.json"), testStoreIdentityProvider("primary-sso"))
|
||||
writeDeploymentProto(t, filepath.Join(dir, "memos-instance-setting-general.json"), &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{WeekStartDayOffset: 1}},
|
||||
})
|
||||
writeDeploymentProto(t, filepath.Join(dir, "memos-instance-setting-storage.json"), &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_STORAGE,
|
||||
Value: &storepb.InstanceSetting_StorageSetting{StorageSetting: &storepb.InstanceStorageSetting{
|
||||
StorageType: storepb.InstanceStorageSetting_LOCAL,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, ts.Store.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
providers, err := ts.Service.ListIdentityProviders(ctx, &v1pb.ListIdentityProvidersRequest{})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, providers.IdentityProviders, 1)
|
||||
require.Equal(t, "Primary SSO", providers.IdentityProviders[0].Title)
|
||||
|
||||
setting, err := ts.Service.GetInstanceSetting(ctx, &v1pb.GetInstanceSettingRequest{Name: "instance/settings/GENERAL"})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, int32(1), setting.GetGeneralSetting().WeekStartDayOffset)
|
||||
storageSetting, err := ts.Service.GetInstanceSetting(adminCtx, &v1pb.GetInstanceSettingRequest{Name: "instance/settings/STORAGE"})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, int64(30), storageSetting.GetStorageSetting().UploadSizeLimitMb)
|
||||
require.Equal(t, "assets/{timestamp}_{uuid}_{filename}", storageSetting.GetStorageSetting().FilepathTemplate)
|
||||
|
||||
_, err = ts.Service.CreateIdentityProvider(adminCtx, &v1pb.CreateIdentityProviderRequest{
|
||||
IdentityProviderId: "primary-sso",
|
||||
IdentityProvider: testAPIIdentityProvider("Replacement"),
|
||||
})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
|
||||
_, err = ts.Service.UpdateIdentityProvider(adminCtx, &v1pb.UpdateIdentityProviderRequest{
|
||||
IdentityProvider: &v1pb.IdentityProvider{Name: "identity-providers/primary-sso", Title: "Changed"},
|
||||
UpdateMask: &fieldmaskpb.FieldMask{Paths: []string{"title"}},
|
||||
})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
|
||||
_, err = ts.Service.DeleteIdentityProvider(adminCtx, &v1pb.DeleteIdentityProviderRequest{Name: "identity-providers/primary-sso"})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
|
||||
_, err = ts.Service.UpdateInstanceSetting(adminCtx, &v1pb.UpdateInstanceSettingRequest{
|
||||
Setting: &v1pb.InstanceSetting{
|
||||
Name: "instance/settings/GENERAL",
|
||||
Value: &v1pb.InstanceSetting_GeneralSetting_{GeneralSetting: &v1pb.InstanceSetting_GeneralSetting{}},
|
||||
},
|
||||
UpdateMask: &fieldmaskpb.FieldMask{Paths: []string{"general_setting"}},
|
||||
})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
}
|
||||
|
||||
func TestAuthenticationMutationAPIRejectsLockout(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
t.Run("GENERAL update requires an effective identity provider", func(t *testing.T) {
|
||||
ts := NewTestService(t)
|
||||
defer ts.Cleanup()
|
||||
admin, err := ts.CreateHostUser(ctx, "admin")
|
||||
require.NoError(t, err)
|
||||
adminCtx := ts.CreateUserContext(ctx, admin.ID)
|
||||
|
||||
_, err = ts.Service.UpdateInstanceSetting(adminCtx, &v1pb.UpdateInstanceSettingRequest{Setting: &v1pb.InstanceSetting{
|
||||
Name: "instance/settings/GENERAL",
|
||||
Value: &v1pb.InstanceSetting_GeneralSetting_{GeneralSetting: &v1pb.InstanceSetting_GeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
}})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
})
|
||||
|
||||
t.Run("last IdP cannot be deleted while regular password auth is disabled", func(t *testing.T) {
|
||||
ts := NewTestService(t)
|
||||
defer ts.Cleanup()
|
||||
admin, err := ts.CreateHostUser(ctx, "admin")
|
||||
require.NoError(t, err)
|
||||
adminCtx := ts.CreateUserContext(ctx, admin.ID)
|
||||
created, err := ts.Service.CreateIdentityProvider(adminCtx, &v1pb.CreateIdentityProviderRequest{
|
||||
IdentityProviderId: "primary-sso",
|
||||
IdentityProvider: testAPIIdentityProvider("Primary"),
|
||||
})
|
||||
require.NoError(t, err)
|
||||
_, err = ts.Service.UpdateInstanceSetting(adminCtx, &v1pb.UpdateInstanceSettingRequest{Setting: &v1pb.InstanceSetting{
|
||||
Name: "instance/settings/GENERAL",
|
||||
Value: &v1pb.InstanceSetting_GeneralSetting_{GeneralSetting: &v1pb.InstanceSetting_GeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
}})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = ts.Service.DeleteIdentityProvider(adminCtx, &v1pb.DeleteIdentityProviderRequest{Name: created.Name})
|
||||
require.Equal(t, codes.FailedPrecondition, status.Code(err))
|
||||
})
|
||||
}
|
||||
|
||||
func testStoreIdentityProvider(uid string) *storepb.IdentityProvider {
|
||||
return &storepb.IdentityProvider{
|
||||
Uid: uid,
|
||||
Name: "Primary SSO",
|
||||
Type: storepb.IdentityProvider_OAUTH2,
|
||||
Config: &storepb.IdentityProviderConfig{Config: &storepb.IdentityProviderConfig_Oauth2Config{Oauth2Config: &storepb.OAuth2Config{
|
||||
ClientId: "client-id",
|
||||
ClientSecret: "client-secret",
|
||||
AuthUrl: "https://example.com/authorize",
|
||||
TokenUrl: "https://example.com/token",
|
||||
UserInfoUrl: "https://example.com/userinfo",
|
||||
Scopes: []string{"openid", "profile"},
|
||||
FieldMapping: &storepb.FieldMapping{Identifier: "sub"},
|
||||
}}},
|
||||
}
|
||||
}
|
||||
|
||||
func testAPIIdentityProvider(title string) *v1pb.IdentityProvider {
|
||||
return &v1pb.IdentityProvider{
|
||||
Title: title,
|
||||
Type: v1pb.IdentityProvider_OAUTH2,
|
||||
Config: &v1pb.IdentityProviderConfig{Config: &v1pb.IdentityProviderConfig_Oauth2Config{Oauth2Config: &v1pb.OAuth2Config{
|
||||
ClientId: "client-id",
|
||||
ClientSecret: "client-secret",
|
||||
AuthUrl: "https://example.com/authorize",
|
||||
TokenUrl: "https://example.com/token",
|
||||
UserInfoUrl: "https://example.com/userinfo",
|
||||
Scopes: []string{"openid", "profile"},
|
||||
FieldMapping: &v1pb.FieldMapping{Identifier: "sub"},
|
||||
}}},
|
||||
}
|
||||
}
|
||||
|
||||
func writeDeploymentProto(t *testing.T, path string, message proto.Message) {
|
||||
t.Helper()
|
||||
content, err := (protojson.MarshalOptions{Indent: " "}).Marshal(message)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, os.WriteFile(path, content, 0600))
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
)
|
||||
|
||||
const authenticationMutationMaxAttempts = 3
|
||||
|
||||
// ErrUnsafeAuthenticationConfiguration indicates a mutation would lock regular users out.
|
||||
var ErrUnsafeAuthenticationConfiguration = errors.New("password authentication for regular users cannot be disabled without an effective identity provider")
|
||||
|
||||
// AuthenticationConfigState is the stored authentication configuration read inside a transaction.
|
||||
type AuthenticationConfigState struct {
|
||||
GeneralSetting *InstanceSetting
|
||||
IdentityProviders []*IdentityProvider
|
||||
}
|
||||
|
||||
// AuthenticationConfigMutation validates and applies one stored authentication mutation atomically.
|
||||
type AuthenticationConfigMutation struct {
|
||||
UpsertGeneralSetting *InstanceSetting
|
||||
DeleteIdentityProviderID *int32
|
||||
Validate func(*AuthenticationConfigState) error
|
||||
}
|
||||
|
||||
// UpsertInstanceGeneralSettingSafely validates and stores GENERAL as one serialized operation.
|
||||
func (s *Store) UpsertInstanceGeneralSettingSafely(ctx context.Context, setting *storepb.InstanceSetting) (*storepb.InstanceSetting, error) {
|
||||
if setting == nil || setting.Key != storepb.InstanceSettingKey_GENERAL || setting.GetGeneralSetting() == nil {
|
||||
return nil, errors.New("GENERAL instance setting is required")
|
||||
}
|
||||
value, err := protojson.Marshal(setting.GetGeneralSetting())
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to marshal GENERAL instance setting")
|
||||
}
|
||||
raw := &InstanceSetting{Name: storepb.InstanceSettingKey_GENERAL.String(), Value: string(value)}
|
||||
mutation := &AuthenticationConfigMutation{
|
||||
UpsertGeneralSetting: raw,
|
||||
Validate: func(state *AuthenticationConfigState) error {
|
||||
return s.validateAuthenticationMutationState(state, setting.GetGeneralSetting(), nil)
|
||||
},
|
||||
}
|
||||
if err := s.applyAuthenticationConfigMutation(ctx, mutation); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result := cloneInstanceSetting(setting)
|
||||
s.cacheInstanceSetting(ctx, result)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// DeleteIdentityProviderSafely validates and deletes an IdP as one serialized operation.
|
||||
func (s *Store) DeleteIdentityProviderSafely(ctx context.Context, delete *DeleteIdentityProvider) error {
|
||||
if delete == nil {
|
||||
return errors.New("identity provider deletion is required")
|
||||
}
|
||||
mutation := &AuthenticationConfigMutation{
|
||||
DeleteIdentityProviderID: &delete.ID,
|
||||
Validate: func(state *AuthenticationConfigState) error {
|
||||
return s.validateAuthenticationMutationState(state, nil, &delete.ID)
|
||||
},
|
||||
}
|
||||
return s.applyAuthenticationConfigMutation(ctx, mutation)
|
||||
}
|
||||
|
||||
func (s *Store) applyAuthenticationConfigMutation(ctx context.Context, mutation *AuthenticationConfigMutation) error {
|
||||
s.authConfigMu.Lock()
|
||||
defer s.authConfigMu.Unlock()
|
||||
|
||||
var err error
|
||||
for attempt := 0; attempt < authenticationMutationMaxAttempts; attempt++ {
|
||||
err = s.driver.ApplyAuthenticationConfigMutation(ctx, mutation)
|
||||
if err == nil || !s.driver.IsRetryableAuthenticationMutationError(err) {
|
||||
return err
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Duration(attempt+1) * 10 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
return errors.Wrap(err, "authentication configuration mutation failed after retries")
|
||||
}
|
||||
|
||||
func (s *Store) validateAuthenticationMutationState(state *AuthenticationConfigState, proposedGeneral *storepb.InstanceGeneralSetting, deletedProviderID *int32) error {
|
||||
var storedGeneral *storepb.InstanceGeneralSetting
|
||||
if state.GeneralSetting != nil {
|
||||
stored, err := convertInstanceSettingFromRaw(state.GeneralSetting)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to decode stored GENERAL setting")
|
||||
}
|
||||
storedGeneral = stored.GetGeneralSetting()
|
||||
}
|
||||
|
||||
storedUIDs := make(map[string]struct{}, len(state.IdentityProviders))
|
||||
for _, provider := range state.IdentityProviders {
|
||||
storedUIDs[provider.UID] = struct{}{}
|
||||
}
|
||||
effectiveUIDs := make(map[string]struct{}, len(state.IdentityProviders))
|
||||
for _, provider := range state.IdentityProviders {
|
||||
if deletedProviderID != nil && provider.ID == *deletedProviderID {
|
||||
continue
|
||||
}
|
||||
effectiveUIDs[provider.UID] = struct{}{}
|
||||
}
|
||||
for _, provider := range s.listDeploymentIdentityProviders() {
|
||||
effectiveUIDs[provider.Uid] = struct{}{}
|
||||
}
|
||||
|
||||
configuredGeneral := s.getDeploymentInstanceSetting(storepb.InstanceSettingKey_GENERAL)
|
||||
if proposedGeneral != nil && configuredGeneral != nil {
|
||||
// A deployment-managed GENERAL setting keeps the proposal from becoming
|
||||
// effective now, but the stored fallback must not move from safe to unsafe
|
||||
// if the deployment file is removed later.
|
||||
if isUnsafeAuthenticationState(proposedGeneral, storedUIDs) && !isUnsafeAuthenticationState(storedGeneral, storedUIDs) {
|
||||
return ErrUnsafeAuthenticationConfiguration
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
oldGeneral := storedGeneral
|
||||
if configuredGeneral != nil {
|
||||
oldGeneral = configuredGeneral.GetGeneralSetting()
|
||||
}
|
||||
newGeneral := oldGeneral
|
||||
if proposedGeneral != nil {
|
||||
newGeneral = proposedGeneral
|
||||
}
|
||||
|
||||
oldEffectiveUIDs := make(map[string]struct{}, len(storedUIDs))
|
||||
for uid := range storedUIDs {
|
||||
oldEffectiveUIDs[uid] = struct{}{}
|
||||
}
|
||||
for _, provider := range s.listDeploymentIdentityProviders() {
|
||||
oldEffectiveUIDs[provider.Uid] = struct{}{}
|
||||
}
|
||||
if isUnsafeAuthenticationState(newGeneral, effectiveUIDs) && !isUnsafeAuthenticationState(oldGeneral, oldEffectiveUIDs) {
|
||||
return ErrUnsafeAuthenticationConfiguration
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func isUnsafeAuthenticationState(general *storepb.InstanceGeneralSetting, providerUIDs map[string]struct{}) bool {
|
||||
return general != nil && general.DisallowPasswordAuth && len(providerUIDs) == 0
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
mysqldriver "github.com/go-sql-driver/mysql"
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"github.com/usememos/memos/store"
|
||||
)
|
||||
|
||||
// IsRetryableAuthenticationMutationError reports whether err is a transient MySQL transaction failure.
|
||||
func (*DB) IsRetryableAuthenticationMutationError(err error) bool {
|
||||
var mysqlErr *mysqldriver.MySQLError
|
||||
if !errors.As(err, &mysqlErr) {
|
||||
return false
|
||||
}
|
||||
return mysqlErr.Number == 1205 || mysqlErr.Number == 1213
|
||||
}
|
||||
|
||||
// ApplyAuthenticationConfigMutation validates and applies an auth mutation in a serializable transaction.
|
||||
func (d *DB) ApplyAuthenticationConfigMutation(ctx context.Context, mutation *store.AuthenticationConfigMutation) error {
|
||||
tx, err := d.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to begin authentication configuration transaction")
|
||||
}
|
||||
defer func() {
|
||||
_ = tx.Rollback()
|
||||
}()
|
||||
|
||||
state := &store.AuthenticationConfigState{}
|
||||
general := &store.InstanceSetting{}
|
||||
err = tx.QueryRowContext(ctx, "SELECT `name`, `value`, `description` FROM `system_setting` WHERE `name` = ? FOR UPDATE", "GENERAL").Scan(
|
||||
&general.Name, &general.Value, &general.Description,
|
||||
)
|
||||
if err == nil {
|
||||
state.GeneralSetting = general
|
||||
} else if !errors.Is(err, sql.ErrNoRows) {
|
||||
return errors.Wrap(err, "failed to read GENERAL setting")
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, "SELECT `id`, `uid` FROM `idp` ORDER BY `id` FOR UPDATE")
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read identity providers")
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
provider := &store.IdentityProvider{}
|
||||
if err := rows.Scan(&provider.ID, &provider.UID); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to scan identity provider")
|
||||
}
|
||||
state.IdentityProviders = append(state.IdentityProviders, provider)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to iterate identity providers")
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return errors.Wrap(err, "failed to close identity provider rows")
|
||||
}
|
||||
if mutation.Validate != nil {
|
||||
if err := mutation.Validate(state); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if setting := mutation.UpsertGeneralSetting; setting != nil {
|
||||
_, err = tx.ExecContext(ctx, "INSERT INTO `system_setting` (`name`, `value`, `description`) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE `value` = ?, `description` = ?", setting.Name, setting.Value, setting.Description, setting.Value, setting.Description)
|
||||
} else if id := mutation.DeleteIdentityProviderID; id != nil {
|
||||
_, err = tx.ExecContext(ctx, "DELETE FROM `idp` WHERE `id` = ?", *id)
|
||||
} else {
|
||||
return errors.New("authentication configuration mutation has no operation")
|
||||
}
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to apply authentication configuration mutation")
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return errors.Wrap(err, "failed to commit authentication configuration transaction")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
mysqldriver "github.com/go-sql-driver/mysql"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestIsRetryableAuthenticationMutationError(t *testing.T) {
|
||||
db := &DB{}
|
||||
require.True(t, db.IsRetryableAuthenticationMutationError(errors.Wrap(&mysqldriver.MySQLError{Number: 1213}, "commit failed")))
|
||||
require.True(t, db.IsRetryableAuthenticationMutationError(&mysqldriver.MySQLError{Number: 1205}))
|
||||
require.False(t, db.IsRetryableAuthenticationMutationError(&mysqldriver.MySQLError{Number: 1062}))
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"github.com/lib/pq"
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"github.com/usememos/memos/store"
|
||||
)
|
||||
|
||||
// IsRetryableAuthenticationMutationError reports whether err is a transient PostgreSQL transaction failure.
|
||||
func (*DB) IsRetryableAuthenticationMutationError(err error) bool {
|
||||
var pqErr *pq.Error
|
||||
if !errors.As(err, &pqErr) {
|
||||
return false
|
||||
}
|
||||
return pqErr.Code == "40001" || pqErr.Code == "40P01"
|
||||
}
|
||||
|
||||
// ApplyAuthenticationConfigMutation validates and applies an auth mutation in a serializable transaction.
|
||||
func (d *DB) ApplyAuthenticationConfigMutation(ctx context.Context, mutation *store.AuthenticationConfigMutation) error {
|
||||
tx, err := d.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to begin authentication configuration transaction")
|
||||
}
|
||||
defer func() {
|
||||
_ = tx.Rollback()
|
||||
}()
|
||||
|
||||
state := &store.AuthenticationConfigState{}
|
||||
general := &store.InstanceSetting{}
|
||||
err = tx.QueryRowContext(ctx, `SELECT name, value, description FROM system_setting WHERE name = $1 FOR UPDATE`, "GENERAL").Scan(
|
||||
&general.Name, &general.Value, &general.Description,
|
||||
)
|
||||
if err == nil {
|
||||
state.GeneralSetting = general
|
||||
} else if !errors.Is(err, sql.ErrNoRows) {
|
||||
return errors.Wrap(err, "failed to read GENERAL setting")
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, `SELECT id, uid FROM idp ORDER BY id FOR UPDATE`)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read identity providers")
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
provider := &store.IdentityProvider{}
|
||||
if err := rows.Scan(&provider.ID, &provider.UID); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to scan identity provider")
|
||||
}
|
||||
state.IdentityProviders = append(state.IdentityProviders, provider)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to iterate identity providers")
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return errors.Wrap(err, "failed to close identity provider rows")
|
||||
}
|
||||
if mutation.Validate != nil {
|
||||
if err := mutation.Validate(state); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if setting := mutation.UpsertGeneralSetting; setting != nil {
|
||||
_, err = tx.ExecContext(ctx, `INSERT INTO system_setting (name, value, description) VALUES ($1, $2, $3) ON CONFLICT(name) DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description`, setting.Name, setting.Value, setting.Description)
|
||||
} else if id := mutation.DeleteIdentityProviderID; id != nil {
|
||||
_, err = tx.ExecContext(ctx, `DELETE FROM idp WHERE id = $1`, *id)
|
||||
} else {
|
||||
return errors.New("authentication configuration mutation has no operation")
|
||||
}
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to apply authentication configuration mutation")
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return errors.Wrap(err, "failed to commit authentication configuration transaction")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/lib/pq"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestIsRetryableAuthenticationMutationError(t *testing.T) {
|
||||
db := &DB{}
|
||||
require.True(t, db.IsRetryableAuthenticationMutationError(errors.Wrap(&pq.Error{Code: "40001"}, "commit failed")))
|
||||
require.True(t, db.IsRetryableAuthenticationMutationError(&pq.Error{Code: "40P01"}))
|
||||
require.False(t, db.IsRetryableAuthenticationMutationError(&pq.Error{Code: "23505"}))
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
msqlite "modernc.org/sqlite"
|
||||
sqlite3 "modernc.org/sqlite/lib"
|
||||
|
||||
"github.com/usememos/memos/store"
|
||||
)
|
||||
|
||||
// IsRetryableAuthenticationMutationError reports whether err is a transient SQLite locking failure.
|
||||
func (*DB) IsRetryableAuthenticationMutationError(err error) bool {
|
||||
var sqliteErr *msqlite.Error
|
||||
if !errors.As(err, &sqliteErr) {
|
||||
return false
|
||||
}
|
||||
code := sqliteErr.Code() & 0xff
|
||||
return code == sqlite3.SQLITE_BUSY || code == sqlite3.SQLITE_LOCKED
|
||||
}
|
||||
|
||||
// ApplyAuthenticationConfigMutation validates and applies an auth mutation in a serializable transaction.
|
||||
func (d *DB) ApplyAuthenticationConfigMutation(ctx context.Context, mutation *store.AuthenticationConfigMutation) error {
|
||||
tx, err := d.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to begin authentication configuration transaction")
|
||||
}
|
||||
defer func() {
|
||||
_ = tx.Rollback()
|
||||
}()
|
||||
|
||||
state := &store.AuthenticationConfigState{}
|
||||
general := &store.InstanceSetting{}
|
||||
err = tx.QueryRowContext(ctx, `SELECT name, value, description FROM system_setting WHERE name = ?`, "GENERAL").Scan(
|
||||
&general.Name, &general.Value, &general.Description,
|
||||
)
|
||||
if err == nil {
|
||||
state.GeneralSetting = general
|
||||
} else if !errors.Is(err, sql.ErrNoRows) {
|
||||
return errors.Wrap(err, "failed to read GENERAL setting")
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx, `SELECT id, uid FROM idp ORDER BY id`)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read identity providers")
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
provider := &store.IdentityProvider{}
|
||||
if err := rows.Scan(&provider.ID, &provider.UID); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to scan identity provider")
|
||||
}
|
||||
state.IdentityProviders = append(state.IdentityProviders, provider)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return errors.Wrap(err, "failed to iterate identity providers")
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return errors.Wrap(err, "failed to close identity provider rows")
|
||||
}
|
||||
if mutation.Validate != nil {
|
||||
if err := mutation.Validate(state); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if setting := mutation.UpsertGeneralSetting; setting != nil {
|
||||
_, err = tx.ExecContext(ctx, `INSERT INTO system_setting (name, value, description) VALUES (?, ?, ?) ON CONFLICT(name) DO UPDATE SET value = EXCLUDED.value, description = EXCLUDED.description`, setting.Name, setting.Value, setting.Description)
|
||||
} else if id := mutation.DeleteIdentityProviderID; id != nil {
|
||||
_, err = tx.ExecContext(ctx, `DELETE FROM idp WHERE id = ?`, *id)
|
||||
} else {
|
||||
return errors.New("authentication configuration mutation has no operation")
|
||||
}
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to apply authentication configuration mutation")
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return errors.Wrap(err, "failed to commit authentication configuration transaction")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+13
-1
@@ -13,7 +13,7 @@ import (
|
||||
"github.com/usememos/memos/store/db/sqlite"
|
||||
)
|
||||
|
||||
func TestDemoSeedRequiresIdentityProviderSignIn(t *testing.T) {
|
||||
func TestDemoSeedUsesDeploymentAuthenticationPolicy(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
p := &profile.Profile{
|
||||
Demo: true,
|
||||
@@ -31,8 +31,20 @@ func TestDemoSeedRequiresIdentityProviderSignIn(t *testing.T) {
|
||||
require.NoError(t, stores.Migrate(ctx))
|
||||
generalSetting, err := stores.GetInstanceGeneralSetting(ctx)
|
||||
require.NoError(t, err)
|
||||
require.False(t, generalSetting.DisallowPasswordAuth, "authentication policy must come from deployment configuration")
|
||||
require.False(t, generalSetting.DisallowUserRegistration, "SSO first-login provisioning must remain enabled")
|
||||
|
||||
secretDir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(secretDir, "memos-idp-primary-sso.json"), "primary-sso", "Primary SSO", "secret")
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(secretDir, "memos-instance-setting-general.json"), 0, true)
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, secretDir))
|
||||
generalSetting, err = stores.GetInstanceGeneralSetting(ctx)
|
||||
require.NoError(t, err)
|
||||
require.True(t, generalSetting.DisallowPasswordAuth)
|
||||
require.False(t, generalSetting.DisallowUserRegistration, "SSO first-login provisioning must remain enabled")
|
||||
provider, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, provider)
|
||||
|
||||
demoUsername := "demo"
|
||||
demoUser, err := stores.GetUser(ctx, &store.FindUser{Username: &demoUsername})
|
||||
|
||||
@@ -0,0 +1,466 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/usememos/memos/internal/base"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
)
|
||||
|
||||
const (
|
||||
// DefaultDeploymentConfigurationDir is the directory scanned for runtime configuration files.
|
||||
DefaultDeploymentConfigurationDir = "/etc/secrets"
|
||||
maxDeploymentConfigurationSize = 1 << 20
|
||||
maxTranscriptionModelLength = 256
|
||||
maxTranscriptionLanguageLength = 32
|
||||
maxTranscriptionPromptLength = 4096
|
||||
)
|
||||
|
||||
var (
|
||||
idpDeploymentFilenameMatcher = regexp.MustCompile(`^memos-idp-[a-z0-9]+(?:-[a-z0-9]+)*\.json$`)
|
||||
instanceSettingDeploymentFilenameMatcher = regexp.MustCompile(`^memos-instance-setting-[a-z0-9]+(?:-[a-z0-9]+)*\.json$`)
|
||||
protoJSONUnknownFieldMatcher = regexp.MustCompile(`unknown field "([^"]+)"`)
|
||||
)
|
||||
|
||||
// LoadDeploymentConfiguration loads the default runtime deployment configuration.
|
||||
func (s *Store) LoadDeploymentConfiguration(ctx context.Context) error {
|
||||
return s.LoadDeploymentConfigurationDir(ctx, DefaultDeploymentConfigurationDir)
|
||||
}
|
||||
|
||||
// LoadDeploymentConfigurationDir loads and atomically publishes runtime configuration from dir.
|
||||
func (s *Store) LoadDeploymentConfigurationDir(ctx context.Context, dir string) error {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
s.setDeploymentConfiguration(newDeploymentConfiguration())
|
||||
return nil
|
||||
}
|
||||
return errors.Wrap(err, "failed to read deployment configuration directory")
|
||||
}
|
||||
|
||||
config := newDeploymentConfiguration()
|
||||
idpFiles := map[string]string{}
|
||||
settingFiles := map[storepb.InstanceSettingKey]string{}
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
path := filepath.Join(dir, name)
|
||||
switch {
|
||||
case isIdentityProviderDeploymentFilename(name):
|
||||
if !idpDeploymentFilenameMatcher.MatchString(name) {
|
||||
slog.Warn("loading identity provider deployment file with a legacy filename; rename it to lowercase kebab case", "filename", name)
|
||||
}
|
||||
provider := &storepb.IdentityProvider{}
|
||||
if err := readDeploymentProtoJSON(path, provider); err != nil {
|
||||
return errors.Wrapf(err, "invalid identity provider deployment file %q", name)
|
||||
}
|
||||
if err := validateDeploymentIdentityProvider(provider); err != nil {
|
||||
return errors.Wrapf(err, "invalid identity provider deployment file %q", name)
|
||||
}
|
||||
if previous, ok := idpFiles[provider.Uid]; ok {
|
||||
return errors.Errorf("identity provider UID %q is declared by both %q and %q", provider.Uid, previous, name)
|
||||
}
|
||||
idpFiles[provider.Uid] = name
|
||||
config.identityProviders[provider.Uid] = cloneIdentityProvider(provider)
|
||||
case instanceSettingDeploymentFilenameMatcher.MatchString(name):
|
||||
setting := &storepb.InstanceSetting{}
|
||||
if err := readDeploymentProtoJSON(path, setting); err != nil {
|
||||
return errors.Wrapf(err, "invalid instance setting deployment file %q", name)
|
||||
}
|
||||
if err := validateAndNormalizeDeploymentInstanceSetting(setting); err != nil {
|
||||
return errors.Wrapf(err, "invalid instance setting deployment file %q", name)
|
||||
}
|
||||
if previous, ok := settingFiles[setting.Key]; ok {
|
||||
return errors.Errorf("instance setting key %q is declared by both %q and %q", setting.Key, previous, name)
|
||||
}
|
||||
settingFiles[setting.Key] = name
|
||||
config.instanceSettings[setting.Key] = cloneInstanceSetting(setting)
|
||||
case strings.HasPrefix(name, "memos-"):
|
||||
slog.Warn("ignoring unrecognized Memos deployment configuration filename", "filename", name)
|
||||
default:
|
||||
// The directory may contain unrelated platform secret files.
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.validateDeploymentAuthenticationState(ctx, config); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.warnShadowedStoredIdentityProviders(ctx, config); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.setDeploymentConfiguration(config)
|
||||
slog.Info("loaded deployment configuration", "identityProviders", len(config.identityProviders), "instanceSettings", len(config.instanceSettings))
|
||||
return nil
|
||||
}
|
||||
|
||||
func newDeploymentConfiguration() *deploymentConfiguration {
|
||||
return &deploymentConfiguration{
|
||||
identityProviders: map[string]*storepb.IdentityProvider{},
|
||||
instanceSettings: map[storepb.InstanceSettingKey]*storepb.InstanceSetting{},
|
||||
}
|
||||
}
|
||||
|
||||
func isIdentityProviderDeploymentFilename(name string) bool {
|
||||
// The original database-writing bootstrap accepted every filename with this
|
||||
// prefix and suffix. Continue loading those names so an upgrade cannot
|
||||
// silently fall back to stale credentials stored in the database.
|
||||
return strings.HasPrefix(name, "memos-idp-") && strings.HasSuffix(name, ".json")
|
||||
}
|
||||
|
||||
func readDeploymentProtoJSON(path string, message proto.Message) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to inspect file")
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return errors.New("file must resolve to a regular file")
|
||||
}
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to open file")
|
||||
}
|
||||
defer file.Close()
|
||||
info, err = file.Stat()
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to inspect file")
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return errors.New("file must resolve to a regular file")
|
||||
}
|
||||
content, err := io.ReadAll(io.LimitReader(file, maxDeploymentConfigurationSize+1))
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read file")
|
||||
}
|
||||
if len(content) > maxDeploymentConfigurationSize {
|
||||
return errors.Errorf("file exceeds %d bytes", maxDeploymentConfigurationSize)
|
||||
}
|
||||
if err := (protojson.UnmarshalOptions{DiscardUnknown: false}).Unmarshal(content, message); err != nil {
|
||||
if matches := protoJSONUnknownFieldMatcher.FindStringSubmatch(err.Error()); len(matches) == 2 {
|
||||
return errors.Errorf("failed to decode protobuf JSON: unknown field %q", matches[1])
|
||||
}
|
||||
return errors.New("failed to decode protobuf JSON; verify field names, value types, and JSON syntax")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDeploymentIdentityProvider(provider *storepb.IdentityProvider) error {
|
||||
if provider.Id != 0 {
|
||||
return errors.New("id must be omitted")
|
||||
}
|
||||
if !base.UIDMatcher.MatchString(provider.Uid) {
|
||||
return errors.New("uid is invalid")
|
||||
}
|
||||
if strings.TrimSpace(provider.Name) == "" {
|
||||
return errors.New("name is required")
|
||||
}
|
||||
if provider.Type != storepb.IdentityProvider_OAUTH2 {
|
||||
return errors.New("type must be OAUTH2")
|
||||
}
|
||||
if provider.IdentifierFilter != "" {
|
||||
if _, err := regexp.Compile(provider.IdentifierFilter); err != nil {
|
||||
return errors.Wrap(err, "identifierFilter must be a valid regular expression")
|
||||
}
|
||||
}
|
||||
config := provider.Config.GetOauth2Config()
|
||||
if config == nil {
|
||||
return errors.New("config.oauth2Config is required")
|
||||
}
|
||||
required := []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{name: "clientId", value: config.ClientId},
|
||||
{name: "clientSecret", value: config.ClientSecret},
|
||||
{name: "authUrl", value: config.AuthUrl},
|
||||
{name: "tokenUrl", value: config.TokenUrl},
|
||||
{name: "userInfoUrl", value: config.UserInfoUrl},
|
||||
}
|
||||
for _, field := range required {
|
||||
if strings.TrimSpace(field.value) == "" {
|
||||
return errors.Errorf("config.oauth2Config.%s is required", field.name)
|
||||
}
|
||||
}
|
||||
for _, field := range []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{name: "authUrl", value: config.AuthUrl},
|
||||
{name: "tokenUrl", value: config.TokenUrl},
|
||||
{name: "userInfoUrl", value: config.UserInfoUrl},
|
||||
} {
|
||||
parsed, err := url.ParseRequestURI(field.value)
|
||||
if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Host == "" {
|
||||
return errors.Errorf("config.oauth2Config.%s must be an absolute HTTP(S) URL", field.name)
|
||||
}
|
||||
}
|
||||
if len(config.Scopes) == 0 {
|
||||
return errors.New("config.oauth2Config.scopes is required")
|
||||
}
|
||||
for i, scope := range config.Scopes {
|
||||
if strings.TrimSpace(scope) == "" {
|
||||
return errors.Errorf("config.oauth2Config.scopes[%d] must not be empty", i)
|
||||
}
|
||||
}
|
||||
if config.FieldMapping == nil || strings.TrimSpace(config.FieldMapping.Identifier) == "" {
|
||||
return errors.New("config.oauth2Config.fieldMapping.identifier is required")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateAndNormalizeDeploymentInstanceSetting(setting *storepb.InstanceSetting) error {
|
||||
switch setting.Key {
|
||||
case storepb.InstanceSettingKey_GENERAL:
|
||||
if setting.GetGeneralSetting() == nil {
|
||||
return errors.New("generalSetting must be populated for key GENERAL")
|
||||
}
|
||||
if offset := setting.GetGeneralSetting().WeekStartDayOffset; offset < -1 || offset > 6 {
|
||||
return errors.New("generalSetting.weekStartDayOffset must be between -1 and 6")
|
||||
}
|
||||
case storepb.InstanceSettingKey_STORAGE:
|
||||
storage := setting.GetStorageSetting()
|
||||
if storage == nil {
|
||||
return errors.New("storageSetting must be populated for key STORAGE")
|
||||
}
|
||||
if storage.UploadSizeLimitMb < 0 {
|
||||
return errors.New("storageSetting.uploadSizeLimitMb must not be negative")
|
||||
}
|
||||
if storage.StorageType == storepb.InstanceStorageSetting_S3 {
|
||||
if storage.S3Config == nil {
|
||||
return errors.New("storageSetting.s3Config is required for S3")
|
||||
}
|
||||
for _, field := range []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{name: "accessKeyId", value: storage.S3Config.AccessKeyId},
|
||||
{name: "accessKeySecret", value: storage.S3Config.AccessKeySecret},
|
||||
{name: "endpoint", value: storage.S3Config.Endpoint},
|
||||
{name: "region", value: storage.S3Config.Region},
|
||||
{name: "bucket", value: storage.S3Config.Bucket},
|
||||
} {
|
||||
if strings.TrimSpace(field.value) == "" {
|
||||
return errors.Errorf("storageSetting.s3Config.%s is required", field.name)
|
||||
}
|
||||
}
|
||||
}
|
||||
case storepb.InstanceSettingKey_MEMO_RELATED:
|
||||
if setting.GetMemoRelatedSetting() == nil {
|
||||
return errors.New("memoRelatedSetting must be populated for key MEMO_RELATED")
|
||||
}
|
||||
case storepb.InstanceSettingKey_NOTIFICATION:
|
||||
notification := setting.GetNotificationSetting()
|
||||
if notification == nil {
|
||||
return errors.New("notificationSetting must be populated for key NOTIFICATION")
|
||||
}
|
||||
if email := notification.Email; email != nil && email.Enabled {
|
||||
if strings.TrimSpace(email.SmtpHost) == "" || email.SmtpPort <= 0 || strings.TrimSpace(email.FromEmail) == "" {
|
||||
return errors.New("enabled notification email requires smtpHost, a positive smtpPort, and fromEmail")
|
||||
}
|
||||
if email.UseTls && email.UseSsl {
|
||||
return errors.New("notification email cannot enable both useTls and useSsl")
|
||||
}
|
||||
}
|
||||
case storepb.InstanceSettingKey_AI:
|
||||
if setting.GetAiSetting() == nil {
|
||||
return errors.New("aiSetting must be populated for key AI")
|
||||
}
|
||||
if err := normalizeDeploymentAISetting(setting.GetAiSetting()); err != nil {
|
||||
return err
|
||||
}
|
||||
case storepb.InstanceSettingKey_BASIC, storepb.InstanceSettingKey_TAGS:
|
||||
return errors.Errorf("key %s cannot be deployment configured", setting.Key)
|
||||
default:
|
||||
return errors.Errorf("unsupported instance setting key %s", setting.Key)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeDeploymentAISetting(setting *storepb.InstanceAISetting) error {
|
||||
providers := map[string]struct{}{}
|
||||
for i, provider := range setting.Providers {
|
||||
if provider == nil {
|
||||
return errors.Errorf("aiSetting.providers[%d] must not be null", i)
|
||||
}
|
||||
provider.Id = strings.TrimSpace(provider.Id)
|
||||
provider.Title = strings.TrimSpace(provider.Title)
|
||||
provider.Endpoint = strings.TrimSpace(provider.Endpoint)
|
||||
if provider.Id == "" || provider.Title == "" || provider.ApiKey == "" {
|
||||
return errors.Errorf("aiSetting.providers[%d] requires id, title, and apiKey", i)
|
||||
}
|
||||
if _, ok := providers[provider.Id]; ok {
|
||||
return errors.Errorf("aiSetting provider ID %q is duplicated", provider.Id)
|
||||
}
|
||||
providers[provider.Id] = struct{}{}
|
||||
switch provider.Type {
|
||||
case storepb.AIProviderType_OPENAI:
|
||||
if provider.Endpoint == "" {
|
||||
provider.Endpoint = "https://api.openai.com/v1"
|
||||
}
|
||||
case storepb.AIProviderType_GEMINI:
|
||||
if provider.Endpoint == "" {
|
||||
provider.Endpoint = "https://generativelanguage.googleapis.com/v1beta"
|
||||
}
|
||||
default:
|
||||
return errors.Errorf("aiSetting provider %q has unsupported type", provider.Id)
|
||||
}
|
||||
}
|
||||
if transcription := setting.Transcription; transcription != nil {
|
||||
transcription.ProviderId = strings.TrimSpace(transcription.ProviderId)
|
||||
transcription.Model = strings.TrimSpace(transcription.Model)
|
||||
transcription.Language = strings.TrimSpace(transcription.Language)
|
||||
transcription.Prompt = strings.TrimSpace(transcription.Prompt)
|
||||
if transcription.ProviderId != "" {
|
||||
if _, ok := providers[transcription.ProviderId]; !ok {
|
||||
return errors.Errorf("aiSetting transcription providerId %q does not reference a provider", transcription.ProviderId)
|
||||
}
|
||||
}
|
||||
if len(transcription.Model) > maxTranscriptionModelLength || len(transcription.Language) > maxTranscriptionLanguageLength || len(transcription.Prompt) > maxTranscriptionPromptLength {
|
||||
return errors.New("aiSetting transcription configuration exceeds a supported length limit")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) validateDeploymentAuthenticationState(ctx context.Context, config *deploymentConfiguration) error {
|
||||
_, generalConfigured := config.instanceSettings[storepb.InstanceSettingKey_GENERAL]
|
||||
if !generalConfigured && len(config.identityProviders) == 0 {
|
||||
general, err := s.getRawInstanceSetting(ctx, storepb.InstanceSettingKey_GENERAL.String())
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to inspect stored GENERAL setting")
|
||||
}
|
||||
if general == nil || !general.GetGeneralSetting().DisallowPasswordAuth {
|
||||
return nil
|
||||
}
|
||||
providers, err := s.listStoredIdentityProviders(ctx, &FindIdentityProvider{})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to inspect stored identity providers")
|
||||
}
|
||||
if len(providers) == 0 {
|
||||
slog.Warn("stored configuration disables password authentication for regular users but has no identity provider; unrelated deployment files remain loadable because administrator password sign-in is available")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
general, err := s.getRawInstanceSetting(ctx, storepb.InstanceSettingKey_GENERAL.String())
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read stored GENERAL setting")
|
||||
}
|
||||
if configured := config.instanceSettings[storepb.InstanceSettingKey_GENERAL]; configured != nil {
|
||||
general = cloneInstanceSetting(configured)
|
||||
}
|
||||
if general == nil || !general.GetGeneralSetting().DisallowPasswordAuth {
|
||||
return nil
|
||||
}
|
||||
providers, err := s.listStoredIdentityProviders(ctx, &FindIdentityProvider{})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to read stored identity providers")
|
||||
}
|
||||
effectiveUIDs := map[string]struct{}{}
|
||||
for _, provider := range providers {
|
||||
effectiveUIDs[provider.Uid] = struct{}{}
|
||||
}
|
||||
for uid := range config.identityProviders {
|
||||
effectiveUIDs[uid] = struct{}{}
|
||||
}
|
||||
if len(effectiveUIDs) == 0 {
|
||||
return errors.New("deployment configuration disables password authentication for regular users but has no effective identity provider")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) warnShadowedStoredIdentityProviders(ctx context.Context, config *deploymentConfiguration) error {
|
||||
if len(config.identityProviders) == 0 {
|
||||
return nil
|
||||
}
|
||||
providers, err := s.listStoredIdentityProviders(ctx, &FindIdentityProvider{})
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to inspect stored identity providers")
|
||||
}
|
||||
for _, provider := range providers {
|
||||
if _, ok := config.identityProviders[provider.Uid]; ok {
|
||||
slog.Warn("deployment identity provider shadows a stored provider; the stored configuration remains in the database", "uid", provider.Uid)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Store) setDeploymentConfiguration(config *deploymentConfiguration) {
|
||||
copy := newDeploymentConfiguration()
|
||||
for uid, provider := range config.identityProviders {
|
||||
copy.identityProviders[uid] = cloneIdentityProvider(provider)
|
||||
}
|
||||
for key, setting := range config.instanceSettings {
|
||||
copy.instanceSettings[key] = cloneInstanceSetting(setting)
|
||||
}
|
||||
s.deploymentConfigMu.Lock()
|
||||
s.deploymentConfig = copy
|
||||
s.deploymentConfigMu.Unlock()
|
||||
}
|
||||
|
||||
// IsIdentityProviderDeploymentConfigured reports whether uid is file-backed.
|
||||
func (s *Store) IsIdentityProviderDeploymentConfigured(uid string) bool {
|
||||
s.deploymentConfigMu.RLock()
|
||||
defer s.deploymentConfigMu.RUnlock()
|
||||
_, ok := s.deploymentConfig.identityProviders[uid]
|
||||
return ok
|
||||
}
|
||||
|
||||
// IsInstanceSettingDeploymentConfigured reports whether key is file-backed.
|
||||
func (s *Store) IsInstanceSettingDeploymentConfigured(key storepb.InstanceSettingKey) bool {
|
||||
s.deploymentConfigMu.RLock()
|
||||
defer s.deploymentConfigMu.RUnlock()
|
||||
_, ok := s.deploymentConfig.instanceSettings[key]
|
||||
return ok
|
||||
}
|
||||
|
||||
func (s *Store) getDeploymentIdentityProvider(uid string) *storepb.IdentityProvider {
|
||||
s.deploymentConfigMu.RLock()
|
||||
defer s.deploymentConfigMu.RUnlock()
|
||||
return cloneIdentityProvider(s.deploymentConfig.identityProviders[uid])
|
||||
}
|
||||
|
||||
func (s *Store) listDeploymentIdentityProviders() []*storepb.IdentityProvider {
|
||||
s.deploymentConfigMu.RLock()
|
||||
defer s.deploymentConfigMu.RUnlock()
|
||||
providers := make([]*storepb.IdentityProvider, 0, len(s.deploymentConfig.identityProviders))
|
||||
for _, provider := range s.deploymentConfig.identityProviders {
|
||||
providers = append(providers, cloneIdentityProvider(provider))
|
||||
}
|
||||
slices.SortFunc(providers, func(a, b *storepb.IdentityProvider) int { return strings.Compare(a.Uid, b.Uid) })
|
||||
return providers
|
||||
}
|
||||
|
||||
func (s *Store) getDeploymentInstanceSetting(key storepb.InstanceSettingKey) *storepb.InstanceSetting {
|
||||
s.deploymentConfigMu.RLock()
|
||||
defer s.deploymentConfigMu.RUnlock()
|
||||
return cloneInstanceSetting(s.deploymentConfig.instanceSettings[key])
|
||||
}
|
||||
|
||||
func cloneIdentityProvider(provider *storepb.IdentityProvider) *storepb.IdentityProvider {
|
||||
if provider == nil {
|
||||
return nil
|
||||
}
|
||||
cloned := &storepb.IdentityProvider{}
|
||||
proto.Merge(cloned, provider)
|
||||
return cloned
|
||||
}
|
||||
|
||||
func cloneInstanceSetting(setting *storepb.InstanceSetting) *storepb.InstanceSetting {
|
||||
if setting == nil {
|
||||
return nil
|
||||
}
|
||||
cloned := &storepb.InstanceSetting{}
|
||||
proto.Merge(cloned, setting)
|
||||
return cloned
|
||||
}
|
||||
@@ -0,0 +1,497 @@
|
||||
package store_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/usememos/memos/internal/profile"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
"github.com/usememos/memos/store"
|
||||
"github.com/usememos/memos/store/db/sqlite"
|
||||
)
|
||||
|
||||
func TestLoadDeploymentConfigurationPublishesRuntimeOnlyIdentityProvider(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-primary.json"), "primary-sso", "File SSO", "file-secret")
|
||||
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
effective, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, effective)
|
||||
assert.Zero(t, effective.Id)
|
||||
assert.Equal(t, "File SSO", effective.Name)
|
||||
assert.Equal(t, "file-secret", effective.Config.GetOauth2Config().ClientSecret)
|
||||
assert.True(t, stores.IsIdentityProviderDeploymentConfigured("primary-sso"))
|
||||
|
||||
stored, err := stores.GetStoredIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, stored)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationShadowsWithoutChangingStoredResources(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
stored, err := stores.CreateIdentityProvider(ctx, deploymentIdentityProvider("primary-sso", "Stored SSO", "stored-secret"))
|
||||
require.NoError(t, err)
|
||||
storedID := stored.Id
|
||||
_, err = stores.UpsertInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
WeekStartDayOffset: 1,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-primary.json"), "primary-sso", "File SSO", "file-secret")
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(dir, "memos-instance-setting-general.json"), 4, true)
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
effectiveProvider, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "File SSO", effectiveProvider.Name)
|
||||
storedProvider, err := stores.GetStoredIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, storedID, storedProvider.Id)
|
||||
assert.Equal(t, "Stored SSO", storedProvider.Name)
|
||||
|
||||
effectiveGeneral, err := stores.GetInstanceGeneralSetting(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int32(4), effectiveGeneral.WeekStartDayOffset)
|
||||
assert.True(t, effectiveGeneral.DisallowPasswordAuth)
|
||||
rawGeneral, err := stores.GetStoredInstanceSetting(ctx, &store.FindInstanceSetting{Name: storepb.InstanceSettingKey_GENERAL.String()})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, rawGeneral)
|
||||
assert.Equal(t, int32(1), rawGeneral.GetGeneralSetting().WeekStartDayOffset)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationReturnsDefensiveClones(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-primary.json"), "primary-sso", "File SSO", "file-secret")
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(dir, "memos-instance-setting-general.json"), 2, false)
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
provider, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
provider.Name = "Mutated"
|
||||
provider.Config.GetOauth2Config().ClientSecret = "mutated-secret"
|
||||
providerAgain, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "File SSO", providerAgain.Name)
|
||||
assert.Equal(t, "file-secret", providerAgain.Config.GetOauth2Config().ClientSecret)
|
||||
|
||||
general, err := stores.GetInstanceGeneralSetting(ctx)
|
||||
require.NoError(t, err)
|
||||
general.WeekStartDayOffset = 6
|
||||
generalAgain, err := stores.GetInstanceGeneralSetting(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int32(2), generalAgain.WeekStartDayOffset)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationPublishesAtomically(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
validDir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(validDir, "memos-idp-primary.json"), "primary-sso", "File SSO", "file-secret")
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, validDir))
|
||||
|
||||
invalidDir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(invalidDir, "memos-idp-primary.json"), "primary-sso", "Changed SSO", "changed-secret")
|
||||
require.NoError(t, os.WriteFile(filepath.Join(invalidDir, "memos-instance-setting-invalid.json"), []byte(`{"key":"GENERAL","unknown":true}`), 0600))
|
||||
err := stores.LoadDeploymentConfigurationDir(ctx, invalidDir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, `unknown field "unknown"`)
|
||||
|
||||
provider, getErr := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, getErr)
|
||||
assert.Equal(t, "File SSO", provider.Name)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationRejectsDuplicateResources(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-first.json"), "primary-sso", "First", "first-secret")
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-second.json"), "primary-sso", "Second", "second-secret")
|
||||
|
||||
err := stores.LoadDeploymentConfigurationDir(context.Background(), dir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, `identity provider UID "primary-sso" is declared by both`)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationValidatesAffectedAuthState(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
t.Run("managed GENERAL cannot disable regular password auth without SSO", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(dir, "memos-instance-setting-general.json"), 0, true)
|
||||
err := stores.LoadDeploymentConfigurationDir(ctx, dir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, "has no effective identity provider")
|
||||
})
|
||||
|
||||
t.Run("unrelated file does not reject unmanaged legacy state", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
_, err := stores.UpsertInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "memos-instance-setting-storage.json"), []byte(`{
|
||||
"key": "STORAGE",
|
||||
"storageSetting": {"storageType": "LOCAL"}
|
||||
}`), 0600))
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
})
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationIgnoresUnrelatedFilesAndMissingDirectory(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, ".env"), []byte("DATABASE_PASSWORD=secret"), 0600))
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, filepath.Join(t.TempDir(), "missing")))
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationSupportsLegacyIdentityProviderFilenames(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-Primary_SSO.json"), "primary-sso", "Primary", "secret")
|
||||
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
assert.True(t, stores.IsIdentityProviderDeploymentConfigured("primary-sso"))
|
||||
|
||||
invalidDir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(invalidDir, "memos-idp-Bad.json"), []byte("not-json"), 0600))
|
||||
require.Error(t, stores.LoadDeploymentConfigurationDir(ctx, invalidDir))
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationAcceptsSaturdayWeekStart(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(dir, "memos-instance-setting-general.json"), -1, false)
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(context.Background(), dir))
|
||||
|
||||
general, err := stores.GetInstanceGeneralSetting(context.Background())
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, int32(-1), general.WeekStartDayOffset)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationAcceptsRegularFileSymlink(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
target := filepath.Join(t.TempDir(), "provider.json")
|
||||
writeDeploymentIdentityProvider(t, target, "primary-sso", "Primary", "secret")
|
||||
require.NoError(t, os.Symlink(target, filepath.Join(dir, "memos-idp-primary.json")))
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
assert.True(t, stores.IsIdentityProviderDeploymentConfigured("primary-sso"))
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationSupportsEveryProvisionableSettingGroup(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
settings := map[string]*storepb.InstanceSetting{
|
||||
"memos-instance-setting-general.json": {
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{WeekStartDayOffset: 1}},
|
||||
},
|
||||
"memos-instance-setting-storage.json": {
|
||||
Key: storepb.InstanceSettingKey_STORAGE,
|
||||
Value: &storepb.InstanceSetting_StorageSetting{StorageSetting: &storepb.InstanceStorageSetting{StorageType: storepb.InstanceStorageSetting_LOCAL}},
|
||||
},
|
||||
"memos-instance-setting-memo-related.json": {
|
||||
Key: storepb.InstanceSettingKey_MEMO_RELATED,
|
||||
Value: &storepb.InstanceSetting_MemoRelatedSetting{MemoRelatedSetting: &storepb.InstanceMemoRelatedSetting{
|
||||
ContentLengthLimit: store.DefaultContentLengthLimit,
|
||||
Reactions: []string{"👍"},
|
||||
}},
|
||||
},
|
||||
"memos-instance-setting-notification.json": {
|
||||
Key: storepb.InstanceSettingKey_NOTIFICATION,
|
||||
Value: &storepb.InstanceSetting_NotificationSetting{NotificationSetting: &storepb.InstanceNotificationSetting{}},
|
||||
},
|
||||
"memos-instance-setting-ai.json": {
|
||||
Key: storepb.InstanceSettingKey_AI,
|
||||
Value: &storepb.InstanceSetting_AiSetting{AiSetting: &storepb.InstanceAISetting{Providers: []*storepb.AIProviderConfig{
|
||||
{Id: "primary", Title: "Primary", Type: storepb.AIProviderType_OPENAI, ApiKey: "ai-secret"},
|
||||
}}},
|
||||
},
|
||||
}
|
||||
for filename, setting := range settings {
|
||||
writeDeploymentMessage(t, filepath.Join(dir, filename), setting)
|
||||
}
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
for _, setting := range settings {
|
||||
assert.True(t, stores.IsInstanceSettingDeploymentConfigured(setting.Key))
|
||||
}
|
||||
ai, err := stores.GetInstanceAISetting(ctx)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, ai.Providers, 1)
|
||||
assert.Equal(t, "https://api.openai.com/v1", ai.Providers[0].Endpoint)
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationRejectsInvalidSettingResources(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
content string
|
||||
errorString string
|
||||
}{
|
||||
{name: "BASIC", content: `{"key":"BASIC","basicSetting":{}}`, errorString: "cannot be deployment configured"},
|
||||
{name: "TAGS", content: `{"key":"TAGS","tagsSetting":{}}`, errorString: "cannot be deployment configured"},
|
||||
{name: "mismatched oneof", content: `{"key":"GENERAL","storageSetting":{}}`, errorString: "generalSetting must be populated"},
|
||||
{name: "invalid week start", content: `{"key":"GENERAL","generalSetting":{"weekStartDayOffset":-2}}`, errorString: "must be between -1 and 6"},
|
||||
{name: "unknown field", content: `{"key":"GENERAL","generalSetting":{},"typo":true}`, errorString: `unknown field "typo"`},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "memos-instance-setting-invalid.json"), []byte(test.content), 0600))
|
||||
err := stores.LoadDeploymentConfigurationDir(context.Background(), dir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, test.errorString)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadDeploymentConfigurationBoundsFilesAndRedactsDecodeErrors(t *testing.T) {
|
||||
t.Run("oversized file", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "memos-idp-oversized.json"), []byte(strings.Repeat("x", (1<<20)+1)), 0600))
|
||||
err := stores.LoadDeploymentConfigurationDir(context.Background(), dir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, "exceeds 1048576 bytes")
|
||||
})
|
||||
|
||||
t.Run("secret value is not included in a type error", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
content := `{
|
||||
"uid":"primary-sso",
|
||||
"name":"Primary",
|
||||
"type":"OAUTH2",
|
||||
"config":{"oauth2Config":{"clientSecret":{"value":"must-not-appear"}}}
|
||||
}`
|
||||
require.NoError(t, os.WriteFile(filepath.Join(dir, "memos-idp-primary.json"), []byte(content), 0600))
|
||||
err := stores.LoadDeploymentConfigurationDir(context.Background(), dir)
|
||||
require.Error(t, err)
|
||||
assert.NotContains(t, err.Error(), "must-not-appear")
|
||||
})
|
||||
}
|
||||
|
||||
func TestAuthenticationConfigurationMutationsAreSafe(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
|
||||
t.Run("rejects disabling regular password auth without an IdP", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
_, err := stores.UpsertInstanceGeneralSettingSafely(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.ErrorIs(t, err, store.ErrUnsafeAuthenticationConfiguration)
|
||||
stored, getErr := stores.GetStoredInstanceSetting(ctx, &store.FindInstanceSetting{Name: storepb.InstanceSettingKey_GENERAL.String()})
|
||||
require.NoError(t, getErr)
|
||||
assert.Nil(t, stored)
|
||||
})
|
||||
|
||||
t.Run("rejects deleting the last effective IdP", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
provider, err := stores.CreateIdentityProvider(ctx, deploymentIdentityProvider("primary-sso", "Primary", "secret"))
|
||||
require.NoError(t, err)
|
||||
_, err = stores.UpsertInstanceGeneralSettingSafely(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
err = stores.DeleteIdentityProviderSafely(ctx, &store.DeleteIdentityProvider{ID: provider.Id})
|
||||
require.ErrorIs(t, err, store.ErrUnsafeAuthenticationConfiguration)
|
||||
stored, getErr := stores.GetStoredIdentityProvider(ctx, &store.FindIdentityProvider{ID: &provider.Id})
|
||||
require.NoError(t, getErr)
|
||||
assert.NotNil(t, stored)
|
||||
})
|
||||
|
||||
t.Run("file-backed IdP satisfies the invariant", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-primary.json"), "primary-sso", "Primary", "secret")
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
_, err := stores.UpsertInstanceGeneralSettingSafely(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
})
|
||||
|
||||
t.Run("allows unrelated edits to an existing unsafe GENERAL state", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
_, err := stores.UpsertInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
_, err = stores.UpsertInstanceGeneralSettingSafely(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
WeekStartDayOffset: 1,
|
||||
}},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
})
|
||||
|
||||
t.Run("rejects making the stored fallback unsafe under deployment GENERAL", func(t *testing.T) {
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
dir := t.TempDir()
|
||||
writeDeploymentGeneralSetting(t, filepath.Join(dir, "memos-instance-setting-general.json"), 0, false)
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
_, err := stores.UpsertInstanceGeneralSettingSafely(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
})
|
||||
require.ErrorIs(t, err, store.ErrUnsafeAuthenticationConfiguration)
|
||||
})
|
||||
}
|
||||
|
||||
func TestListIdentityProvidersPreservesStoredOrder(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newDeploymentConfigurationTestStore(t)
|
||||
_, err := stores.CreateIdentityProvider(ctx, deploymentIdentityProvider("zeta-sso", "Stored Zeta", "stored-secret"))
|
||||
require.NoError(t, err)
|
||||
_, err = stores.CreateIdentityProvider(ctx, deploymentIdentityProvider("alpha-test", "Stored Alpha", "stored-secret"))
|
||||
require.NoError(t, err)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-zeta.json"), "zeta-sso", "File Zeta", "file-secret")
|
||||
writeDeploymentIdentityProvider(t, filepath.Join(dir, "memos-idp-beta.json"), "beta-file", "File Beta", "file-secret")
|
||||
require.NoError(t, stores.LoadDeploymentConfigurationDir(ctx, dir))
|
||||
|
||||
providers, err := stores.ListIdentityProviders(ctx, &store.FindIdentityProvider{})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, providers, 3)
|
||||
assert.Equal(t, []string{"zeta-sso", "alpha-test", "beta-file"}, []string{providers[0].Uid, providers[1].Uid, providers[2].Uid})
|
||||
assert.Equal(t, "File Zeta", providers[0].Name)
|
||||
}
|
||||
|
||||
func newDeploymentConfigurationTestStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
p := &profile.Profile{
|
||||
Data: t.TempDir(),
|
||||
Driver: "sqlite",
|
||||
DSN: filepath.Join(t.TempDir(), "deployment.db"),
|
||||
}
|
||||
driver, err := sqlite.NewDB(p)
|
||||
require.NoError(t, err)
|
||||
stores := store.New(driver, p)
|
||||
require.NoError(t, stores.Migrate(context.Background()))
|
||||
t.Cleanup(func() {
|
||||
require.NoError(t, stores.Close())
|
||||
})
|
||||
return stores
|
||||
}
|
||||
|
||||
func deploymentIdentityProvider(uid, title, secret string) *storepb.IdentityProvider {
|
||||
return &storepb.IdentityProvider{
|
||||
Uid: uid,
|
||||
Name: title,
|
||||
Type: storepb.IdentityProvider_OAUTH2,
|
||||
Config: &storepb.IdentityProviderConfig{Config: &storepb.IdentityProviderConfig_Oauth2Config{Oauth2Config: &storepb.OAuth2Config{
|
||||
ClientId: "client-id",
|
||||
ClientSecret: secret,
|
||||
AuthUrl: "https://example.com/authorize",
|
||||
TokenUrl: "https://example.com/token",
|
||||
UserInfoUrl: "https://example.com/userinfo",
|
||||
Scopes: []string{"profile", "email"},
|
||||
FieldMapping: &storepb.FieldMapping{Identifier: "sub"},
|
||||
}}},
|
||||
}
|
||||
}
|
||||
|
||||
func writeDeploymentIdentityProvider(t *testing.T, path, uid, title, secret string) {
|
||||
t.Helper()
|
||||
content := `{
|
||||
"uid": "` + uid + `",
|
||||
"name": "` + title + `",
|
||||
"type": "OAUTH2",
|
||||
"config": {
|
||||
"oauth2Config": {
|
||||
"clientId": "client-id",
|
||||
"clientSecret": "` + secret + `",
|
||||
"authUrl": "https://example.com/authorize",
|
||||
"tokenUrl": "https://example.com/token",
|
||||
"userInfoUrl": "https://example.com/userinfo",
|
||||
"scopes": ["profile", "email"],
|
||||
"fieldMapping": {"identifier": "sub"}
|
||||
}
|
||||
}
|
||||
}`
|
||||
require.False(t, strings.Contains(uid+title+secret, `"`))
|
||||
require.NoError(t, os.WriteFile(path, []byte(content), 0600))
|
||||
}
|
||||
|
||||
func writeDeploymentGeneralSetting(t *testing.T, path string, weekStart int32, disallowPasswordAuth bool) {
|
||||
t.Helper()
|
||||
content := `{
|
||||
"key": "GENERAL",
|
||||
"generalSetting": {
|
||||
"weekStartDayOffset": ` + assertInt32(weekStart) + `,
|
||||
"disallowPasswordAuth": ` + assertBool(disallowPasswordAuth) + `
|
||||
}
|
||||
}`
|
||||
require.NoError(t, os.WriteFile(path, []byte(content), 0600))
|
||||
}
|
||||
|
||||
func writeDeploymentMessage(t *testing.T, path string, message proto.Message) {
|
||||
t.Helper()
|
||||
content, err := (protojson.MarshalOptions{Indent: " "}).Marshal(message)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, os.WriteFile(path, content, 0600))
|
||||
}
|
||||
|
||||
func assertInt32(value int32) string {
|
||||
return strconv.FormatInt(int64(value), 10)
|
||||
}
|
||||
|
||||
func assertBool(value bool) string {
|
||||
if value {
|
||||
return "true"
|
||||
}
|
||||
return "false"
|
||||
}
|
||||
|
||||
func ptr[T any](value T) *T {
|
||||
return &value
|
||||
}
|
||||
@@ -58,6 +58,8 @@ type Driver interface {
|
||||
ListIdentityProviders(ctx context.Context, find *FindIdentityProvider) ([]*IdentityProvider, error)
|
||||
UpdateIdentityProvider(ctx context.Context, update *UpdateIdentityProvider) (*IdentityProvider, error)
|
||||
DeleteIdentityProvider(ctx context.Context, delete *DeleteIdentityProvider) error
|
||||
ApplyAuthenticationConfigMutation(ctx context.Context, mutation *AuthenticationConfigMutation) error
|
||||
IsRetryableAuthenticationMutationError(err error) bool
|
||||
|
||||
// Inbox model related methods.
|
||||
CreateInbox(ctx context.Context, create *Inbox) (*Inbox, error)
|
||||
|
||||
+58
-1
@@ -52,6 +52,48 @@ func (s *Store) CreateIdentityProvider(ctx context.Context, create *storepb.Iden
|
||||
}
|
||||
|
||||
func (s *Store) ListIdentityProviders(ctx context.Context, find *FindIdentityProvider) ([]*storepb.IdentityProvider, error) {
|
||||
stored, err := s.listStoredIdentityProviders(ctx, find)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// File-backed providers do not have database IDs. ID-filtered reads are raw
|
||||
// stored-resource lookups used by mutation paths.
|
||||
if find.ID != nil {
|
||||
return stored, nil
|
||||
}
|
||||
if find.UID != nil {
|
||||
if provider := s.getDeploymentIdentityProvider(*find.UID); provider != nil {
|
||||
return []*storepb.IdentityProvider{provider}, nil
|
||||
}
|
||||
return stored, nil
|
||||
}
|
||||
|
||||
deploymentProviders := s.listDeploymentIdentityProviders()
|
||||
deploymentByUID := make(map[string]*storepb.IdentityProvider, len(deploymentProviders))
|
||||
for _, provider := range deploymentProviders {
|
||||
deploymentByUID[provider.Uid] = provider
|
||||
}
|
||||
identityProviders := make([]*storepb.IdentityProvider, 0, len(stored)+len(deploymentProviders))
|
||||
for _, provider := range stored {
|
||||
if configured := deploymentByUID[provider.Uid]; configured != nil {
|
||||
identityProviders = append(identityProviders, configured)
|
||||
delete(deploymentByUID, provider.Uid)
|
||||
continue
|
||||
}
|
||||
identityProviders = append(identityProviders, provider)
|
||||
}
|
||||
// listDeploymentIdentityProviders is UID-sorted, so providers that exist
|
||||
// only in deployment configuration are appended deterministically without
|
||||
// reordering existing database-backed providers.
|
||||
for _, provider := range deploymentProviders {
|
||||
if deploymentByUID[provider.Uid] != nil {
|
||||
identityProviders = append(identityProviders, provider)
|
||||
}
|
||||
}
|
||||
return identityProviders, nil
|
||||
}
|
||||
|
||||
func (s *Store) listStoredIdentityProviders(ctx context.Context, find *FindIdentityProvider) ([]*storepb.IdentityProvider, error) {
|
||||
list, err := s.driver.ListIdentityProviders(ctx, find)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -77,13 +119,28 @@ func (s *Store) GetIdentityProvider(ctx context.Context, find *FindIdentityProvi
|
||||
return nil, nil
|
||||
}
|
||||
if len(list) > 1 {
|
||||
return nil, errors.Errorf("Found multiple identity providers with ID %d", *find.ID)
|
||||
return nil, errors.New("found multiple identity providers")
|
||||
}
|
||||
|
||||
identityProvider := list[0]
|
||||
return identityProvider, nil
|
||||
}
|
||||
|
||||
// GetStoredIdentityProvider returns a database-backed provider without deployment shadowing.
|
||||
func (s *Store) GetStoredIdentityProvider(ctx context.Context, find *FindIdentityProvider) (*storepb.IdentityProvider, error) {
|
||||
list, err := s.listStoredIdentityProviders(ctx, find)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(list) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
if len(list) > 1 {
|
||||
return nil, errors.New("found multiple stored identity providers")
|
||||
}
|
||||
return list[0], nil
|
||||
}
|
||||
|
||||
type UpdateIdentityProviderV1 struct {
|
||||
ID int32
|
||||
Type storepb.IdentityProvider_Type
|
||||
|
||||
@@ -1,165 +0,0 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
|
||||
"github.com/usememos/memos/internal/base"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
)
|
||||
|
||||
const (
|
||||
maxIdentityProviderBootstrapSize = 1 << 20
|
||||
defaultIdentityProviderBootstrapDir = "/etc/secrets"
|
||||
identityProviderBootstrapPrefix = "memos-idp-"
|
||||
identityProviderBootstrapSuffix = ".json"
|
||||
)
|
||||
|
||||
// ApplyIdentityProviderBootstrapDir validates and reconciles identity providers
|
||||
// from memos-idp-*.json files in a directory. Providers not named in those files
|
||||
// are left unchanged.
|
||||
func (s *Store) ApplyIdentityProviderBootstrapDir(ctx context.Context, dir string) error {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return errors.Wrap(err, "failed to read identity provider bootstrap directory")
|
||||
}
|
||||
|
||||
providers := []*storepb.IdentityProvider{}
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() || !strings.HasPrefix(entry.Name(), identityProviderBootstrapPrefix) || !strings.HasSuffix(entry.Name(), identityProviderBootstrapSuffix) {
|
||||
continue
|
||||
}
|
||||
provider, err := readIdentityProviderBootstrap(filepath.Join(dir, entry.Name()))
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "invalid identity provider bootstrap file %q", entry.Name())
|
||||
}
|
||||
providers = append(providers, provider)
|
||||
}
|
||||
if err := validateIdentityProviderBootstrap(providers); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, provider := range providers {
|
||||
existing, err := s.GetIdentityProvider(ctx, &FindIdentityProvider{UID: &provider.Uid})
|
||||
if err != nil {
|
||||
return errors.Wrapf(err, "failed to look up identity provider %q", provider.Uid)
|
||||
}
|
||||
if existing == nil {
|
||||
if _, err := s.CreateIdentityProvider(ctx, provider); err != nil {
|
||||
return errors.Wrapf(err, "failed to create identity provider %q", provider.Uid)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if existing.Type != provider.Type {
|
||||
return errors.Errorf("identity provider %q has type %s, expected %s", provider.Uid, existing.Type, provider.Type)
|
||||
}
|
||||
|
||||
title := provider.Name
|
||||
identifierFilter := provider.IdentifierFilter
|
||||
if _, err := s.UpdateIdentityProvider(ctx, &UpdateIdentityProviderV1{
|
||||
ID: existing.Id,
|
||||
Type: existing.Type,
|
||||
Name: &title,
|
||||
IdentifierFilter: &identifierFilter,
|
||||
Config: provider.Config,
|
||||
}); err != nil {
|
||||
return errors.Wrapf(err, "failed to update identity provider %q", provider.Uid)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func readIdentityProviderBootstrap(path string) (*storepb.IdentityProvider, error) {
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to read identity provider bootstrap file")
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
content, err := io.ReadAll(io.LimitReader(file, maxIdentityProviderBootstrapSize+1))
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to read identity provider bootstrap file")
|
||||
}
|
||||
if len(content) > maxIdentityProviderBootstrapSize {
|
||||
return nil, errors.Errorf("identity provider bootstrap file exceeds %d bytes", maxIdentityProviderBootstrapSize)
|
||||
}
|
||||
|
||||
provider := &storepb.IdentityProvider{}
|
||||
if err := protojson.Unmarshal(content, provider); err != nil {
|
||||
return nil, errors.Wrap(err, "failed to decode identity provider bootstrap file")
|
||||
}
|
||||
return provider, nil
|
||||
}
|
||||
|
||||
func validateIdentityProviderBootstrap(providers []*storepb.IdentityProvider) error {
|
||||
seenUIDs := make(map[string]struct{}, len(providers))
|
||||
for i, provider := range providers {
|
||||
if provider.Id != 0 {
|
||||
return errors.Errorf("identityProviders[%d].id must be omitted", i)
|
||||
}
|
||||
if !base.UIDMatcher.MatchString(provider.Uid) {
|
||||
return errors.Errorf("identityProviders[%d].uid is invalid", i)
|
||||
}
|
||||
if _, exists := seenUIDs[provider.Uid]; exists {
|
||||
return errors.Errorf("identityProviders[%d].uid duplicates %q", i, provider.Uid)
|
||||
}
|
||||
seenUIDs[provider.Uid] = struct{}{}
|
||||
if strings.TrimSpace(provider.Name) == "" {
|
||||
return errors.Errorf("identityProviders[%d].name is required", i)
|
||||
}
|
||||
if provider.Type != storepb.IdentityProvider_OAUTH2 {
|
||||
return errors.Errorf("identityProviders[%d].type must be OAUTH2", i)
|
||||
}
|
||||
if err := validateBootstrapOAuth2Config(provider.Config.GetOauth2Config(), i); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateBootstrapOAuth2Config(config *storepb.OAuth2Config, index int) error {
|
||||
if config == nil {
|
||||
return errors.Errorf("identityProviders[%d].config.oauth2Config is required", index)
|
||||
}
|
||||
required := []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{name: "clientId", value: config.ClientId},
|
||||
{name: "clientSecret", value: config.ClientSecret},
|
||||
{name: "authUrl", value: config.AuthUrl},
|
||||
{name: "tokenUrl", value: config.TokenUrl},
|
||||
{name: "userInfoUrl", value: config.UserInfoUrl},
|
||||
}
|
||||
if config.FieldMapping == nil {
|
||||
return errors.Errorf("identityProviders[%d].config.oauth2Config.fieldMapping is required", index)
|
||||
}
|
||||
required = append(required, struct {
|
||||
name string
|
||||
value string
|
||||
}{name: "fieldMapping.identifier", value: config.FieldMapping.Identifier})
|
||||
for _, field := range required {
|
||||
if strings.TrimSpace(field.value) == "" {
|
||||
return errors.Errorf("identityProviders[%d].config.oauth2Config.%s is required", index, field.name)
|
||||
}
|
||||
}
|
||||
if len(config.Scopes) == 0 {
|
||||
return errors.Errorf("identityProviders[%d].config.oauth2Config.scopes is required", index)
|
||||
}
|
||||
for scopeIndex, scope := range config.Scopes {
|
||||
if strings.TrimSpace(scope) == "" {
|
||||
return errors.Errorf("identityProviders[%d].config.oauth2Config.scopes[%d] must not be empty", index, scopeIndex)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,180 +0,0 @@
|
||||
package store_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/usememos/memos/internal/profile"
|
||||
"github.com/usememos/memos/store"
|
||||
"github.com/usememos/memos/store/db/sqlite"
|
||||
)
|
||||
|
||||
func TestApplyIdentityProviderBootstrapCreatesAndUpdatesProvider(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
bootstrapPath := filepath.Join(bootstrapDir, "memos-idp-primary.json")
|
||||
|
||||
writeIdentityProviderBootstrap(t, bootstrapPath, "Initial SSO", "initial-secret")
|
||||
require.NoError(t, stores.ApplyIdentityProviderBootstrapDir(ctx, bootstrapDir))
|
||||
|
||||
provider, err := stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, provider)
|
||||
assert.Equal(t, "Initial SSO", provider.Name)
|
||||
assert.Equal(t, "initial-secret", provider.Config.GetOauth2Config().ClientSecret)
|
||||
providerID := provider.Id
|
||||
|
||||
writeIdentityProviderBootstrap(t, bootstrapPath, "Updated SSO", "rotated-secret")
|
||||
require.NoError(t, stores.ApplyIdentityProviderBootstrapDir(ctx, bootstrapDir))
|
||||
|
||||
provider, err = stores.GetIdentityProvider(ctx, &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, provider)
|
||||
assert.Equal(t, providerID, provider.Id)
|
||||
assert.Equal(t, "Updated SSO", provider.Name)
|
||||
assert.Equal(t, "rotated-secret", provider.Config.GetOauth2Config().ClientSecret)
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapValidatesEntireFileBeforeWriting(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
writeIdentityProviderBootstrap(t, filepath.Join(bootstrapDir, "memos-idp-a-valid.json"), "Valid Provider", "must-not-appear-in-errors")
|
||||
invalidContent := `{
|
||||
"uid": "invalid/provider",
|
||||
"name": "Invalid Provider",
|
||||
"type": "OAUTH2",
|
||||
"config": {
|
||||
"oauth2Config": {
|
||||
"clientId": "client-id",
|
||||
"clientSecret": "another-secret",
|
||||
"authUrl": "https://example.com/authorize",
|
||||
"tokenUrl": "https://example.com/token",
|
||||
"userInfoUrl": "https://example.com/userinfo",
|
||||
"scopes": ["profile"],
|
||||
"fieldMapping": {"identifier": "sub"}
|
||||
}
|
||||
}
|
||||
}`
|
||||
require.NoError(t, os.WriteFile(filepath.Join(bootstrapDir, "memos-idp-z-invalid.json"), []byte(invalidContent), 0600))
|
||||
|
||||
err := stores.ApplyIdentityProviderBootstrapDir(ctx, bootstrapDir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, "identityProviders[1].uid is invalid")
|
||||
assert.NotContains(t, err.Error(), "must-not-appear-in-errors")
|
||||
assert.NotContains(t, err.Error(), "another-secret")
|
||||
|
||||
providers, err := stores.ListIdentityProviders(ctx, &store.FindIdentityProvider{})
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, providers)
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapRejectsUnknownFields(t *testing.T) {
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
bootstrapPath := filepath.Join(bootstrapDir, "memos-idp-invalid.json")
|
||||
require.NoError(t, os.WriteFile(bootstrapPath, []byte(`{"uid":"primary-sso","unexpected":true}`), 0600))
|
||||
|
||||
err := stores.ApplyIdentityProviderBootstrapDir(context.Background(), bootstrapDir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, `unknown field "unexpected"`)
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapDirRejectsDuplicateUIDsAcrossFiles(t *testing.T) {
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
writeIdentityProviderBootstrap(t, filepath.Join(bootstrapDir, "memos-idp-first.json"), "First Provider", "first-secret")
|
||||
writeIdentityProviderBootstrap(t, filepath.Join(bootstrapDir, "memos-idp-second.json"), "Second Provider", "second-secret")
|
||||
|
||||
err := stores.ApplyIdentityProviderBootstrapDir(context.Background(), bootstrapDir)
|
||||
require.Error(t, err)
|
||||
assert.ErrorContains(t, err, `uid duplicates "primary-sso"`)
|
||||
|
||||
providers, listErr := stores.ListIdentityProviders(context.Background(), &store.FindIdentityProvider{})
|
||||
require.NoError(t, listErr)
|
||||
assert.Empty(t, providers)
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapDirIgnoresUnrelatedSecretFiles(t *testing.T) {
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(bootstrapDir, ".env"), []byte("DATABASE_PASSWORD=secret"), 0600))
|
||||
writeIdentityProviderBootstrap(t, filepath.Join(bootstrapDir, "memos-idp-primary.json"), "Primary SSO", "oauth-secret")
|
||||
|
||||
require.NoError(t, stores.ApplyIdentityProviderBootstrapDir(context.Background(), bootstrapDir))
|
||||
provider, err := stores.GetIdentityProvider(context.Background(), &store.FindIdentityProvider{UID: ptr("primary-sso")})
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, provider)
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapDirIgnoresDirectoryWithoutMatchingFiles(t *testing.T) {
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := t.TempDir()
|
||||
require.NoError(t, os.WriteFile(filepath.Join(bootstrapDir, "database-password"), []byte("secret"), 0600))
|
||||
|
||||
require.NoError(t, stores.ApplyIdentityProviderBootstrapDir(context.Background(), bootstrapDir))
|
||||
}
|
||||
|
||||
func TestApplyIdentityProviderBootstrapDirIgnoresMissingDirectory(t *testing.T) {
|
||||
stores := newIdentityProviderBootstrapTestStore(t)
|
||||
bootstrapDir := filepath.Join(t.TempDir(), "missing")
|
||||
|
||||
require.NoError(t, stores.ApplyIdentityProviderBootstrapDir(context.Background(), bootstrapDir))
|
||||
}
|
||||
|
||||
func newIdentityProviderBootstrapTestStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
p := &profile.Profile{
|
||||
Data: t.TempDir(),
|
||||
Driver: "sqlite",
|
||||
DSN: filepath.Join(t.TempDir(), "bootstrap.db"),
|
||||
}
|
||||
driver, err := sqlite.NewDB(p)
|
||||
require.NoError(t, err)
|
||||
stores := store.New(driver, p)
|
||||
require.NoError(t, stores.Migrate(context.Background()))
|
||||
t.Cleanup(func() {
|
||||
require.NoError(t, stores.Close())
|
||||
})
|
||||
return stores
|
||||
}
|
||||
|
||||
func writeIdentityProviderBootstrap(t *testing.T, path, title, secret string) {
|
||||
t.Helper()
|
||||
content := `{
|
||||
"uid": "primary-sso",
|
||||
"name": "` + title + `",
|
||||
"type": "OAUTH2",
|
||||
"identifierFilter": "",
|
||||
"config": {
|
||||
"oauth2Config": {
|
||||
"clientId": "client-id",
|
||||
"clientSecret": "` + secret + `",
|
||||
"authUrl": "https://example.com/authorize",
|
||||
"tokenUrl": "https://example.com/token",
|
||||
"userInfoUrl": "https://example.com/userinfo",
|
||||
"scopes": ["profile", "email"],
|
||||
"fieldMapping": {
|
||||
"identifier": "sub",
|
||||
"displayName": "name",
|
||||
"email": "email",
|
||||
"avatarUrl": "picture"
|
||||
}
|
||||
}
|
||||
}
|
||||
}`
|
||||
require.False(t, strings.Contains(title, `"`))
|
||||
require.False(t, strings.Contains(secret, `"`))
|
||||
require.NoError(t, os.WriteFile(path, []byte(content), 0600))
|
||||
}
|
||||
|
||||
func ptr[T any](value T) *T {
|
||||
return &value
|
||||
}
|
||||
@@ -1,7 +1,9 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"context"
|
||||
"slices"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
"google.golang.org/protobuf/encoding/protojson"
|
||||
@@ -59,11 +61,43 @@ func (s *Store) UpsertInstanceSetting(ctx context.Context, upsert *storepb.Insta
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "Failed to convert instance setting")
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, instanceSetting.Key.String(), instanceSetting)
|
||||
s.cacheInstanceSetting(ctx, instanceSetting)
|
||||
return instanceSetting, nil
|
||||
}
|
||||
|
||||
func (s *Store) ListInstanceSettings(ctx context.Context, find *FindInstanceSetting) ([]*storepb.InstanceSetting, error) {
|
||||
stored, err := s.listStoredInstanceSettings(ctx, find)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if find.Name != "" {
|
||||
key, ok := storepb.InstanceSettingKey_value[find.Name]
|
||||
if ok {
|
||||
if configured := s.getDeploymentInstanceSetting(storepb.InstanceSettingKey(key)); configured != nil {
|
||||
return []*storepb.InstanceSetting{configured}, nil
|
||||
}
|
||||
}
|
||||
return stored, nil
|
||||
}
|
||||
|
||||
byKey := make(map[storepb.InstanceSettingKey]*storepb.InstanceSetting, len(stored))
|
||||
for _, setting := range stored {
|
||||
byKey[setting.Key] = setting
|
||||
}
|
||||
s.deploymentConfigMu.RLock()
|
||||
for key, setting := range s.deploymentConfig.instanceSettings {
|
||||
byKey[key] = cloneInstanceSetting(setting)
|
||||
}
|
||||
s.deploymentConfigMu.RUnlock()
|
||||
settings := make([]*storepb.InstanceSetting, 0, len(byKey))
|
||||
for _, setting := range byKey {
|
||||
settings = append(settings, setting)
|
||||
}
|
||||
slices.SortFunc(settings, func(a, b *storepb.InstanceSetting) int { return cmp.Compare(a.Key, b.Key) })
|
||||
return settings, nil
|
||||
}
|
||||
|
||||
func (s *Store) listStoredInstanceSettings(ctx context.Context, find *FindInstanceSetting) ([]*storepb.InstanceSetting, error) {
|
||||
list, err := s.driver.ListInstanceSettings(ctx, find)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -78,13 +112,37 @@ func (s *Store) ListInstanceSettings(ctx context.Context, find *FindInstanceSett
|
||||
if instanceSetting == nil {
|
||||
continue
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, instanceSetting.Key.String(), instanceSetting)
|
||||
s.cacheInstanceSetting(ctx, instanceSetting)
|
||||
instanceSettings = append(instanceSettings, instanceSetting)
|
||||
}
|
||||
return instanceSettings, nil
|
||||
}
|
||||
|
||||
func (s *Store) getRawInstanceSetting(ctx context.Context, name string) (*storepb.InstanceSetting, error) {
|
||||
list, err := s.listStoredInstanceSettings(ctx, &FindInstanceSetting{Name: name})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(list) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
if len(list) > 1 {
|
||||
return nil, errors.Errorf("found multiple stored instance settings with key %s", name)
|
||||
}
|
||||
return list[0], nil
|
||||
}
|
||||
|
||||
// GetStoredInstanceSetting returns a database-backed setting without deployment shadowing.
|
||||
func (s *Store) GetStoredInstanceSetting(ctx context.Context, find *FindInstanceSetting) (*storepb.InstanceSetting, error) {
|
||||
return s.getRawInstanceSetting(ctx, find.Name)
|
||||
}
|
||||
|
||||
func (s *Store) GetInstanceSetting(ctx context.Context, find *FindInstanceSetting) (*storepb.InstanceSetting, error) {
|
||||
if key, ok := storepb.InstanceSettingKey_value[find.Name]; ok {
|
||||
if setting := s.getDeploymentInstanceSetting(storepb.InstanceSettingKey(key)); setting != nil {
|
||||
return setting, nil
|
||||
}
|
||||
}
|
||||
if cache, ok := s.instanceSettingCache.Get(ctx, find.Name); ok {
|
||||
instanceSetting, ok := cache.(*storepb.InstanceSetting)
|
||||
if ok {
|
||||
@@ -105,6 +163,13 @@ func (s *Store) GetInstanceSetting(ctx context.Context, find *FindInstanceSettin
|
||||
return list[0], nil
|
||||
}
|
||||
|
||||
func (s *Store) cacheInstanceSetting(ctx context.Context, setting *storepb.InstanceSetting) {
|
||||
if setting == nil || s.IsInstanceSettingDeploymentConfigured(setting.Key) {
|
||||
return
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, setting.Key.String(), setting)
|
||||
}
|
||||
|
||||
func (s *Store) GetInstanceBasicSetting(ctx context.Context) (*storepb.InstanceBasicSetting, error) {
|
||||
instanceSetting, err := s.GetInstanceSetting(ctx, &FindInstanceSetting{
|
||||
Name: storepb.InstanceSettingKey_BASIC.String(),
|
||||
@@ -117,7 +182,7 @@ func (s *Store) GetInstanceBasicSetting(ctx context.Context) (*storepb.InstanceB
|
||||
if instanceSetting != nil {
|
||||
instanceBasicSetting = instanceSetting.GetBasicSetting()
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_BASIC.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_BASIC,
|
||||
Value: &storepb.InstanceSetting_BasicSetting{BasicSetting: instanceBasicSetting},
|
||||
})
|
||||
@@ -136,7 +201,7 @@ func (s *Store) GetInstanceGeneralSetting(ctx context.Context) (*storepb.Instanc
|
||||
if instanceSetting != nil {
|
||||
instanceGeneralSetting = instanceSetting.GetGeneralSetting()
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_GENERAL.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: instanceGeneralSetting},
|
||||
})
|
||||
@@ -167,7 +232,7 @@ func (s *Store) GetInstanceMemoRelatedSetting(ctx context.Context) (*storepb.Ins
|
||||
if len(instanceMemoRelatedSetting.Reactions) == 0 {
|
||||
instanceMemoRelatedSetting.Reactions = append(instanceMemoRelatedSetting.Reactions, DefaultReactions...)
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_MEMO_RELATED.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_MEMO_RELATED,
|
||||
Value: &storepb.InstanceSetting_MemoRelatedSetting{MemoRelatedSetting: instanceMemoRelatedSetting},
|
||||
})
|
||||
@@ -189,7 +254,7 @@ func (s *Store) GetInstanceTagsSetting(ctx context.Context) (*storepb.InstanceTa
|
||||
if instanceTagsSetting.Tags == nil {
|
||||
instanceTagsSetting.Tags = map[string]*storepb.InstanceTagMetadata{}
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_TAGS.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_TAGS,
|
||||
Value: &storepb.InstanceSetting_TagsSetting{TagsSetting: instanceTagsSetting},
|
||||
})
|
||||
@@ -211,7 +276,7 @@ func (s *Store) GetInstanceNotificationSetting(ctx context.Context) (*storepb.In
|
||||
if instanceNotificationSetting.Email == nil {
|
||||
instanceNotificationSetting.Email = &storepb.InstanceNotificationSetting_EmailSetting{}
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_NOTIFICATION.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_NOTIFICATION,
|
||||
Value: &storepb.InstanceSetting_NotificationSetting{NotificationSetting: instanceNotificationSetting},
|
||||
})
|
||||
@@ -231,7 +296,7 @@ func (s *Store) GetInstanceAISetting(ctx context.Context) (*storepb.InstanceAISe
|
||||
if instanceSetting != nil {
|
||||
instanceAISetting = instanceSetting.GetAiSetting()
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_AI.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_AI,
|
||||
Value: &storepb.InstanceSetting_AiSetting{AiSetting: instanceAISetting},
|
||||
})
|
||||
@@ -265,7 +330,7 @@ func (s *Store) GetInstanceStorageSetting(ctx context.Context) (*storepb.Instanc
|
||||
if instanceStorageSetting.FilepathTemplate == "" {
|
||||
instanceStorageSetting.FilepathTemplate = defaultInstanceFilepathTemplate
|
||||
}
|
||||
s.instanceSettingCache.Set(ctx, storepb.InstanceSettingKey_STORAGE.String(), &storepb.InstanceSetting{
|
||||
s.cacheInstanceSetting(ctx, &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_STORAGE,
|
||||
Value: &storepb.InstanceSetting_StorageSetting{StorageSetting: instanceStorageSetting},
|
||||
})
|
||||
|
||||
+1
-5
@@ -28,7 +28,7 @@ import (
|
||||
// 2. checkMinimumUpgradeVersion: Verify installation can be upgraded (reject pre-0.22 installations)
|
||||
// 3. Migrate (prod mode): Apply incremental migrations from current to target version
|
||||
// 4. Migrate (demo mode): Seed database with demo data
|
||||
// 5. Reconcile identity providers from /etc/secrets/memos-idp-*.json
|
||||
// Deployment configuration is loaded separately after migration completes.
|
||||
//
|
||||
// Version Tracking:
|
||||
// - New installations: Schema version set in system_setting immediately
|
||||
@@ -132,10 +132,6 @@ func (s *Store) Migrate(ctx context.Context) error {
|
||||
return errors.Wrap(err, "failed to seed")
|
||||
}
|
||||
}
|
||||
if err := s.ApplyIdentityProviderBootstrapDir(ctx, defaultIdentityProviderBootstrapDir); err != nil {
|
||||
return errors.Wrap(err, "failed to apply identity provider bootstrap")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -59,5 +59,4 @@ INSERT INTO reaction (id,creator_id,content_id,reaction_type) VALUES(12,2,'memos
|
||||
INSERT INTO user_setting (user_id,key,value) VALUES(1,'PERSONAL_ACCESS_TOKENS','{"tokens":[{"tokenId":"demo-access-token","tokenHash":"7631cdaa5b56a39371dab01d5d186fd73f05602cc8ad29bf72ffef3713badd9d","description":"Demo access token","createdAt":"2024-01-01T00:00:00Z"}]}');
|
||||
|
||||
-- System Settings
|
||||
INSERT INTO system_setting VALUES ('GENERAL', '{"disallowPasswordAuth":true}', 'Require identity provider sign-in for the public demo.');
|
||||
INSERT INTO system_setting VALUES ('MEMO_RELATED', '{"contentLengthLimit":8192,"enableAutoCompact":true,"enableComment":true,"enableLocation":true,"defaultVisibility":"PUBLIC","reactions":["👍","💛","🔥","👏","😂","👌","🚀","👀","🤔","🤡","❓","+1","🎉","💡","✅"]}', '');
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/usememos/memos/internal/profile"
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
"github.com/usememos/memos/store/cache"
|
||||
)
|
||||
|
||||
@@ -14,6 +15,10 @@ type Store struct {
|
||||
driver Driver
|
||||
|
||||
userCreateMu sync.Mutex
|
||||
authConfigMu sync.Mutex
|
||||
|
||||
deploymentConfigMu sync.RWMutex
|
||||
deploymentConfig *deploymentConfiguration
|
||||
|
||||
// Cache settings
|
||||
cacheConfig cache.Config
|
||||
@@ -24,6 +29,11 @@ type Store struct {
|
||||
userSettingCache *cache.Cache // cache for user settings
|
||||
}
|
||||
|
||||
type deploymentConfiguration struct {
|
||||
identityProviders map[string]*storepb.IdentityProvider
|
||||
instanceSettings map[storepb.InstanceSettingKey]*storepb.InstanceSetting
|
||||
}
|
||||
|
||||
// New creates a new instance of Store.
|
||||
func New(driver Driver, profile *profile.Profile) *Store {
|
||||
// Default cache settings
|
||||
@@ -41,6 +51,10 @@ func New(driver Driver, profile *profile.Profile) *Store {
|
||||
instanceSettingCache: cache.New(cacheConfig),
|
||||
userCache: cache.New(cacheConfig),
|
||||
userSettingCache: cache.New(cacheConfig),
|
||||
deploymentConfig: &deploymentConfiguration{
|
||||
identityProviders: map[string]*storepb.IdentityProvider{},
|
||||
instanceSettings: map[storepb.InstanceSettingKey]*storepb.InstanceSetting{},
|
||||
},
|
||||
}
|
||||
|
||||
return store
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
storepb "github.com/usememos/memos/proto/gen/store"
|
||||
"github.com/usememos/memos/store"
|
||||
)
|
||||
|
||||
func TestAuthenticationConfigurationMutation(t *testing.T) {
|
||||
t.Parallel()
|
||||
ctx := context.Background()
|
||||
ts := NewTestingStore(ctx, t)
|
||||
t.Cleanup(func() { require.NoError(t, ts.Close()) })
|
||||
|
||||
general := &storepb.InstanceSetting{
|
||||
Key: storepb.InstanceSettingKey_GENERAL,
|
||||
Value: &storepb.InstanceSetting_GeneralSetting{GeneralSetting: &storepb.InstanceGeneralSetting{
|
||||
DisallowPasswordAuth: true,
|
||||
}},
|
||||
}
|
||||
_, err := ts.UpsertInstanceGeneralSettingSafely(ctx, general)
|
||||
require.ErrorIs(t, err, store.ErrUnsafeAuthenticationConfiguration)
|
||||
|
||||
provider, err := ts.CreateIdentityProvider(ctx, createTestOAuth2IDP("Primary", "primary-sso"))
|
||||
require.NoError(t, err)
|
||||
_, err = ts.UpsertInstanceGeneralSettingSafely(ctx, general)
|
||||
require.NoError(t, err)
|
||||
|
||||
err = ts.DeleteIdentityProviderSafely(ctx, &store.DeleteIdentityProvider{ID: provider.Id})
|
||||
require.ErrorIs(t, err, store.ErrUnsafeAuthenticationConfiguration)
|
||||
}
|
||||
Reference in New Issue
Block a user