FEN — Operations & Debugging (§7)

Part of the FEN design set. Overview, index, and the legacy-terminology map: fen.md.

7. Operations & Debugging

Governing constraint: FEN's E2EE means the support operator — like the storage-backend operator — is untrusted. Logs, crash reports, and remote config responses must be safe to read by an adversary. They must never allow reconstruction of group names, expense amounts, member identities, or key material.

---

7.1 Support Ticket Flow

Entry point: Settings → Help → Report a problem.

Flow:

1. User picks an issue type: Sync / Storage backend / Import or restore / Crash / Other
2. User writes a free-text description
3. App shows an expandable "What will be sent" disclosure — first and last 20 log lines visible in-app
4. Two buttons: Send (with logs) · Send without logs
5. App opens the OS mail composer pre-addressed to [email protected] with the log bundle attached as a JSON file

No account is required. No third-party SDK is introduced. The user can inspect the file before sending.

Support bundle contents

Field	What's included
App metadata	Version, build hash, platform, OS version, device class (phone/tablet — not model or serial)
Install age	Bucket only: new / 1–7d / 7–30d / 30d+ — not the install timestamp
Network state	Offline / Wi-Fi / cellular / VPN at time of report
Backend info	Backend alias (backend_0) — not the full URL
Sync health	Last sync time bucket, pending outbox count, failed event count by error code, DB schema version
Remote config	Config version/hash currently active
Log lines	Last N lines from the rolling log (user chooses 1 h / 24 h / 3 days; default 24 h)

Log format — structured JSONL

{"ts":"2026-06-28T10:42:31Z","level":"WARN","component":"backend_sync","code":"BK_CONN_TIMEOUT","backend_index":0,"attempt":3,"backoff_ms":8000,"network":"wifi"}

Safe to log: timestamps · log levels · component names · machine-readable error codes · HTTP status codes (not full URLs) · sync state machine transitions (IDLE → SYNCING → PARTIAL_FAILURE) · local DB row counts (not values) · retry counts and backoff durations · active feature flag names.

Never log:

• Plaintext event content: group names, expense names, amounts, notes, currency codes, split ratios
• Public or private keys, group keys, seed phrases, auth tokens
• Full backend URLs, blob hashes, full pubkeys, event IDs — all are stable correlation handles
• Member names, device identifiers, IP addresses, clipboard contents

Redaction pass (runs before attachment)

Pattern	Replacement
Hex string ≥ 32 chars	[HEX_REDACTED]
Base64 ≥ 44 chars outside known safe fields	[B64_REDACTED]
URL query string	stripped
Email address or IP address	[REDACTED]

If cross-line pubkey correlation is needed within one ticket, use HMAC(bundle_random_salt, pubkey)[:8] — correlatable within that report, not across reports.

Log bounds

• Ring buffer: 5,000 lines or 2 MB in-memory
• On-disk: 3-day rotation, 2 MB total cap; files older than 72 h deleted on app launch
• Debug logging: off by default; stored as an expiry timestamp (not a boolean) so it auto-expires 24 h after being enabled
• Bundle cap: 5 MB; verbose lines dropped first, then info, then oldest warnings

CI gate: test that fails if any forbidden field name (amount, name, memo, privkey, seed, plaintext) appears in log call sites.

---

7.2 Remote Config (fen-config.json)

The .well-known/fen-config.json file allows app update notices and feature rollouts without an app-store release. It is a static, signed artifact the app developer publishes (e.g. on a CDN) — not a backend or control-plane service. FEN operates no developer backend, so it carries no storage/provisioner endpoints; each group's backend is the owner's own, configured in-app (see settings-ui.md) and never distributed through remote config.

Schema

{
  "version": 12,
  "issued_at": "2026-06-28T00:00:00Z",
  "expires_at": "2026-07-28T00:00:00Z",
  "signature": "<Ed25519 over canonical JSON, this field excluded from signing>",

  "min_version":          { "ios": "1.4.0", "android": "1.4.0" },
  "min_version_deadline": "2026-07-15T00:00:00Z",
  "recommended_version":  { "ios": "1.6.2", "android": "1.6.2" },

  "limits": {
    "max_group_members":    20,
    "max_blob_bytes":  5242880,
    "max_events_per_sync": 500
  },

  "flags": {
    "receipt_ocr":        { "enabled": false, "min_app_version": "1.5.0" },
    "new_sync_scheduler": { "enabled": true,  "rollout_percent": 10 }
  },

  "support_email": "[email protected]",
  "status_url":    "https://status.fenapp.net",

  "banner": {
    "id":      "maint-2026-07-01",
    "text":    "Backend maintenance July 1, 02:00-04:00 UTC.",
    "severity": "info",
    "expires": "2026-07-01T04:00:00Z"
  },

  "next_fetch_after": "PT12H"
}

Behavior on change

Change	When applied	User-visible?
deprecated entry removed	Next cold launch	Silent
Feature flag flipped	Next cold launch	Only if the feature has UI
limits tightened	Next operation that hits the limit	Limits apply to new operations only — never retroactive
min_version tightened	Immediate banner on foreground	Non-dismissable; hard block after min_version_deadline
banner added	Next foreground event	Shown once; app stores banner.id and won't re-show

Fetch mechanics

• Fetch on cold launch if last fetch was > next_fetch_after ago
• Background refresh (WorkManager / BackgroundTasks) throttled by next_fetch_after
• Cache last-known-good; app must function fully offline using it
• Fetch failure → use cache; log WARN remote_config_fetch_failed
• Fetched version < cached version → reject silently; log WARN remote_config_downgrade_rejected
• Cache > 30 days old → log WARN remote_config_stale

Signature verification

• Config signing key is a dedicated Ed25519 keypair, separate from any backend or member identity
• The public key is hardcoded in the app binary
• Signed payload: canonical JSON (keys sorted, no insignificant whitespace, UTF-8); signature field excluded
• On verification failure: reject the entire config, use cache, log ERROR remote_config_sig_invalid

What remote config must never do

• Change cryptographic algorithms or key derivation paths
• Disable E2EE verification
• Add new trust anchors
• Ship or evaluate executable code
• Set, override, or distribute any storage backend — the owner's backend is configured only in-app and never travels through remote config

Risks

Risk	Mitigation
Config endpoint compromised	Signature + hardcoded key; attacker also needs the signing key
Signing key leaked	Issue app update with new hardcoded public key; overlap window accepting both; treat like PKI root
Bad config strands users	Last-known-good cache always wins on failure; min_version_deadline buffer
Rollout bug breaks sync globally	rollout_percent staged rollout; per-feature kill-switch flag
Limits applied retroactively	Policy: limits only apply to new operations

---

7.3 Crash Reporting

Use a self-hosted crash reporting instance (e.g. Sentry self-hosted) — not a SaaS provider — to keep crash data off third-party infrastructure.

Safe to capture: stack traces (file + line, no argument values), app version, OS version, device class, memory / CPU pressure at crash time, crash type.

Never capture: UI state derived from group or expense content, screen routes if derived from group IDs, raw database rows, decrypted event payloads, any key material.

Configure the crash SDK with explicit field allowlisting before any event leaves the device. Disable automatic breadcrumb capture; add only manually-whitelisted breadcrumbs.

---

7.4 Sync Health Panel

Accessible from Settings → Sync Status.

Shows:

• Per-group backend connection state (connected / reconnecting / failed) — backend alias, not URL (each group is bound to exactly one backend, §7.7)
• Last successful sync: time bucket (justnow / < 5 min / < 1 h / > 1 h)
• Outbox: pending count, failed count by error code
• Attachments: pending upload count, last upload time bucket
• DB schema version, remote config version

"Run diagnostics" button — 7-step backend health check:

1. DNS resolution
2. TLS handshake
3. Storage bucket access
4. Credential check (S3 SigV4 signed request accepted)
5. Write synthetic test event (not real group data)
6. Read back test event
7. Round-trip latency bucket (< 200 ms / < 1 s / > 1 s / failed)

Results shown as a checklist. Errors shown as code + description — never raw strings that might contain URLs or user data.

Attachment diagnostics: upload then download a synthetic encrypted test blob to/from the group bucket's attachments/ prefix. Log HTTP status class and latency bucket only — not real blob hashes.

---

7.5 Backend Operator Guide

Note. FEN operates no developer backend. The backend operator is always the group owner, self-hosting one of two BYO backends (S3-compatible or Nextcloud). This section is guidance for that owner. Full backend operations guidance is in backend-architecture.md.

What the backend cannot see: group names, expense amounts, member identities, receipt images — all content is encrypted before transmission.

What the backend can see: IP addresses (mitigatable by VPN/Tor), event sizes and timing, pubkeys as stable pseudonyms, connection frequency.

API contract: the S3 and Nextcloud data planes, and the in-app control plane (GroupProvisioner), are specified in backend-architecture.md.

Recommended practices (for the self-hosting owner):

• No long-term IP address logs
• TLS with a valid certificate in production
• Keep the backend software patched (a currently-maintained Nextcloud major; a current Garage/MinIO release)
• Regular backups of the store with a tested restore procedure (§7.7)
• Because every group member shares one storage credential, treat that credential as a group secret and rotate it (best-effort) on member removal (§4.8, backend-architecture.md §6)

---

7.6 Security Incident Response

FEN operates no backend, so developer-side incident response covers only the app and the remote-config artifact. Backend incidents (a compromised or leaky store) are the self-hosting owner's to handle for their own group.

Scenario	Response
Config signing key compromised	App update with new hardcoded public key; overlap window accepting both keys; revoke old key after adoption threshold reached
Owner's backend compromised (owner-handled)	The owner rotates the group's shared storage credential and re-hands it off to remaining members (§4.8, backend-architecture.md §6); forward secrecy for new events already holds via key-epoch rotation
Critical app bug (data loss risk)	Set min_version + min_version_deadline in remote config; publish fix immediately; hard block after deadline
Backend metadata leak (IP logs exposed)	Owner-handled: post/adjust their own retention policy; members with high privacy requirements use VPN/Tor

Draft user-facing notification templates for each scenario in a separate runbook. Templates must describe what happened and what users should do — not technical details that could aid further exploitation.

---

7.7 Backend loss & restore

A group is bound to exactly one backend — the owner's self-hosted S3-compatible store or Nextcloud instance (security.md §4.8, invariant E-4). There is deliberately no "publish to ≥2 backends" fan-out; that mitigation was removed because a single bound backend made it unrealisable. Durability instead rests on the member devices: every fully-synced device holds a complete copy of the signed event log (data-model.md §3.9).

Recovery Point Objective (RPO). RPO is governed entirely by peer-sync frequency, not by the backend:

Situation	RPO
Event already pulled by ≥1 other member device (or still on the backend)	≈ 0 — the union of device replicas heals it
Event living on only its author's device (authored offline, or the backend lost before any peer polled)	Bounded by the sync interval — the gap until the next successful peer pull; unbounded if that peer never opens the app

Sync is opportunistic (on open/resume/timer, appendices.md), so operators cannot quote a fixed RPO number; it is "as fresh as the last time two members' devices exchanged." The single-replica event is the only irreducible loss surface and is contained by the §3.12 settlement gate (money never moves on a log that cannot be proven gap-free).

Recovery Time Objective (RTO) — the tested restore procedure. If the owner's backend is destroyed, the group is recovered by standing up a blank backend and having any one fully-synced member push its local replica to it:

1. Stand up a fresh, empty backend (a new S3-compatible store or Nextcloud share) and derive a new StorageDescriptor for the group (same layout; the group key is unchanged, so no re-encryption is needed).
2. On a fully-synced member device, run GroupRestoreService.restore(groupId, backend). It reconstructs each member's groups/<groupId>/logs/<pubkey>.jsonl byte-for-byte from the originally-signed wire lines that device retained on ingest (group_events.raw_line), grouped by author and ordered by author_seq. It cannot re-sign another member's events — the exact ciphertext + nonce + signature are replayed, never regenerated — so signature verification on every other device still passes after restore.
3. Re-bind the group to the new descriptor and re-distribute it to members via the normal credential channel (re-invite / StorageMigrated, §3.3). Members then pull the restored log and converge on the identical state.

The restore is idempotent (re-running writes the same per-author files) and completeness-checked: if the restoring device is missing the retained wire bytes for any event it knows about, the restore refuses rather than publishing a partial log. This procedure is proven end-to-end by group_restore_service_test.dart (a peer restores to a blank backend; a third, blank device binds it, pulls, and reconstructs byte-identical ledger state).