FEST: in-repo flattener bundles slim.json; release builds auto-refresh when stale

Amends decision #8: the producer now lives in tools/fest_flatten.py
(stdlib Python, streaming iterparse over the 115 MB M30 XML) instead of
an out-of-repo server job. It pulls fest251.zip from dmp.no, flattens
8934 active human-use brand entries (name/strength/unit/form/ATC +
modal package size from pakningsinfo), and writes the asset checked in
at app/src/main/assets/fest/slim.json — so autocomplete works offline
out of the box. preReleaseBuild depends on refreshFestData: fast-exits
under 30 days, keeps the old file on download failure (offline release
builds never break). FestRepository: downloaded cache wins over the
bundled asset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -38,8 +38,9 @@ The name says the brief: nag me to take the dose *now*.
    never lock me into this app. JCE AES-GCM is the baseline behind an interface; verify
    age against the age test vectors before trusting it.
 8. **FEST, not Felleskatalogen**, and **not** as a live API: its open Rekvirent extract
-   is a SOAP/WCF M30 XML dump. The phone reads a slim pre-flattened JSON synced from my
+   is a SOAP/WCF M30 XML dump. `tools/fest_flatten.py` (in-repo, amended 2026-06) flattens
-   server (that job lives outside this repo) and does autocomplete offline.
+   it into a slim JSON bundled as an APK asset; release builds auto-refresh it when >30
+   days old. Autocomplete is offline; a downloaded file via Settings can override the bundle.
 9. Refill is **derived** from inventory + consumption. Prescription renewal
    (`rxExpiryEpochDay`, `refillsRemaining`) is tracked **separately** from stock.
 

app/build.gradle.kts

@@ -71,6 +71,24 @@ android {
     sourceSets.getByName("androidTest").assets.srcDir("$projectDir/schemas")
 }
 
+// Release builds refresh the bundled FEST dataset when it's >30 days old.
+// The script fast-exits when fresh and keeps the old file on download failure,
+// so offline/CI release builds never break on this.
+val refreshFestData by tasks.registering(Exec::class) {
+    group = "fest"
+    description = "Regenerate app/src/main/assets/fest/slim.json from DMP's M30 extract if stale"
+    workingDir = rootProject.projectDir
+    commandLine(
+        "python3", "tools/fest_flatten.py",
+        "--out", "app/src/main/assets/fest/slim.json",
+        "--max-age-days", "30",
+    )
+}
+
+tasks.matching { it.name == "preReleaseBuild" }.configureEach {
+    dependsOn(refreshFestData)
+}
+
 dependencies {
     implementation(libs.kotlinx.coroutines.android)
     implementation(libs.androidx.core.ktx)

app/src/main/java/no/naiv/meddetsamme/fest/FestRepository.kt

@@ -39,18 +39,34 @@ class FestRepository(
     private val json = Json { ignoreUnknownKeys = true }
     private val cacheFile get() = File(context.filesDir, "fest-slim.json")
 
+    private companion object {
+        const val BUNDLED_ASSET = "fest/slim.json"
+    }
+
     @Volatile private var loaded: FestDataset? = null
 
-    /** Cached dataset, loading from disk on first use. Null until downloaded once. */
+    /**
+     * Cached dataset: a downloaded refresh wins over the asset bundled at build
+     * time, but the bundle means autocomplete works out of the box, offline,
+     * with zero configuration.
+     */
     fun dataset(): FestDataset? {
         loaded?.let { return it }
         synchronized(this) {
             loaded?.let { return it }
-            if (!cacheFile.exists()) return null
+            if (cacheFile.exists()) {
+                try {
+                    return json.decodeFromString<FestDataset>(cacheFile.readText()).also { loaded = it }
+                } catch (e: Exception) {
+                    // corrupt cache → fall through to the bundled asset
+                }
+            }
             return try {
-                json.decodeFromString<FestDataset>(cacheFile.readText()).also { loaded = it }
+                context.assets.open(BUNDLED_ASSET).bufferedReader().use { reader ->
+                    json.decodeFromString<FestDataset>(reader.readText()).also { loaded = it }
+                }
             } catch (e: Exception) {
-                null // corrupt cache → behave as not-downloaded; refresh overwrites
+                null
             }
         }
     }

docs/fest-dataset.md

@@ -1,10 +1,17 @@
 # FEST slim dataset — contract
 
 The app does **offline** autocomplete against a slim, pre-flattened JSON file.
-The job that produces it lives **outside this repo**: it pulls FEST's open
+The producer is `tools/fest_flatten.py` in this repo: it downloads FEST's open
-Rekvirent extract (the M30 SOAP/WCF XML dump from DMP), flattens the
+Rekvirent extract (the M30 XML dump, `fest251.zip` from dmp.no), flattens the
-human-use medication entries, and publishes the JSON behind Caddy or into the
+human-use brand entries, and writes `app/src/main/assets/fest/slim.json`,
-Garage bucket. FEST changes slowly; refresh the published file monthly-ish.
+which is checked in and **bundled in the APK** — autocomplete works out of the
+box with zero configuration.
+
+Release builds run the script automatically (`preReleaseBuild` →
+`refreshFestData`): it regenerates when the bundled file is >30 days old,
+fast-exits when fresh, and keeps the old file if the download fails, so an
+offline release build never breaks. Manual run:
+`python3 tools/fest_flatten.py --out app/src/main/assets/fest/slim.json --force`.
 
 Why FEST and not Felleskatalogen: FEST is the open national dataset;
 Felleskatalogen is licensed editorial content with no open API. Why not live
@@ -13,11 +20,10 @@ work offline.
 
 ## Where the app loads it from
 
-`Innstillinger → FEST-URL` (e.g. `https://<host>/fest/slim.json`). The app
+Bundled asset `fest/slim.json` by default. Optionally, `Innstillinger →
-downloads on demand, caches to `filesDir/fest-slim.json`, and reads only the
+FEST-URL` downloads a newer file to `filesDir/fest-slim.json`, which then
-cache afterwards. A staleness nudge appears after ~45 days (dataset `updated`
+wins over the bundle. Old data keeps working — autocomplete is a convenience,
-field), but old data keeps working — autocomplete is a convenience, never a
+never a gate.
-gate.
 
 ## JSON shape (version 1)
 

tools/fest_flatten.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""Flatten DMP's FEST M30 extract into the app's slim.json (docs/fest-dataset.md).
+
+Downloads fest251.zip (the human-use Rekvirent extract) from DMP, streams the
+~115 MB XML with iterparse, and writes the slim autocomplete dataset:
+
+    {"version": 1, "updated": "YYYY-MM-DD", "entries": [
+        {"name", "strength", "unit", "form", "atc", "packageSize"}, ...]}
+
+Stdlib only. Designed to be run by Gradle before release builds:
+- exits 0 immediately if the existing output is younger than --max-age-days;
+- on download failure with an existing output, warns and keeps it (an offline
+  release build must never break on a stale-but-working dataset).
+"""
+
+import argparse
+import datetime as dt
+import io
+import json
+import re
+import sys
+import tempfile
+import urllib.request
+import xml.etree.ElementTree as ET
+import zipfile
+from collections import Counter
+from pathlib import Path
+
+FEST_URL = (
+    "https://www.dmp.no/globalassets/documents/om-oss/"
+    "distribusjon-av-legemiddeldata/fest/festfiler/fest251.zip"
+)
+
+# Strength starts at the first token containing a digit, e.g.
+# "Thylip kaps 200 mg" -> "200 mg"; "Kodimagnyl ... 9,6/500 mg" works too.
+STRENGTH_RE = re.compile(r"\s(\d[\d,./]*\s?\S*.*)$")
+
+
+def local(tag: str) -> str:
+    """Strip XML namespace: '{ns}Varenavn' -> 'Varenavn'."""
+    return tag.rsplit("}", 1)[-1]
+
+
+def parse_fest(xml_file) -> dict:
+    """One streaming pass: brand entries + package sizes keyed by brand id."""
+    updated = ""
+    brands: dict[str, dict] = {}  # merkevare Id -> entry
+    pack_sizes: dict[str, Counter] = {}  # merkevare Id -> Counter of Mengde
+
+    context = ET.iterparse(xml_file, events=("end",))
+    for _, elem in context:
+        tag = local(elem.tag)
+
+        if tag == "HentetDato":
+            updated = (elem.text or "")[:10]
+
+        elif tag == "OppfLegemiddelMerkevare":
+            status = elem.find("./{*}Status")
+            mv = elem.find("./{*}LegemiddelMerkevare")
+            if mv is not None and status is not None and status.get("V") == "A":
+                brand = flatten_brand(mv)
+                if brand:
+                    brands[brand.pop("_id")] = brand
+            elem.clear()
+
+        elif tag == "OppfLegemiddelpakning":
+            status = elem.find("./{*}Status")
+            if status is not None and status.get("V") == "A":
+                for info in elem.iter():
+                    if local(info.tag) == "Pakningsinfo":
+                        ref = info.find("./{*}RefLegemiddelMerkevare")
+                        mengde = info.find("./{*}Mengde")
+                        if (
+                            ref is not None
+                            and ref.text
+                            and mengde is not None
+                            and mengde.text
+                        ):
+                            try:
+                                size = float(mengde.text.replace(",", "."))
+                            except ValueError:
+                                continue
+                            pack_sizes.setdefault(ref.text, Counter())[size] += 1
+            elem.clear()
+
+    for brand_id, sizes in pack_sizes.items():
+        if brand_id in brands and sizes:
+            # Most common package size; ties resolved towards the smaller pack.
+            best = max(sorted(sizes), key=lambda s: sizes[s])
+            brands[brand_id]["packageSize"] = best
+
+    entries = sorted(brands.values(), key=lambda e: (e["name"].lower(), e["strength"]))
+    return {"version": 1, "updated": updated, "entries": entries}
+
+
+def flatten_brand(mv) -> dict | None:
+    fields = {}
+    for child in mv:
+        fields[local(child.tag)] = child
+    name_el = fields.get("Varenavn")
+    if name_el is None or not (name_el.text or "").strip():
+        return None
+    name = name_el.text.strip()
+
+    nfs = (
+        fields.get("NavnFormStyrke") is not None and fields["NavnFormStyrke"].text
+    ) or ""
+    # Names may themselves contain digits ("2-Hydroxyethyl …") — strip the name
+    # prefix before hunting for the strength part.
+    if nfs.lower().startswith(name.lower()):
+        nfs = nfs[len(name) :]
+    m = STRENGTH_RE.search(nfs)
+    strength = m.group(1).strip() if m else ""
+
+    form_el = fields.get("LegemiddelformKort")
+    form = form_el.get("DN", "") if form_el is not None else ""
+
+    unit = ""
+    adm = fields.get("AdministreringLegemiddel")
+    if adm is not None:
+        enhet = adm.find("./{*}EnhetDosering")
+        if enhet is not None:
+            unit = enhet.get("DN", "")
+
+    atc_el = fields.get("Atc")
+    atc = atc_el.get("V") if atc_el is not None else None
+
+    inner_id = None
+    for child in mv:
+        if local(child.tag) == "Id":
+            inner_id = child.text
+            break
+    if not inner_id:
+        return None
+
+    entry = {
+        "_id": inner_id,
+        "name": name,
+        "strength": strength,
+        "unit": unit or "dose",
+        "form": form,
+    }
+    if atc:
+        entry["atc"] = atc
+    return entry
+
+
+def is_fresh(out_path: Path, max_age_days: int) -> bool:
+    if max_age_days <= 0 or not out_path.exists():
+        return False
+    try:
+        updated = json.loads(out_path.read_text())["updated"]
+        age = dt.date.today() - dt.date.fromisoformat(updated)
+        return age.days <= max_age_days
+    except (KeyError, ValueError, json.JSONDecodeError):
+        return False  # unreadable/odd file -> regenerate
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--out", required=True, type=Path)
+    ap.add_argument("--url", default=FEST_URL)
+    ap.add_argument("--max-age-days", type=int, default=30)
+    ap.add_argument("--force", action="store_true")
+    args = ap.parse_args()
+
+    if not args.force and is_fresh(args.out, args.max_age_days):
+        print(
+            f"fest_flatten: {args.out} is fresh (<= {args.max_age_days} days), skipping"
+        )
+        return 0
+
+    print(f"fest_flatten: downloading {args.url} ...", flush=True)
+    try:
+        with tempfile.TemporaryFile() as tmp:
+            with urllib.request.urlopen(args.url, timeout=300) as resp:
+                while chunk := resp.read(1 << 20):
+                    tmp.write(chunk)
+            tmp.seek(0)
+            with zipfile.ZipFile(tmp) as zf:
+                inner = zf.namelist()[0]
+                print(f"fest_flatten: parsing {inner} ...", flush=True)
+                with zf.open(inner) as xml_file:
+                    dataset = parse_fest(io.BufferedReader(xml_file))
+    except Exception as e:  # noqa: BLE001 - any failure has the same handling
+        if args.out.exists():
+            print(
+                f"fest_flatten: WARNING: refresh failed ({e}); keeping existing {args.out}"
+            )
+            return 0
+        print(f"fest_flatten: ERROR: {e} and no existing dataset", file=sys.stderr)
+        return 1
+
+    args.out.parent.mkdir(parents=True, exist_ok=True)
+    args.out.write_text(json.dumps(dataset, ensure_ascii=False, separators=(",", ":")))
+    print(
+        f"fest_flatten: wrote {len(dataset['entries'])} entries (updated {dataset['updated']}) to {args.out}"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())