Models and backends

Scenario

You have gigastt running with the default rnnt head on the CPU backend, and now you need to make an informed change: a different recognition head (punctuation baked in, or languages beyond Russian), a leaner model download, a faster execution provider for your hardware, or a bigger session pool. This chapter answers four questions with checkable recipes: which head, INT8 install, which backend, and how much RAM the pool needs.

WER and RTF numbers are not duplicated here — the canonical, CI-annotated tables live in docs/benchmarks.md; this chapter links into them. Flags are checked against gigastt <command> --help; the full flag reference lives in docs/cli.md.

Prerequisites

• gigastt installed (binary, package, or image) — see Getting started.
• Disk: ~250–400 MB free for the lean INT8 install (the only runtime path; optional punct/VAD side models extra).
• For building a non-default backend from source: Rust 1.94+ and protoc on PATH (build requirements: docs/architecture.md).

Recipe

Choosing a recognition head

The recognition head is selected with --model-variant (env GIGASTT_MODEL_VARIANT) on serve / download / transcribe. All heads share the mel frontend and the 16 kHz mono input contract; they differ in ONNX files, vocabulary, and decoding.

Two hard constraints the table implies:

• rnnt / e2e_rnnt have a Cyrillic-only vocabulary — they cannot transcribe English at all. For English (or kk/ky/uz) the Multilingual heads are the only option.
• The Multilingual heads are encoder-only CTC: they always ship and run as pre-quantized INT8 — there is no FP32 download and no on-device quantization step for them.

Download and run a different head:

gigastt download --model-variant e2e_rnnt
gigastt serve --model-variant e2e_rnnt

Auto-detection rules (from crates/gigastt-core/src/model/variant.rs):

• No --model-variant → the engine detects the installed head from the files in the model directory and uses a complete install as-is (no network request at all).
• An explicit --model-variant that differs from the installed head → the requested set is downloaded alongside; variants are never mixed within one inference, and a warning is logged. When several heads’ files coexist and no variant is requested, auto-detect prefers rnnt.
• An empty model directory + no flag → the default rnnt is downloaded.

Verify:

curl -s http://127.0.0.1:9876/health
# {"status":"ok","model":"gigaam-v3-e2e-rnnt","variant":"e2e_rnnt",...}
curl -s http://127.0.0.1:9876/v1/models
# .id / .name reflect the loaded head; .encoder is int8 on the ORT path
# (candle reports fp32 — FP32 safetensors, is_int8()==false)

Readable text: punctuation, ITN, and hotwords

The default rnnt head emits bare lowercase. Readable Russian needs a post-pass (or the e2e_rnnt head, which bakes punctuation/casing/ITN in).

# Explicit readable pipeline on the default head
gigastt serve --punctuation on --itn on
curl -s http://127.0.0.1:9876/health
# "punctuation":true,"itn":true

# Per-request override (409 punctuation_not_available if model missing and forced on)
curl -s -X POST "http://127.0.0.1:9876/v1/transcribe?punctuation=true&itn=true" \
  -H "Content-Type: application/octet-stream" --data-binary @clip.wav

Hotword bias lifts brand names and domain terms that the model otherwise misses. One phrase per line; optional built-in Russian brand/acronym lexicon:

cat > /tmp/hotwords.txt <<'EOF'
гигачат
сбер
еком
EOF

gigastt serve --hotwords-file /tmp/hotwords.txt --hotwords-default \
  --hotwords-boost 5.0
# Same flags work on `transcribe` / `transcribe-batch` / `watch` offline.

Verify: transcribe a clip that says a listed brand; without the file the hypothesis often mangled the token, with hotwords the spelling matches the list. Boost too high can invent brands from noise — start at the default 5.0 and only raise if needed. Flags: docs/cli.md.

INT8 install (runtime)

Short answer: runtime is INT8 only. The shipped encoder runs as true integer compute (DynamicQuantizeLinear + MatMulInteger/ConvInteger kernels), ~215 MB on disk, ~0% WER degradation vs FP32 source weights — numbers and methodology: docs/benchmarks.md.

What happens on each path:

• Default (RNN-T) — gigastt download (or the first serve / transcribe) fetches the pre-quantized INT8 bundle (INT8 encoder + decoder + joiner + vocab) from the pinned GitHub Release. Runtime never loads FP32.
• Multilingual — ml_ctc / ml_ctc_large download istupakov’s pre-quantized INT8 encoder directly from HuggingFace.
• Packaging — gigastt quantize [--force] needs a local FP32 encoder ONNX as source (e.g. after a custom fine-tune). Not a runtime path.

Verify:

ls ~/.gigastt/models/
# v3_rnnt_encoder_int8.onnx present alongside decoder/joint/vocab
gigastt transcribe sample.wav 2>&1 | grep 'transcribe complete'
# ... encoder=int8/cpu ... rtf=0.1xx

The encoder=int8/<backend> field in the completion log line is the ground truth for which encoder file was loaded.

Execution provider for your hardware

The backend is a compile-time Cargo feature, not a runtime flag — the provider is baked into the binary you install or build:

Build the one that matches:

cargo build --release                      # CPU, any platform
cargo build --release --features coreml    # macOS ARM64
cargo build --release --features cuda      # Linux x86_64 + NVIDIA (CUDA 12+)
cargo build --release --features nnapi     # Android / ARM64

Exclusivity is enforced at compile time: coreml and cuda are mutually exclusive; ane conflicts with coreml/cuda/nnapi/candle; candle conflicts with coreml/cuda. A compile_error! fires on a bad combination.

Runtime behavior worth knowing:

• CPU fallback is deliberate, never a crash. The CoreML build runs a ~1 s warmup probe at startup; on failure it logs falling back to CPU execution provider and rebuilds the sessions on CPU. The CUDA binary likewise runs on CPU when no GPU is visible (e.g. a container started without --gpus all).
• CUDA packaging. There is no prebuilt CUDA tarball — the release matrix ships CPU binaries for Linux (x86_64, aarch64), Windows, and a CoreML-enabled macOS ARM64 binary. For GPU use the published ghcr.io/ekhodzitsky/gigastt:<ver>-cuda image (recipes: Deployment & ops) or build with Dockerfile.cuda.
• Native ANE backend (--features ane, macOS ARM64): runs the rnnt encoder on the Neural Engine via per-bucket fixed-shape Core ML packages, ~10× warm end-to-end over the CPU build (decode-bound). Fetch the packages with gigastt download --ane (into ~/.gigastt/models/ane/); an e2e_rnnt model transparently falls back to the ort encoder, and streaming windows always take the CPU path — ANE is a file-mode accelerator. Full design and honest numbers: docs/ane-backend.md.
• Candle backend (--features candle, experimental): pure-Rust inference on the Metal GPU, byte-for-byte parity with ort, rnnt head only, FP32 weights converted once with scripts/convert_gigaam_candle.py. Details: docs/candle-backend.md.

Verify:

gigastt transcribe sample.wav 2>&1 | grep 'transcribe complete'
# encoder=int8/coreml  → the CoreML build is active (int8/cuda, int8/cpu, ...)
# encoder=int8/ane     → the native ANE backend handled the encoder

On a CoreML build, also watch startup: the absence of falling back to CPU execution provider means the warmup probe passed.

Model files in an offline perimeter

Everything the engine needs lives in one directory — ~/.gigastt/models/ by default, overridable with --model-dir on serve / download / transcribe / quantize. That makes the model set a plain file artifact you can stage and move around:

# On a connected machine:
gigastt download --model-dir /srv/gigastt-models

# Copy to the target host any way you like (rsync, USB, artifact store):
rsync -a /srv/gigastt-models/ offline-host:/srv/gigastt-models/

# On the offline host — refuse every network fetch, fail fast instead of hanging:
gigastt --offline serve --model-dir /srv/gigastt-models

Properties that make this safe:

• Integrity is verified by the binary, not by you. Every download is SHA-256-checked against checksums pinned in the code, staged as .partial, and atomically renamed into place; a corrupt file is removed, never promoted. A checksum failure exits with code 65 (network 69, disk 74, Ctrl-C 130) — scriptable.
• A complete INT8 model set means zero network. With a full head’s INT8 files present (INT8 encoder + decoder/joiner/vocab for the RNN-T heads, or INT8 encoder + vocab for Multilingual), startup performs no network request, even without --offline. FP32-only trees are not usable. --offline / GIGASTT_OFFLINE=1 turns any missing optional model (punctuation, VAD, diarization) into an immediate error naming the file to provide.
• The head is auto-detected from the files — a copied directory needs no --model-variant flag on the target host.
• When copying, include the *_int8.onnx (or CTC *.int8.onnx) encoder, the vocab, and — for rnnt/e2e_rnnt — the decoder and joiner. gigastt quantize is packaging-only (needs a local FP32 source); do not rely on it at runtime.

For a fully packaged offline install (tarball with binary + INT8 model + punctuation model + systemd unit, or the two-deb variant), use the release offline bundle — recipe and verification steps in Deployment & ops; inventory in README-OFFLINE.md.

Verify:

gigastt --offline transcribe sample.wav --model-dir /srv/gigastt-models
# transcribes with no network; a missing file errors immediately, naming the file

Sizing the session pool (RAM)

The INT8 encoder is memory-mapped and shared. Extra slots add decoder/joiner state and ORT arenas — not another encoder copy. Process RAM: ~46 / ~66 MB resident at pool 1 / 2 (~277 / ~510 MB ps RSS); extra slot ~20 MB. --pool-size 1 is an RTF / edge choice, not a RAM necessity. Load still budgets 2 × encoder-file-size per slot. Figures: docs/benchmarks.md.

Verify:

# no "Capping pool size" warning at startup, and:
curl -s http://127.0.0.1:9876/ready
# {"status":"ready","pool_available":2,"pool_total":2}

Verifying the result

End-to-end smoke after any change in this chapter:

ls ~/.gigastt/models/                  # the head's full file set, incl. *_int8.onnx + vocab
gigastt transcribe sample.wav 2>&1 | grep 'transcribe complete'
# encoder=<int8|fp32>/<cpu|coreml|cuda|ane|candle> — fp32 only on candle, rtf well under 1.0
curl -s http://127.0.0.1:9876/health   # "model"/"variant" match the head you picked
curl -s http://127.0.0.1:9876/ready    # ready, pool_available >= 1

Then confirm the accuracy expectation against docs/benchmarks.md rather than re-measuring casually — the harness, manifests, and normalization matter more than the stopwatch.

Common pitfalls

• English audio comes out as garbage with the default head. Not a bug: rnnt / e2e_rnnt have a Cyrillic-only vocabulary and cannot emit Latin text at all. Switch to --model-variant ml_ctc (or ml_ctc_large).
• --model-variant seems to be ignored after the first download. The engine auto-detects the head from the model directory; an explicit variant that differs from the installed one triggers a second download alongside (with a variants are never mixed warning), and a later flag-less start prefers rnnt again. Delete the unused head’s files to keep the directory unambiguous and reclaim disk.
• First serve looks hung. That is the one-time lean INT8 download (~225 MB) if the model dir is empty; /health answers 200 with model:"loading" while /ready stays 503 initializing. Pre-seed with gigastt download, and gate clients on /ready, never on /health.
• OOM after switching to ml_ctc_large. Resident RAM after mmap is unmeasured (encoder file ~592 MB; do not invent 2 × file size). Prefer --pool-size 1 until measured. The load-time cap still budgets 2 × encoder-file-size, so the pool may clamp even when resident would fit. Keep --pool-min-size 1 so a tight host boots degraded.
• The CoreML build is no faster than CPU. Look for falling back to CPU execution provider in the startup log — the warmup probe failed and the engine is (deliberately) running on CPU. The completion log’s encoder=int8/cpu field confirms it.
• The CUDA container runs at CPU speed. The GPU is not visible: the container needs the NVIDIA Container Toolkit and --gpus all. The binary falls back to CPU silently — check encoder=int8/cuda in the completion log.
• error: ane and coreml are mutually exclusive (or similar) at build time. Backend features conflict by design; build exactly one of coreml / cuda / ane / candle (nnapi is the exception).
• SHA-256 mismatch during gigastt download. The staged download was corrupt or tampered with; it is removed, not promoted, and the CLI exits 65. Re-run the command — do not hand-rename a .partial file into place.

Links

• Getting started — install and first transcription
• CLI and batch processing — throughput/memory recipe for the offline CLI
• Deployment & ops — offline bundle, systemd, OOM runbook entries
• docs/benchmarks.md — canonical WER / RTF / footprint tables
• docs/architecture.md — pipeline, providers, quantization internals
• docs/cli.md — full flag reference
• docs/ane-backend.md — native Apple Neural Engine backend
• docs/candle-backend.md — Candle/Metal backend
• docs/verifying-releases.md — release artifact verification
• packaging/offline/README-OFFLINE.md — offline bundle inventory