Taking Oremi Izwi to the GPU: What We Learned from a Deep TTS Performance Investigation

Building a fast text-to-speech service is not simply a matter of choosing a smaller model or enabling a faster inference backend.

Over the past few performance investigations, I have been working on Oremi Izwi, the TTS service used by the Oremi personal-assistant ecosystem, with one concrete objective: determine how far the current Supertonic-based implementation can be pushed before architectural changes become necessary.

The investigation covered model architecture, ONNX Runtime, quantization, concurrency, model loading, streaming, and finally GPU deployment.

The conclusion is both simpler and more interesting than expected:

The biggest performance opportunity for Oremi Izwi is not a generic CPU optimization or an off-the-shelf INT8 conversion. It is getting the Supertonic inference pipeline onto the GPU correctly, while avoiding unnecessary CPU↔GPU transfers between its four ONNX models.

This article documents what was measured, what was ruled out, and what remains to be tested.

Oremi Izwi

Oremi Izwi — project repository and documentation

Oremi Izwi is a Python-based REST TTS service for the Oremi ecosystem. The application exposes a FastAPI API and uses the external Supertonic package as its synthesis engine.

The current architecture is intentionally small:

FastAPI / Uvicorn
        │
        ▼
    TTSEngine
        │
        ▼
    Supertonic
        │
        ├── duration predictor
        ├── text encoder
        ├── vector estimator
        └── vocoder
                │
                ▼
              PCM/WAV

The service keeps a single TTSEngine instance in the application state and executes synthesis outside the asyncio event loop using asyncio.to_thread.

The repository's engineering guidelines explicitly emphasize preserving this architecture, avoiding unnecessary abstractions, and keeping inference away from the event loop.

The service currently targets Python 3.12+ and uses FastAPI, Pydantic, SoundFile and Supertonic as its principal runtime dependencies.

Why Supertonic?

Supertonic is designed as an efficient, local TTS system built around ONNX Runtime. The current Supertonic-3 release supports 31 languages and provides public ONNX assets.

Its architecture is fundamentally different from many modern autoregressive TTS systems.

Rather than generating speech token by token, Supertonic uses a non-autoregressive diffusion-style pipeline.

For Oremi Izwi, the practical inference path is approximately:

text
  │
  ├── duration prediction
  │
  ├── text encoding
  │
  ├── diffusion / vector estimation
  │       × N steps
  │
  └── vocoder
          │
          ▼
        waveform

This distinction became extremely important when comparing Oremi with another recent TTS system: Audio8-TTS.

The Audio8-TTS Comparison

Audio8-TTS on GitHub

Audio8-TTS uses a very different architecture.

The project describes a 0.6B-parameter multilingual TTS system based on a DualAR architecture:

Slow AR
   ↓
Fast AR
   ↓
Codec
   ↓
Waveform

It uses autoregressive generation, acoustic codebooks and static KV caches. The repository describes two autoregressive branches and static KV caches for both during generation.

This difference matters because many of the optimizations that make Audio8 interesting are architecture-specific.

For example:

• KV caching
• incremental autoregressive decoding
• codec-window streaming
• token-level generation
• weight-only INT4
• CUDA-graph-oriented execution

are not generic switches that can simply be copied into Supertonic.

Supertonic does not have an autoregressive token decoder whose growing attention history needs to be cached.

Therefore:

A fast optimization in one TTS architecture is not necessarily an optimization in another.

That became one of the central lessons of this investigation.

Establishing the CPU Baseline

The first step was to measure the existing Oremi implementation rather than optimize blindly.

The main CPU environment used for the experiments was:

• Intel Core i9-13900HK
• 14 physical cores / 20 threads
• Python 3.12
• ONNX Runtime 1.28
• Supertonic 1.3.1
• Supertonic-3
• CPUExecutionProvider

The four FP32 ONNX models were approximately:

The most important observation was immediately visible:

The vector estimator is by far the largest model and the dominant inference cost.

The vector estimator contains a mixture of MatMul, Conv and other operations and is executed repeatedly during diffusion.

That makes it a natural target for performance work.

CPU Performance

With total_steps=2, the measured real-time factor was approximately:

At total_steps=8, RTF increased to approximately 0.4.

The benchmark also showed that the fixed overhead becomes less significant for longer inputs.

For example, longer synthesis amortizes the fixed cost of model invocation and other per-request operations.

The diffusion vector_estimator, however, scales with the number of diffusion steps.

This made total_steps one of the most direct performance knobs available to the application.

Model Loading Is Not the Main Inference Problem

Cold startup was approximately 0.7–1.0 seconds for TTSEngine initialization in the measured environment.

Roughly 85% of that time came from ONNX Runtime session creation.

The important point is that this work happens inside the external Supertonic loader rather than inside Oremi's own Python inference code.

This limits how much Oremi can optimize startup without changing the dependency boundary.

Voice-style loading, by comparison, was a relatively small part of initialization and was deliberately kept eager.

That provides predictable request behavior and avoids adding cache-related synchronization to the request path.

Concurrency Revealed Another CPU Bottleneck

The CPU benchmark also exposed an important operational characteristic.

With the default ONNX Runtime threading behavior, one synthesis could consume roughly 16 CPU cores.

Increasing application-level concurrency therefore caused thread oversubscription.

Measured throughput improved with concurrency, but not linearly:

Concurrency      Throughput
1                 ~0.41 req/s
2                 ~0.41 req/s
4                 ~0.66 req/s
8                 ~0.99 req/s

The service therefore benefits from concurrent requests, but the ONNX Runtime thread pools become a limiting factor.

Oremi already exposes:

OREMI_IZWI_TTS_INTRA_OP_NUM_THREADS
OREMI_IZWI_TTS_INTER_OP_NUM_THREADS

so deployment-specific tuning is possible without imposing a global synthesis lock.

The project intentionally avoids serializing inference because the engine is designed to support concurrent requests.

Streaming Is Not Free

Oremi's streaming implementation splits text into chunks and synthesizes each chunk independently.

For short text, this means the entire synthesis may still consist of a single chunk.

Consequently, for a 200-character input, streaming does not magically reduce time to first audio:

request
   ↓
one chunk
   ↓
full inference
   ↓
first audio

For longer text, subsequent chunks can be delivered progressively.

This is fundamentally different from an autoregressive codec-based streaming architecture such as Audio8, where generation itself is incremental.

Again, the architecture determines the optimization space.

The First Quantization Experiment

The next question was obvious:

Can we simply reduce the precision of the Supertonic models?

Three approaches were investigated:

1. FP16
2. dynamic INT8
3. static INT8 QDQ

The results were revealing.

FP16: Smaller, but Not Yet a Solution

FP16 conversion succeeded for three of the four Supertonic models.

The dominant vector_estimator, however, failed to convert cleanly with the standard conversion tooling used in the experiment.

The converted models were approximately half the original disk size.

But this alone is not sufficient.

A smaller model does not automatically mean:

• faster CPU inference
• faster startup
• lower end-to-end latency

In particular, CPU FP16 acceleration depends heavily on the underlying hardware and runtime implementation.

Therefore the experiment did not justify replacing the production FP32 models with FP16.

Dynamic INT8: Memory Improved, Performance Did Not

Dynamic INT8 quantization was successfully applied to all four models.

It reduced the measured resident memory substantially:

FP32       ~483 MB
INT8       ~193 MB

That is an impressive memory reduction.

But inference became significantly slower.

The measured dynamic INT8 configuration was approximately 2–2.5× slower end-to-end.

Audio quality also degraded according to the objective waveform diagnostics.

So dynamic INT8 failed both of the important production criteria:

less memory
      ≠
better TTS performance

and

smaller model
      ≠
same audio quality

Static INT8 QDQ: The Interesting Failure

Static QDQ quantization produced the most interesting result.

It was actually faster.

The heavy models showed approximately:

The vector estimator was approximately 2.4× faster in the isolated measurement.

End-to-end synthesis was also substantially faster.

Memory dropped by approximately 43%.

On paper, this looked like the breakthrough.

Then the audio was compared.

The waveform correlation collapsed.

Even after increasing calibration data and using per-channel quantization, the generated waveform remained strongly divergent from the FP32 reference.

More importantly, isolating the stages showed that both major components contributed to the problem:

• quantized vector_estimator alone degraded the output
• quantized vocoder alone degraded the output

The combined pipeline therefore could not be considered production-ready.

No subjective MOS study was performed, so the conclusion is deliberately limited:

The objective waveform diagnostics demonstrated a major regression, but a formal perceptual evaluation was not performed.

That distinction matters.

The CPU Quantization Conclusion

The quantization experiments therefore produced a useful negative result:

The lesson is not that quantization is impossible.

The lesson is:

Generic quantization is not automatically appropriate for a diffusion TTS pipeline.

The numerical sensitivity of the diffusion denoiser and vocoder must be considered.

Then Came the GPU Investigation

At this point, the real target became clear.

The intended deployment target for Oremi Izwi is a GPU.

The test machine contains an NVIDIA GeForce RTX 5070.

Unfortunately, the GPU POC could not actually execute a single CUDA inference.

This was not a failed benchmark.

It was a deployment-readiness discovery.

The RTX 5070 Was Present — But Not Usable

The machine reported the RTX 5070 hardware.

However:

nvidia-smi

could not communicate with the NVIDIA driver.

The environment contained NVIDIA/CUDA libraries, but the actual CUDA device was not operational.

Therefore:

GPU hardware
    ✓

CUDA device
    ✗

GPU inference
    ✗

No GPU performance numbers were invented.

This distinction is important when publishing performance results.

A machine having a GPU installed does not mean that a benchmark actually exercised the GPU.

ONNX Runtime Was Also CPU-Only

The installed ONNX Runtime exposed:

AzureExecutionProvider
CPUExecutionProvider

but not:

CUDAExecutionProvider

This is consistent with the installed package being the CPU build.

ONNX Runtime's architecture is explicitly based around Execution Providers, which allow supported subgraphs to be executed on specialized hardware such as NVIDIA GPUs.

The official CUDA Execution Provider documentation confirms that NVIDIA GPU acceleration requires a GPU-capable ONNX Runtime installation and compatible CUDA/cuDNN dependencies.

Therefore the presence of CUDA tooling on disk was not sufficient.

The complete runtime stack needs to work:

NVIDIA driver
      ↓
CUDA runtime
      ↓
cuDNN
      ↓
ONNX Runtime GPU
      ↓
CUDAExecutionProvider
      ↓
Supertonic

There Is a Third Blocker: Supertonic's Provider Selection

Even after fixing the NVIDIA driver and installing a GPU-capable ONNX Runtime, another problem remains.

The version of Supertonic used by Oremi currently selects:

CPUExecutionProvider

internally.

Oremi's own TTSEngine does not currently expose an execution-provider setting.

This means the intended production architecture cannot simply be changed from:

CPUExecutionProvider

to:

CUDAExecutionProvider

inside Oremi without considering the external Supertonic dependency boundary.

This is important because Oremi does not contain the Supertonic model implementation itself. It consumes Supertonic as an external package. The repository's project documentation explicitly treats Supertonic as a hard dependency.

The Hidden GPU Problem: CPU ↔ GPU Transfers

This is arguably the most interesting finding from the GPU investigation.

Supertonic currently uses multiple independent ONNX Runtime sessions.

Conceptually:

duration predictor
        ↓
text encoder
        ↓
vector estimator
        ↓
vocoder

The intermediate results are passed between sessions through NumPy arrays.

On CPU, this is perfectly natural.

On GPU, it potentially becomes:

CPU
 ↓
GPU
 ↓
CPU
 ↓
GPU
 ↓
CPU
 ↓
GPU
 ↓
CPU

This is currently an architectural inference, not a measured GPU result.

But it is a critical hypothesis.

The GPU may accelerate the neural-network arithmetic while the application repeatedly moves intermediate tensors between host and device memory.

That can erase a surprising amount of the expected benefit.

I/O Binding Becomes Particularly Interesting

ONNX Runtime provides I/O Binding specifically for situations where inputs and outputs need to remain on device rather than being copied unnecessarily between host and accelerator memory.

The CUDA Execution Provider documentation also explicitly recommends I/O Binding when using a user compute stream to bind inputs and outputs to device tensors.

This makes the next GPU experiment much more interesting than simply:

CPU → CUDA

The real question becomes:

CPU
 ↓
GPU
 ┌──────────────────────────────┐
 │ duration predictor           │
 │ text encoder                 │
 │ vector estimator × N         │
 │ vocoder                      │
 └──────────────────────────────┘
 ↓
CPU

Can the intermediate tensors stay on the GPU?

That is still unverified.

But it is now one of the highest-value hypotheses for the next experiment.

Why TensorRT Comes Later

ONNX Runtime supports both CUDA and TensorRT Execution Providers.

TensorRT is certainly worth investigating for a production GPU deployment.

But it should not be the first step.

The correct progression is:

1. Working NVIDIA driver
        ↓
2. CUDAExecutionProvider
        ↓
3. FP32 correctness
        ↓
4. FP32 performance
        ↓
5. FP16
        ↓
6. memory-transfer analysis
        ↓
7. I/O binding
        ↓
8. concurrency
        ↓
9. TensorRT

Otherwise, a TensorRT failure could simply be hiding a more fundamental problem in the deployment stack.

What We Have Ruled Out

After the investigation, several ideas can be removed from the immediate optimization roadmap.

KV cache

Not applicable to Supertonic.

KV caching is useful for autoregressive generation because the sequence grows token by token.

Supertonic's diffusion process does not have that same autoregressive decoding structure.

Audio8's static KV cache is therefore an architectural optimization specific to its AR generation path.

Audio8's INT4 pipeline

Also not directly transferable.

Audio8's repository describes its 0.6B DualAR model and its static KV-cache-based generation architecture.

Its optimization strategy is closely coupled to that model.

Replacing Supertonic's FP32 ONNX graphs with Audio8-style INT4 kernels is not a generic runtime optimization.

Global request serialization

Not appropriate for Oremi.

Audio8's CPU service uses serialization, but Oremi deliberately keeps inference concurrent and delegates the blocking model execution to worker threads.

These are different serving strategies with different model/runtime characteristics.

CPU micro-optimizations

Text validation, chunking and PCM/WAV handling represent a tiny fraction of total synthesis time.

Optimizing these paths would not address the main bottleneck.

What Has Already Been Optimized

One optimization did provide a concrete operational improvement.

Oremi originally acquired its model download lock even when the model was already completely present.

The implementation was changed so the exclusive lock is only taken when a download is actually required.

With two concurrent replicas and the model already present, measured mean wall time improved from:

2.42 s

to:

1.81 s

It also enabled read-only model volumes, because a replica no longer needs to create/write the lock file when no download is necessary.

This change was implemented without changing the synthesis model itself.

The benchmark tooling was also extended to report startup, RTF, audio duration, streaming and concurrency behavior.

The Current Bottleneck Map

The investigation now gives us a much clearer picture.

Oremi Izwi
│
├── HTTP / FastAPI
│      └── not the main bottleneck
│
├── Python preprocessing
│      └── negligible
│
├── model loading
│      └── ~0.65–0.95 s
│
├── Supertonic
│      │
│      ├── duration predictor
│      │
│      ├── text encoder
│      │
│      ├── vector estimator
│      │      └── PRIMARY inference bottleneck
│      │
│      └── vocoder
│
├── CPU threading
│      └── oversubscription under concurrency
│
└── GPU deployment
       │
       ├── driver
       ├── ORT CUDA EP
       ├── provider injection
       └── CPU↔GPU transfers

The final four items are now the main research path.

The Next GPU Experiment

The next milestone should deliberately be small.

Not TensorRT.

Not INT8.

Not a model rewrite.

The first goal is simply:

Generate one valid Supertonic waveform through CUDAExecutionProvider on the RTX 5070.

The desired path is:

RTX 5070
   ↓
working NVIDIA driver
   ↓
CUDA-compatible ONNX Runtime
   ↓
CUDAExecutionProvider
   ↓
Supertonic FP32
   ↓
valid WAV

Once that works, measure it.

Only then should we investigate FP16.

And only after understanding FP32 and FP16 should we investigate I/O binding and TensorRT.

The GPU Benchmark That Actually Matters

The eventual benchmark should compare:

And this should be tested across concurrency levels:

1
2
4
8

The objective is not merely to obtain the lowest single-request latency.

A production TTS service needs to understand the trade-off between:

• latency
• throughput
• VRAM
• CPU usage
• concurrency
• time to first audio

A More Important Question Than "How Fast Is It?"

The real production question is:

Where should the boundary between CPU and GPU be?

A naive GPU implementation might look like:

CPU
 → model 1 → CPU
 → model 2 → CPU
 → model 3 → CPU
 → model 4 → CPU

A better implementation may eventually look like:

CPU
 │
 └──────► GPU
          │
          ├── duration
          ├── encoder
          ├── diffusion × N
          └── vocoder
                │
                └──────► CPU

If the latter is possible without breaking Supertonic's API or correctness, it could be substantially more important than another generic model quantization experiment.

But that remains a hypothesis until the GPU is operational.

Lessons From the Investigation

Several conclusions have emerged.

1. Benchmark the architecture, not just the model

A model can be fast in isolation while the complete serving pipeline remains inefficient.

2. Memory reduction is not performance

INT8 dynamic quantization proved this directly.

3. Numerical similarity matters differently in generative audio

A model can remain executable, produce the correct duration and avoid NaNs while still producing an unacceptable waveform.

4. Architecture-specific optimizations do not transfer automatically

Audio8's KV cache and INT4 strategy are excellent examples.

5. Negative experiments are valuable

The INT8 QDQ experiment produced a significant speed improvement.

It was still a failure for production because the audio regression was too large.

That is useful information.

6. A GPU installed in a machine is not a GPU benchmark

The RTX 5070 was physically present.

The driver was not functional.

No CUDA inference was performed.

Therefore no GPU performance claim should be made from that machine yet.

Where Oremi Izwi Goes Next

The immediate roadmap is now much more focused:

Repair NVIDIA driver
        ↓
Install/verify GPU-capable ONNX Runtime
        ↓
Expose CUDA provider to Supertonic
        ↓
Run FP32 CUDA synthesis
        ↓
Benchmark
        ↓
FP16
        ↓
CPU↔GPU transfer analysis
        ↓
I/O Binding
        ↓
Concurrency
        ↓
TensorRT
        ↓
Only then revisit GPU quantization

This is deliberately conservative.

The goal is not to turn Oremi Izwi into a complicated inference framework.

The goal is to preserve the existing API and architecture while moving the expensive computation to the hardware that is actually intended to run it.

Conclusion

The performance investigation started with a seemingly simple question:

Can Oremi Izwi be made significantly faster?

The answer turned out to depend much more on architecture and execution placement than on Python micro-optimizations.

The CPU investigation established a solid FP32 baseline and ruled out several generic quantization approaches.

The Audio8 comparison demonstrated why model-specific optimizations such as KV caching and INT4 cannot simply be transplanted into a diffusion-based TTS pipeline.

The GPU investigation then revealed that the current environment cannot yet perform CUDA inference at all, and that Supertonic's current provider selection introduces an additional integration boundary.

The most promising path is therefore not another blind optimization.

It is a clean GPU execution path:

Supertonic
    ↓
CUDA
    ↓
FP32 baseline
    ↓
FP16
    ↓
minimize host/device transfers
    ↓
I/O binding
    ↓
TensorRT if justified

Only measurements from a functioning GPU environment will tell us which of these steps actually matters.

For now, the most defensible production configuration remains:

Oremi Izwi
+
Supertonic FP32
+
CPUExecutionProvider

—not because CPU is necessarily the final architecture, but because it is currently the only configuration that has been fully measured and validated.

The next meaningful milestone is much simpler:

make the RTX 5070 execute one correct Supertonic inference, then measure everything from there.

References

Oremi Izwi

Oremi Izwi — GitLab repository

Oremi Izwi — project documentation

The Oremi Izwi repository contains the FastAPI service, TTSEngine, configuration, HTTP API, tests and deployment artifacts discussed in this article.

Supertonic

Supertonic — GitHub project

Supertonic's project documentation describes the system as an ONNX Runtime-based local TTS system and documents Supertonic-3's multilingual capabilities.

The associated research includes SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech System, which describes the architecture behind Supertonic.

Audio8-TTS

Audio8-TTS — GitHub repository

The Audio8 repository documents its 0.6B DualAR architecture, autoregressive generation, codec, supported languages and static KV caches.

ONNX Runtime

ONNX Runtime — Execution Providers

ONNX Runtime's Execution Provider architecture allows supported model nodes and subgraphs to execute on hardware-specific backends such as CUDA and TensorRT.

ONNX Runtime — CUDA Execution Provider

The CUDA Execution Provider documentation covers NVIDIA GPU execution, CUDA/cuDNN compatibility, provider configuration and I/O Binding.