HUMAIN m3 Inference Performance Assessment

Scope: inference serving assessment as deployed on the HUMAIN Node gateway (OpenAI-compatible /v1/chat/completions, streaming) — not a base-model capability comparison. 30 s per benchmark (~38 min total incl. rerun); max concurrency tested 512 (deliberate overload probe), max sustained 115. Errors are dominated by the deliberate overload probes.

Executive Summary

• Fast generator, moderate first-token latency. Inter-token latency is 3.6–7.3 ms p50 (≈140–280 tok/s per stream — very fast streaming); TTFT p50 is 559–867 ms across shapes. Users see the first token in well under a second, then a rapid stream.
• Throughput plateaus at ~21–22 req/s regardless of prompt/output shape; output-token throughput scales with output length (peak ≈3.0k tok/s chat, 5.9k long-ctx, 11.5k balanced; ≈30k total tok/s).
• Scaling is near-linear up to a clear knee, then fail-fast 429s. The gateway sheds load with HTTP 429 (31,188 of 31,218 errors) instead of letting latency explode — good overload behavior, but clients must implement backoff.
• Sweet spots (measured): chat 256/128 → concurrency ~16–24; long-ctx 1024/256 → ~24–40; balanced 512/512 → ~40–78. Each holds 15–20 req/s with zero errors.
• Steady 17 RPS (1024 RPM budget) is sustainable for all three shapes: 16.3 / 16.0 / 15.1 req/s with zero errors on the rerun; TTFT p50 591 / 756 / 804 ms.
• End-to-end is output-dominated: E2E p50 ≈1.1 s (128 out) → 1.8 s (256 out) → 3.5 s (512 out). TTFT is roughly constant; generation time drives E2E.
• Caution — first 512/512 attempt collapsed entirely (0/510 and 0/2610 successful) after the suite had saturated the endpoint; a rerun ~30 min later succeeded fully. Treat 512-token outputs as the capacity-sensitive shape: cap concurrency, keep headroom, retry with backoff.
• Verdict: conditionally production-ready for chat, copilot, and agentic workloads at capped concurrency with 429-aware clients; strong fit for batch. Not approved for uncapped autoscaling or burst traffic without further testing.

Key results

What is HUMAIN Node

HUMAIN Node is HUMAIN’s gateway to frontier and sovereign AI models: one account, one API key, and one bill across many providers, with OpenAI-compatible endpoints, team budgets, and a choice of Global, in-Kingdom, or sovereign hosting per workload — without changing client code.

The model under test, humain-m3-research-preview, was commissioned by HUMAIN and delivered by MiniMax: a 428-billion-parameter mixture-of-experts with 23B active parameters per token. It launched in research preview in September 2026, with open weights planned to follow under the MiniMax Community License. Access is a controlled, territory-by-territory preview — exactly the kind of early window I was grateful to be given.

Benchmark Configuration

Sweep step 1 in every run is a deliberate throughput overload probe (max_concurrency 512) that intentionally triggers thousands of 429s. It is marked OVERLOAD PROBE below and excluded from scaling curves.

Performance Overview

Across 16,126 successful requests the endpoint behaves like a well-batched streaming server behind a strict rate limiter. Throughput rises near-linearly with concurrency until a per-shape knee, flattens at ~21–22 req/s, and then sheds excess load with HTTP 429 rather than queueing indefinitely. TTFT is dominated by prompt processing (≈560 ms at 256 tokens → ≈750–870 ms at 512–1024 tokens) and stays flat across the efficient zone. Generation cadence (ITL 3.6–7 ms) is excellent and only degrades past saturation. End-to-end latency is therefore a near-linear function of requested output tokens — a customer-visible property worth documenting in any SLA.

Concurrency Analysis

Chat 256/128 — sweep steps

Long-context 1024/256 — sweep steps

Balanced 512/512 — sweep steps (rerun)

Steady-load checks (30 s each)

Throughput Scaling

Concurrency vs requests/sec

Near-linear to each shape's knee, then a hard plateau at ~21–22 req/s. The plateau is identical across shapes — consistent with a gateway/request-rate ceiling rather than GPU exhaustion (which would plateau token throughput, not request rate). Hypothesis only; server telemetry would be needed to confirm.

Concurrency vs output tokens/sec

Token throughput keeps climbing with output length long after request rate plateaus: the server converts the same ~21 req/s into far more tokens for 512-output workloads (peak 11.5k out tok/s, ~30k total tok/s).

Latency Analysis

End-to-end p95 vs concurrency (ms)

E2E is output-dominated: ~1.4 s (128 out) → ~2.2 s (256 out) → ~5.0 s (512 out) inside the efficient zone. Past the knee it grows steadily rather than exploding — because excess load is rejected (429) instead of queued.

TTFT Analysis

TTFT p50 vs concurrency (ms)

TTFT p99 vs concurrency, log scale (ms)

What TTFT means: time from sending a request to the first visible token — the "did it hear me?" delay users feel. p50 559–867 ms is acceptable for chat/copilots (feels responsive, especially with a typing indicator); p99 1–2.5 s under load means occasional slow starts. TTFT scales with prompt length (prefill cost) and is flat across concurrency until saturation — good scheduler behavior. Only the overload probe (excluded) and post-knee steps show TTFT p99 degradation.

Token Generation Performance

Inter-token latency p50 vs concurrency (ms)

Perceived streaming speed: 3.6–5 ms ITL ≈ 200–280 tok/s per stream in the efficient zone — text visibly pours out. Even at conc 115 (balanced) ITL p50 is 7.3 ms (~140 tok/s), still faster than reading speed. TPOT p50 (includes TTFT amortized) is ~9 ms chat, ~7 ms long-ctx, ~9 ms balanced — consistent with E2E ÷ output tokens. Generation consistency is high: ITL p99 stays under 12 ms everywhere except the deliberate overload probe (22 ms).

Scalability Curve

Efficient zone (conc ≲ 30): linear gains, flat TTFT, zero errors for chat/long-ctx. Watch zone (30–80): shape-dependent — long-ctx and balanced still scale cleanly here while chat starts shedding (first 429s at conc 29). Saturated zone (>80): request rate flat at ~21 req/s, rising 429 share, TTFT p99 creeping up. Recommended production concurrency sits inside the efficient zone with headroom (see below).

Reliability

• Error mix (measured): 31,188 × HTTP 429 Too Many Requests, 30 × HTTP 503 (one cluster in balanced-sweep step 9 rerun). No timeouts, no malformed responses, no partial streams in the sampled data.
• In-operating-range success rate: 100% — zero errors on all constant @17 RPS and concurrent @16 runs (rerun), and zero errors on every sweep step below each shape's knee.
• Incomplete requests (e.g. 21–100 per 30 s window) are in-flight requests cut off by the max_duration constraint — a benchmark artifact, not a server signal.
• Anomaly — first balanced attempt: constant (0/510) and concurrent (0/2610) returned zero successes when run immediately after two full suites; the rerun ~30 min later succeeded fully (453 ok / 172 ok). Consistent with lingering server-side throttling after sustained saturation, though the exact mechanism is unknown. Lesson: 512-output workloads need headroom and backoff; do not size capacity from a single post-saturation sample.

Measured failure evidence

request_id : 801b8b86-7fc6-4c40-841d-904da132dccc
status     : errored
targeted   :  +0.000s (t0)
queued     :  -11.088s
dequeued   :  +0.001s
req_start  :  +0.001s
first_tok  :  —
last_tok   :  —
req_end    :  —
resolved   :  +0.165s
req_iters  :  0   tok_iters : 0
error      :  HTTPStatusError("Client error '429 Too Many Requests' for url 'https://api.node.humain.com/v1/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429")

request_id : dcf3000f-c84c-4fc3-9973-5868e15f0a18
status     : errored
targeted   :  +0.000s (t0)
queued     :  -9.515s
dequeued   :  +0.000s
req_start  :  +0.000s
first_tok  :  —
last_tok   :  —
req_end    :  —
resolved   :  +0.163s
req_iters  :  0   tok_iters : 0
error      :  HTTPStatusError("Server error '503 Service Unavailable' for url 'https://api.node.humain.com/v1/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503")

request_id : f0c8805d-fc0f-4317-984f-ae92fe7391f2
status     : completed
targeted   :  +0.000s (t0)
queued     :  -1.139s
dequeued   :  +0.013s
req_start  :  +0.013s
first_tok  :  +0.843s
last_tok   :  +2.483s
req_end    :  +2.497s
resolved   :  +2.497s
req_iters  :  204   tok_iters : 198
error      :  none

Source: measured per-request entries in benchmarks/run-3-balanced/sweep/benchmarks.json (request/response bodies omitted; errored entries carry no request payload). Response headers (Retry-After, X-Request-ID) are Not Available — the load harness does not capture response headers; see Additional Testing Required.

Workload Suitability

Long context (1024 prompt, measured)

4× prompt length adds only ~165 ms to TTFT p50 (591 → 756 ms) and E2E grows 1.14 → 1.80 s — sub-linear, indicating efficient prefill. Tightest tail of all shapes at steady load (TTFT p99 1,006 ms). Larger windows (8k+) were Not Available in this dataset and must be tested before advertising long-context support.

Production Recommendation

• Traffic management: client-side concurrency limiter per shape + exponential backoff with jitter on 429; never retry 429s immediately (the probe shows immediate retries amplify the storm: 17k errors in 30 s).
• Timeouts: client timeout ≥30 s for 512-output workloads (E2E p99 ~5–7 s, plus queueing); ≥10 s for chat.
• Autoscaling: scale on p95 TTFT and 429 rate, not on RPS alone — RPS plateaus while user experience degrades. Keep one replica's headroom above the knee.
• Load balancing: route by expected output length (short vs long generations) since E2E differs 3–4×; isolate 512-output traffic so it cannot starve interactive chat.
• Capacity planning: budget ~21 req/s and ~10k/24k/19k total tok/s per endpoint for chat/long-ctx/balanced respectively at 17 RPS; derate 30% for p99 headroom.

Pricing and what it costs to run

HUMAIN Node prices humain-m3 at $0.30 per 1M input tokens and $1.20 per 1M output tokens — output costs 4× input, so the bill follows generation, exactly like end-to-end latency does. Applied to the measured steady-load token rates:

• Output dominates the bill: roughly 50% of cost on chat shapes, ~45% on long-context, ~72% on 512-output workloads. Prompt caching (priced separately on Node) would move the input share further down.
• FinOps takeaway: cap output length first, concurrency second. A 512-output workload costs ~3× a chat workload per request at the same request rate — route long generations to isolated pools so they cannot starve (or bill like) interactive chat.
• Budgeting rule of thumb: ~$20/hour per saturated chat endpoint, ~$40–48/hour for long-generation shapes at ~17 RPS. Derate 30% for p99 headroom, same as capacity.

Risks / Concerns

Additional Testing Required

• Longer sustained load (15–60 min soak) at recommended concurrency to check for drift, KV-cache pressure, or slow throttling onset.
• Larger context windows (4k/8k/32k prompts) — prefill scaling is currently only measured to 1024 tokens.
• Burst test: idle → 2× recommended concurrency step jump; measure recovery time (the observed ~30 min recovery needs quantification).
• Multi-turn conversations and tool-calling schemas (agentic realism) instead of single-turn synthetic prompts.
• Failure testing: behavior at 2× ceiling, retry-storm resilience, 503 handling.
• Multi-replica and autoscaling behavior; per-key RPM accounting under parallel clients.
• Repeated 512-output runs to confirm the first-attempt collapse root cause and rule out residue effects.

Scorecard

Expert Assessment

If I were approving this endpoint for production today: approve chat/copilot/assistant traffic at ≤24 concurrent (≈16 RPS) with 429-aware retries and ≥10 s timeouts; approve offline batch at paced ≤17 RPS with a queue; conditionally approve agentic workloads with per-step budgets and burst caps. Limit: no uncapped autoscaling, no 512-output traffic sharing a pool with interactive chat, no reliance on >21 req/s per endpoint. Test next: a 60-minute soak at recommended concurrency, a quantified burst-recovery test, and 4k+ context — in that order. The single biggest open question is the post-saturation throttling episode: until its trigger and recovery time are characterized, keep 30% headroom below the knee.

Evidence class: all numeric claims above are measured from the 9 benchmark runs; knees, zones, and sweet spots are observed patterns; infrastructure causes (gateway ceiling, GPU/scheduler saturation) are inferred hypotheses — server-side telemetry was Not Available.

Why this matters for AI sovereignty

Performance numbers are only half the story for a Saudi bank, government entity, or regulated enterprise. The other half is where the tokens are processed. HUMAIN runs inference on sovereign data centers inside the Kingdom — AMD Instinct systems went live in 2026 with a 1 GW buildout target by 2030 — so workloads stay aligned with NCA, SAMA, and PDPL expectations without data crossing borders. That is the same compliance-by-design argument I make in my AI adoption work: sovereignty is not a deployment detail, it is architecture.

The direction of travel is unmistakable: frontier open models hosted on in-Kingdom inference platforms, and partnerships bringing global model builders onto Saudi infrastructure. For regulated buyers, the question is shifting from “can we use AI and stay compliant?” to “which in-Kingdom endpoint serves each workload best?” — and that is exactly what benchmarks like this one answer.

My feedback

Testing HUMAIN inference first-hand was a privilege, and the platform made a strong impression: the OpenAI-compatible API was a genuine drop-in (existing SDKs, streaming with usage stats, no client rewrite), and the gateway’s fail-fast 429 behavior under overload is how I wish every inference provider behaved — shed load loudly instead of queueing silently.

My honest notes back: the post-saturation throttling episode (a full 100% failure window on 512-output workloads after sustained overload, recovering ~30 min later) is the one thing I would want characterized before committing production traffic; the small 503 cluster deserves the same retry coverage as 429s; and preview rough edges like the /healthz vs /health mismatch are worth sanding down for enterprise load-harness integrations. None of these change the verdict — conditionally production-ready at capped concurrency — and I’m looking forward to re-running this suite against the open weights and larger context windows next.