basal
A language-model inference engine. My contributions cover GPU performance, cache correctness, and observability.
-
Fast mode by default for Qwen3.5 on SM120 GPUs #114
Select the Fast backend automatically on Blackwell-class SM120 GPUs when FLA is available. 27 benchmark runs with zero request errors, and 99.52% top-1 agreement with the fp32 eager reference.
- Short prompt, 1 question
- 2.46×
- Short prompt, 5 questions
- 6.32×
- Long prompt, 5 questions
- 8.64×
Figure 1. Median throughput of Fast relative to Eager, log scale, three repetitions per workload. RTX 5060 Ti 16 GB on Windows; figures from PR #114. GitHub lists this pull request as closed because the maintainer landed my commits b6eaf06 and a8fcba2 directly in the release, as noted on the PR.
-
Preserve shared prefixes in StateCache #56
The prefix index kept one entry per block hash, so caching a second branch could hide the first one’s longer match. The fix keeps every branch discoverable, selects the longest usable prefix, and is covered by regression tests and a randomized oracle.
- Reused
- 0
- Computed
- 0
- Prefill skipped
- 0%
Tickets, probabilities, and token counts are illustrative. Reuse changes the work, never the answer.
Figure 2. Prefix reuse across typed-decision requests on support-ticket threads, drawn like a commit graph. Each request is a cached state plus a question, answered with a probability per allowed option in one forward pass; only the uncached part is prefilled. Before #56, caching thread B overwrote the index entry it shares with A, so a question about A matched only the shared prefix. -
Prometheus serving metrics #48
A dependency-free
/metricsendpoint for build, load, and state-cache statistics, collected only at scrape time.