Optimize an A65 kernel
Optimize the complete checked call, not an isolated arithmetic loop. Keep the public result contract stable while reducing memory traffic, repeated addressing and redundant computation. Start with the operation's documented semantics and the qualification workflow.
Choose precision per port
Preprocessing and postprocessing describe a position in a pipeline, not one universal conversion matrix.
| Port role | Design guidance |
|---|---|
| Preprocessing input | Consider BF16, FP16 and INT8 where meaningful; retain applicable FP32/UINT8 support. |
| MLA-facing activation | Use the operation's admitted BF16/INT8 dtype and physical layout. |
| Postprocessing input from MLA | Consume admitted BF16/INT8 directly; do not assume FP16 MLA output. |
| Result output | Choose the operation's result type, such as FP32 scores or INT32 indices. |
This table guides new designs; it does not extend existing APIs. Quant accepts BF16/FP16/FP32 and produces INT8. DetessCast consumes BF16, not FP16. Argmax has additional supported inputs and returns INT32 indices. Check the support matrix and the selected operation's public header.
Keep logical shape, scalar dtype and physical layout independent. Six-dimensional capacity does not require six padded axes. Requirements for tiled or strided storage include more than logical payload bytes; see tensor storage.
Preserve results, not an arbitrary instruction sequence
Write down arithmetic order before replacing it. Quant rounds its product to binary32, applies the selected integer rounding, adds the zero point and then saturates. Moving the zero point inside rounding, clamping too early or changing product precision can change the result. Dequant's staged arithmetic similarly does not authorize an arbitrary fused multiply-add.
Fusion of memory passes and contraction of arithmetic are different changes. The former can remove an intermediate tensor without changing arithmetic. The latter needs proof and independent tests against the numerical contract.
BF16 bits are not FP16 bits. The current Quant BF16-to-half route constructs bounded encodings only for eligible parameters; it does not reinterpret BF16 as IEEE half. Its fallback widens values in registers, not into an allocated FP32 tensor. Integer-only code or narrower arithmetic is not necessarily faster. Read Quant's selection rules before reusing that technique.
FP environment requirements are operation-specific and can depend on the chosen path. Preparation on one thread does not validate another thread's execution environment. Reject unsupported state before writes; do not modify caller FP control settings. Integer-key Argmax comparisons do not inherit Quant's FP-mode requirements. Preserve each operation's NaN, signed-zero and tie rules rather than applying one shared policy everywhere.
Use explicitly qualified, correctly typed math calls. Header cleanup can change overload resolution and intermediate precision even when the expression looks unchanged. For the shared rationale, see numerical behavior.
Fuse around the physical representation
Prefer these paths when the contract allows them:
Preprocessing: load → convert/compute in registers → write MLA layout
Postprocessing: load MLA layout → reconstruct required lanes → compute → results
Postprocessing usually consumes tessellation; it only tessellates its output when the next consumer requests that format. Avoid full detessellation followed by a second full-tensor pass when computation can consume the original layout.
- Distinguish pixel-major
MlaTiledfromMlaCBlockedchannel stripes. - Reconstruct BF16 low/high byte planes in registers.
- Traverse contiguous physical rows, tiles or stripes rather than repeatedly decoding logical coordinates for every value.
- Prepare invariant tile skips and use incremental coordinates when a general traversal remains necessary.
- Respect partial tiles and compact alignment gaps. Padding is not logical data.
- Initialize required output padding and unused result capacity deterministically, including after reuse. Remove redundant clears only when all required bytes are still written on every successful path.
Keep hard-coded physical-layout goldens: matching pack/unpack bugs can pass a round-trip test. Poison input padding to demonstrate that results ignore it.
Use NEON where it reduces complete-call cost
Select eligible implementations during preparation. Keep public typed parameters separate from private route-selection details. Runtime CPU features and numerical eligibility both matter for optional instructions such as native FP16 SIMD.
Useful patterns already used in the library include:
| Workload | Pattern to consider | Constraint to retain |
|---|---|---|
| Short conversion | Scalar or single-vector route without large-loop setup | Same rounding and safe remainder |
| Narrow channels | Vectorize across neighboring pixels | Correct byte-plane placement |
| Independent reductions | Process adjacent outputs together | Full-width indices and stable ties |
| Channel-blocked input | Walk C16 stripes and reuse tile metadata | Exclude partial-stripe padding |
| Detection scores | Prepared cutoffs/tables and lazy geometry | Exact admission, ordering and output policy |
| Repeated addressing | Bounded descriptors reused across enough work | Setup cost must pay off |
Avoid repeated coordinate division, excessive scalar lane extraction and long dependency chains. Independent accumulators can help, but too much unrolling increases register pressure and code size. Check generated code for spills and unexpected library calls. Do not add prefetching or specialization without target evidence for its eligibility range.
SIMD lanes are not CPU cores. Kernels execute on the caller's thread; scheduling independent work across cores belongs to Runtime, not a private kernel pool.
Keep tails and reductions exact
A masked result does not make an out-of-bounds vector load legal. Every access must fit the actual accessible span. A bounded overlapping final vector is only valid when the whole load is in bounds and repeated lanes cannot change ordering or results. Scalar tails must preserve the vector route's numerical contract.
For Argmax, first logical index wins ordinary ties and the first NaN wins; signed zeros compare equal. SIMD trees, physical traversal and integer ordering keys must preserve those rules. Padding must never become a candidate. Do not narrow indices merely because input values are bytes or BF16.
BoxDecode has recipe-specific ordering and numerical rules; do not substitute
Argmax's policy blindly. In particular, final max_detections is not permission
to discard all but that many candidates before geometry validation or NMS. Read
the BoxDecode contract before pruning work.
Exercise protected-page inputs, lengths around vector boundaries, odd channel counts, partial tiles, extreme values and reused outputs. Independent numerical references and layout goldens are required alongside production-path comparisons.
Measure and keep the simpler justified design
Separate preparation, context creation, first call and steady checked execution. Successful execution must not allocate; scratch capacity comes from prepared metadata, not a first-frame occupancy guess. Include binding checks in normal execution timings, and verify outputs outside timed regions.
Use stable workload IDs, fixed distributions/seeds and recorded compiler/device metadata. Sparse and dense detection workloads, tiny transforms and large reductions can favor different routes. Test neighboring shapes and fallbacks, not only the case that motivated the change. A percentile of batched averages is not an individual-call tail percentile.
Serialize device workloads and keep sanitizers out of performance builds. Tie measurements to source and executable hashes; rebuild after source changes rather than trusting timestamps alone. Include order, inlining and code layout can alter timings even when source-level arithmetic is unchanged.
Follow the checked-in latency policy instead of duplicating thresholds here. Median determines regression acceptance; other statistics still help diagnose behavior. A small repeated timing step is a reason to investigate measurement granularity, not proof that a regression is noise. A permitted allowance is not a speedup, and replaying old reports is not a new device run.
Retain a specialization only when correctness and representative measurements justify its complexity. Keep a simpler qualified fallback for the remaining domain. Do not change baselines, tolerances or case coverage to manufacture a passing optimization. Document changed eligibility and implementation rationale; change the mathematical explanation only when the result semantics actually change.