Preproc numerical order

Preproc has four numerical stages. Their order and intermediate precision are part of the contract: changing either can change output bits. Use the operation overview for tensor specifications, parameters and a C++ call. The implementation notes explain how the kernel executes these stages without a full-frame intermediate.

Ordered computation

These rules define Preproc output bits; they do not promise bit-identical results with a default OpenCV resize.

Canvas padding bypasses sampling and color conversion, but not affine/output conversion. Physical padding is a separate storage rule, not an image pixel.

1. ROI sampling and interpolation

Source taps replicate the nearest ROI border. ROI coordinates retain their full-source chroma phase. NV16 selects chroma at (source_x/2, source_y); NV12/I420 use (source_x/2, source_y/2), with integer division. NV16 therefore replicates chroma horizontally only; it must never reuse NV12's vertical row replication or NV12-only fast path.

The example shows four source taps, not resized outputs. Add the ROI origin before dividing full-source coordinates for chroma. Resetting the ROI's chroma origin to (0,0) would select different samples. NV12 and I420 share this coordinate mapping; their UV storage differs.

For a destination-relative coordinate d, source ROI extent S and destination extent D, compute separately rounded FP32 operations:

float ratio = float(S) / float(D);
float coordinate = (float(d) + 0.5f) * ratio - 0.5f;

Mode	Ordered sampling contract
Identity	Source and destination rectangle sizes must match; copy corresponding source samples
Nearest	Floor the half-pixel coordinate and clamp to ROI bounds; not round-to-nearest or asymmetric nearest
Bilinear	FP32 coordinates; fractions rounded to FP16; horizontal top/bottom lerps followed by vertical lerp, each half subtraction/multiplication/addition separately rounded, without FMA
Bicubic	Keys cubic coefficient −0.75; FP32 polynomial evaluation and row-major 16-tap accumulation; clamp source-channel result to 0…255, then round to FP16
BoxAverage	Uniform mean of all touched source cells, exact integer footprint bounds, row-major FP32 accumulation/division, then FP16 rounding; at most 8×8 cells

For BoxAverage, each axis uses the half-open integer interval [floor(d*S/D), ceil((d+1)*S/D)). This is not overlap-weighted Area for fractional ratios. For example, reducing [0,0,120] from 3 pixels to 2 gives 60 for its second box, not the 80 of overlap-weighted area.

Host admission checks this eight-cell limit without iterating over output coordinates. Write S = q*D + r, with 0 <= r < D. The maximum touched-cell count over the axis is:

r == 0:            q
r != 0 and D % r == 0: q + 1
otherwise:         q + 2

The fractional start positions visit multiples of gcd(r,D); an extra cell is crossed exactly when r > gcd(r,D). This is equivalent to D % r != 0 for nonzero r. Both axes must fit eight cells. Dynamic ROI preparation therefore uses constant work per axis, while preserving the same accepted geometries and pixel arithmetic.

Bicubic weights use a=abs(distance):

// Each indicated FP32 operation is separately rounded; no FMA.
float near_weight = ((1.25f * a - 2.25f) * (a * a)) + 1.0f;
float far_weight = ((3.75f - 0.75f * a) * a - 6.0f) * a + 3.0f;
// near for a <= 1; far for 1 < a < 2; zero otherwise.

Each tap contributes pixel * (weight_x * weight_y) to the sequential row-major FP32 sum. Specializations must preserve this order. See sampling implementations for equivalent integer paths, tap reuse and their eligibility.

2. YUV color conversion

Clamp interpolated Y,U,V to 0…255 and truncate to integers. Subtract 128 from U/V, but do not subtract 16 from Y. Compute signed integer RGB using the selected coefficient set and arithmetic right shift by 6:

Matrix	U→B	U→G	V→G	V→R
Bt601Full	113	−22	−46	90
Bt709Full	119	−12	−30	101

R = (64 * Y + coefficient_vr * V_centered) >> 6;
G = (64 * Y + coefficient_ug * U_centered + coefficient_vg * V_centered) >> 6;
B = (64 * Y + coefficient_ub * U_centered) >> 6;

These are the specified quantized coefficient approximations. They are not limited-range conversion or a promise of byte identity with a different color library. Signed RGB can lie outside 0…255; BF16/FP16 retain that range through normalization. INT8 and UInt8 clamp at their later pixel-conversion stage.

3. Affine transformation and output conversion

BF16 and FP16 evaluate (pixel - offset[channel]) * multiplier[channel] as separate FP32 subtraction and multiplication, without FMA. BF16 retains the upper 16 FP32 bits: truncation, not Cast's nearest-even conversion. FP16 uses IEEE binary16 nearest-even, including signed zero and representable subnormals; finite overflow converts to infinity. Finite parameters can still produce floating overflow; this is not an input NaN-admission promise.

INT8 first clamps the semantic pixel to 0…255 and truncates to UInt8, then:

// half(...) denotes a binary16 nearest-even rounding boundary.
a = half(half(pixel_u8) - half(offset[channel]));
b = half(a * half(multiplier[channel]));
c = half(b * half(quantization.multiplier));
q = saturate_int8(saturate_int16(truncate_toward_zero(c))
                 + quantization.zero_point); // zero-point addition is wide

Do not combine the two multipliers or replace subtract/multiply with an affine FMA: those transformations change rounding. UInt8 only clamps/truncates the raw pixel. Semantic channel reordering happens when storing the final output.

4. Canvas and physical padding

Canvas pixels outside destination use padding through the same output conversion path. This is different from extra physical MLA channels: INT8 PadTo16 channels contain the quantization zero point; UInt8/BF16 padded channels contain positive-zero bits. Tile-alignment bytes are zero.

Every required output byte is written on each execution. There is no global output-address cache or “first pixel looks intact” shortcut. Callers may dirty or reuse their output buffers between completed operations.

ROI-list batches

Params::source_rois optionally owns RoiSelection{source_index, rectangle} records. Absent means ordinary preprocessing; an engaged empty list means zero valid crops and a completely zero-filled output. The scalar source_roi and list cannot be combined. ROI lists support RGB, BGR and Gray sources only. YUV supports ordinary in-frame source_roi but not virtual ROI lists.

Input leading dimensions identify source frames. Output leading dimensions independently specify capacity, from one to 50 crops. Output images share one destination, layout, dtype and affine/quantization contract. Each accepted ROI has positive extents at most 4096 by 4096 and may originate anywhere in signed 32-bit coordinates. Identity interpolation requires each accepted ROI extent to match its destination; BoxAverage retains its eight-tap-per-axis limit.

Each RoiSelection may provide an optional destination rectangle. When absent, it inherits Params::destination. Explicit rectangles must have positive extents and fit the output canvas; they permit different letterbox regions for different ROI aspect ratios without CPU image repacking. The kernel does not choose an aspect-ratio policy: the caller supplies the checked rectangle. Plan::rois() retains resolved destinations for accepted selections. In cold named records, destination_x, destination_y, destination_width and destination_height must be supplied together or all omitted.

Admission processes records in order:

1. Drop a record with an invalid source-frame index or nonpositive extent, incrementing Plan::dropped_invalid().
2. If output capacity is already filled, drop the remaining valid record and increment Plan::dropped_overflow().
3. Otherwise retain the record in Plan::rois(), preserving order. The retained size is the valid output count; Plan::requested_rois() records the input count.

These facts belong to the immutable checked kernel plan. The generic preparation_facts() API exposes requested_rois, valid_rois, dropped_invalid, dropped_overflow, source_indices, and accepted_rois in accepted output-slot order. Each accepted record retains its source index, source rectangle, and resolved destination rectangle. Consumers can publish exact per-ROI geometry without repeating filtering or guessing which duplicate source index was admitted. Runtime retains those facts with its prepared operation. Ordinary image mode returns no facts; an explicitly empty list returns zero counts and an empty source-index array. Frameworks publish these facts instead of implementing Preproc-specific filtering. See named requests.

Virtual-image order

For accepted rectangle (x,y,w,h), construct the conceptual w by h raw image:

$$V(v,u,c) = \begin{cases} I_{source}(y+v,x+u,c), & 0\le x+u<W,\quad 0\le y+v<H,\\ P(c), & \text{otherwise}. \end{cases}$$

P uses semantic RGB padding (element zero for Gray), reordered into BGR when needed. Resize first clamps its taps to this virtual ROI, then samples the source or raw padding. It does not clip the ROI to the physical image or replace after-resize pixels with padding. The existing interpolation, color, affine, quantization and storage order then applies unchanged. The implementation stages bounded rows, never materializes a full virtual image.

Accepted crops fill output slots in order. Every unused slot is binary zero, including physical MLA padding—not quantized canvas padding or a zero-point fill. An empty list also clears every slot, so prior output bytes cannot survive as valid crop data. A later execution failure does not roll back already completed crops.