shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix an integer overflow in vp9_mcomp.c

The MV unit test revealed an integer overflow issue in vp9_mcomp.c.
This was caused if the MV was very large. In mv_err_cost(), when
mv->row = 8184, mv->col = 8184 and ref_mv is 0, mv_cost = 34363
and error_per_bit = 132412, causing the overflow.

BUG=webm:1406

Change-Id: I35f8299f22f9bee39cd9153d7b00d0993838845e

--- a/vp9/encoder/vp9_mcomp.c

+++ b/vp9/encoder/vp9_mcomp.c

@@ -101,10 +101,8 @@

                        int *mvcost[2], int error_per_bit) {

   if (mvcost) {

     const MV diff = { mv->row - ref->row, mv->col - ref->col };

-    // This product sits at a 32-bit ceiling right now and any additional

-    // accuracy in either bit cost or error cost will cause it to overflow.

-    return ROUND_POWER_OF_TWO(

-        (unsigned)mv_cost(&diff, mvjcost, mvcost) * error_per_bit,

+    return (int)ROUND64_POWER_OF_TWO(

+        (int64_t)mv_cost(&diff, mvjcost, mvcost) * error_per_bit,

         RDDIV_BITS + VP9_PROB_COST_SHIFT - RD_EPB_SHIFT +

             PIXEL_TRANSFORM_ERROR_SCALE);

   }