shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix overflow in RD error calculation code.

Change-Id: I61ef1f198c876f9f79787ea7d7385a862cfbae19

--- a/vp9/encoder/vp9_rdopt.c

+++ b/vp9/encoder/vp9_rdopt.c

@@ -603,7 +603,7 @@

   for (i = 0; i < block_size; i++) {

     int this_diff = coeff[i] - dqcoeff[i];

-    error += this_diff * this_diff;

+    error += (unsigned)this_diff * this_diff;

   }

   error >>= shift;