shithub: libvpx

Info  •  Files  •  Log  •  Branches

rd_pick_inter_mode: prevent signed integer overflow

calculate the txfm_cache difference first as both values may be
INT64_MAX with the intent that they cancel each other out.

Change-Id: I214d072458e1b24f60289974e6302af1aff7b66c

--- a/vp9/encoder/rdopt.c

+++ b/vp9/encoder/rdopt.c

@@ -4090,7 +4090,9 @@

         for (i = 0; i < NB_TXFM_MODES; i++) {

           int64_t adj_rd;

           if (this_mode != B_PRED) {

-            adj_rd = this_rd + txfm_cache[i] - txfm_cache[cm->txfm_mode];

+            const int64_t txfm_mode_diff =

+                txfm_cache[i] - txfm_cache[cm->txfm_mode];

+            adj_rd = this_rd + txfm_mode_diff;

           } else {

             adj_rd = this_rd;

           }

-- 

⑨