shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix integer overflow issue in bits allocated

When encoding at high bitrates, integer overflow
occurs in the the calculation of bits allocated
for layered ARF frames.

Change-Id: I94ad9eea759367a222235a3b5d1c777578dc6ba9

--- a/vp9/encoder/vp9_firstpass.c

+++ b/vp9/encoder/vp9_firstpass.c

@@ -2340,7 +2340,7 @@

       switch (gf_group->update_type[idx]) {

         case ARF_UPDATE:

           gf_group->bit_allocation[idx] =

-              (int)((arf_depth_bits[gf_group->layer_depth[idx]] *

+              (int)(((int64_t)arf_depth_bits[gf_group->layer_depth[idx]] *

                      gf_group->gfu_boost[idx]) /

                     arf_depth_boost[gf_group->layer_depth[idx]]);

           break;