shithub: libvpx

Download patch

ref: a439b3c97709604a61fa9ff5d8d02aeae633a69c
parent: 0aa83d61a18fbdd5921247e0401b0fbba443cf35
author: Matthias Räncker <theonetruecamper@gmx.de>
date: Thu Sep 20 15:57:25 EDT 2018

better-hw-compatibility: fix out of bounds access

With --enable-better-hw-compatibility an access to array element -1
can be observed for VP9/ActiveMapTest.Test/0
../vp9/encoder/vp9_rdopt.c:3938:53: runtime error:
  index -1 out of bounds for type 'RefBuffer [3]'

There doesn't seem anything that would prevent ref_frame from being 0.
If there is no reference frame it can probably be assumed that it
isn't scaled.

Signed-off-by: Matthias Räncker <theonetruecamper@gmx.de>
Change-Id: I0a29cd0ffc9a19742e5e72203d5ec5d0a16eac7a

--- a/vp9/encoder/vp9_rdopt.c

+++ b/vp9/encoder/vp9_rdopt.c

@@ -3935,7 +3935,8 @@

 #if CONFIG_BETTER_HW_COMPATIBILITY

     // forbid 8X4 and 4X8 partitions if any reference frame is scaled.

     if (bsize == BLOCK_8X4 || bsize == BLOCK_4X8) {

-      int ref_scaled = vp9_is_scaled(&cm->frame_refs[ref_frame - 1].sf);

+      int ref_scaled = ref_frame > INTRA_FRAME &&

+                       vp9_is_scaled(&cm->frame_refs[ref_frame - 1].sf);

       if (second_ref_frame > INTRA_FRAME)

         ref_scaled += vp9_is_scaled(&cm->frame_refs[second_ref_frame - 1].sf);

       if (ref_scaled) continue;