shithub: libvpx

Download patch

ref: 76550dfdc0a898cb9510977f79b43dbac1fd9de0
parent: 945ccfee59fe2d1bde5fc3f30845a6e0503bec59
author: paulwilkins <paulwilkins@google.com>
date: Thu Feb 9 11:30:38 EST 2017

Bug in scale_sse_threshold()

The function scale_sse_threshold() returns a threshold scaled
if necessary for use with 10 and 12 bit from an 8 bit baseline.

SSE error values would be expected to rise for the 10 and 12
bit cases where there are more bits of precision.

Hence the threshold used for the test should also be scaled up.

Change-Id: I4009c98b6eecd1bf64c3c38aaa56598e0136b03d

--- a/vp9/encoder/vp9_firstpass.c

+++ b/vp9/encoder/vp9_firstpass.c

@@ -469,8 +469,8 @@

   if (cm->use_highbitdepth) {

     switch (cm->bit_depth) {

       case VPX_BITS_8: ret_val = thresh; break;

-      case VPX_BITS_10: ret_val = thresh >> 4; break;

-      case VPX_BITS_12: ret_val = thresh >> 8; break;

+      case VPX_BITS_10: ret_val = thresh << 4; break;

+      case VPX_BITS_12: ret_val = thresh << 8; break;

       default:

         assert(0 &&

                "cm->bit_depth should be VPX_BITS_8, "