shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix AQ mode 2 bug where delta causes Q 0.

In Aq mode 2 for kf/arf/gf the segment q delta
is calculated and then applied by re-quantization without
going through the rd loop again. If the base Q != 0
but the segment Q == 0 (lossless) this can could give rise
to a situation where we have an illegal combination of
transform size and Q. (Q == 0 requires that all blocks
are coded 4x4 WHT).

Change-Id: I241a58c6494ed442e9e4630070b0cde0fb99ae45

--- a/vp9/encoder/vp9_aq_complexity.c

+++ b/vp9/encoder/vp9_aq_complexity.c

@@ -47,11 +47,21 @@

     // Use some of the segments for in frame Q adjustment.

     for (segment = 1; segment < 2; segment++) {

-      const int qindex_delta =

+      int qindex_delta =

           vp9_compute_qdelta_by_rate(&cpi->rc, cm->frame_type, cm->base_qindex,

                                      in_frame_q_adj_ratio[segment]);

-      vp9_enable_segfeature(seg, segment, SEG_LVL_ALT_Q);

-      vp9_set_segdata(seg, segment, SEG_LVL_ALT_Q, qindex_delta);

+

+      // For AQ mode 2, we dont allow Q0 in a segment if the base Q is not 0.

+      // Q0 (lossless) implies 4x4 only and in AQ mode 2 a segment Q delta

+      // is sometimes applied without going back around the rd loop.

+      // This could lead to an illegal combination of partition size and q.

+      if ((cm->base_qindex != 0) && ((cm->base_qindex + qindex_delta) == 0)) {

+        qindex_delta = -cm->base_qindex + 1;

+      }

+      if ((cm->base_qindex + qindex_delta) > 0) {

+        vp9_enable_segfeature(seg, segment, SEG_LVL_ALT_Q);

+        vp9_set_segdata(seg, segment, SEG_LVL_ALT_Q, qindex_delta);

+      }

     }

   }

 }