shithub: dav1d

Info  •  Files  •  Log  •  Branches

av1: use chroma txtp inference over default DCT_DCT if qidx=0

Fixes #320. The problem here is that qidx=0 is (in libaom) a shortcut
for lossless, which normally becomes WHT_WHT, but under some obscure
conditions, it can also be non-lossless, in which case qidx=0 implies
DCT_DCT for luma. For chroma, apparently we should use the default
inference pattern, which becomes DCT_DCT for inter also, but requires
the standard lookup table for intra.

--- a/src/recon_tmpl.c

+++ b/src/recon_tmpl.c

@@ -352,13 +352,17 @@

     if (lossless) {

         assert(t_dim->max == TX_4X4);

         *txtp = WHT_WHT;

-    } else if (!f->frame_hdr->segmentation.qidx[b->seg_id] ||

-               t_dim->max + intra >= TX_64X64)

-    {

+    } else if (t_dim->max + intra >= TX_64X64) {

         *txtp = DCT_DCT;

     } else if (chroma) {

+        // inferred from either the luma txtp (inter) or a LUT (intra)

         *txtp = intra ? dav1d_txtp_from_uvmode[b->uv_mode] :

                         get_uv_inter_txtp(t_dim, *txtp);

+    } else if (!f->frame_hdr->segmentation.qidx[b->seg_id]) {

+        // In libaom, lossless is checked by a literal qidx == 0, but not all

+        // such blocks are actually lossless. The remainder gets an implicit

+        // transform type (for luma)

+        *txtp = DCT_DCT;

     } else {

         unsigned idx;

         if (intra) {