shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix another motion vector out of range bug

This patch fixed a motion vector out of range bug:
vpxenc: ../libvpx/vp9/encoder/vp9_mcomp.c:69:
 mv_cost: Assertion `mv->col >= -((1 << (11 + 1 + 2)) - 1) &&
 mv->col < ((1 << (11 + 1 + 2)) - 1)' failed.

For blocks that returned without having full-pixel search, the original
MV limits were not restored, which caused the failure. Moved the set
MV limit function down to fix the bug.

Change-Id: Id7d798fc7214e95c6e4846c588f0233fcf1a4223

--- a/vp9/encoder/vp9_rdopt.c

+++ b/vp9/encoder/vp9_rdopt.c

@@ -2347,8 +2347,6 @@

     vp9_setup_pre_planes(xd, 0, scaled_ref_frame, mi_row, mi_col, NULL);

   }

-  vp9_set_mv_search_range(&x->mv_limits, &ref_mv);

-

   // Work out the size of the first step in the mv step search.

   // 0 here is maximum length first step. 1 is VPXMAX >> 1 etc.

   if (cpi->sf.mv.auto_mv_step_size && cm->show_frame) {

@@ -2395,6 +2393,10 @@

       }

     }

   }

+

+  // Note: MV limits are modified here. Always restore the original values

+  // after full-pixel motion search.

+  vp9_set_mv_search_range(&x->mv_limits, &ref_mv);

   mvp_full = pred_mv[x->mv_best_ref_index[ref]];