shithub: libvpx

Info  •  Files  •  Log  •  Branches

vp9_quantize_ssse3_x86_64: fix out of bounds write

eob is a pointer to a uint16_t. previously the code would store 64-bits
causing a crash or test failure with the right stack layout.

Change-Id: Ibd653baf323db114f2444951b9d8b00c596bf15a

--- a/vp9/encoder/x86/vp9_quantize_ssse3_x86_64.asm

+++ b/vp9/encoder/x86/vp9_quantize_ssse3_x86_64.asm

@@ -169,7 +169,7 @@

   pshuflw                         m7, m8, 0x1

   pmaxsw                          m8, m7

   pextrw                          r6, m8, 0

-  mov                           [r2], r6

+  mov                           [r2], r6w

   RET

 %endmacro