ref: 984734436d8f13fc44ca3f0f3d65e0d057d6fd20
parent: a879b4e6d425e7ec26e6ed9f0ef3e84575a891d4
author: James Zern <jzern@google.com>
date: Tue Nov 6 11:58:11 EST 2012
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f