• Xi Wang's avatar
    vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create() · 8a783896
    Xi Wang authored
    Commit e133e737 didn't correctly fix the integer overflow issue.
    
    -	unsigned int required_size;
    +	u64 required_size;
    	...
    	required_size = mode_cmd->pitch * mode_cmd->height;
    -	if (unlikely(required_size > dev_priv->vram_size)) {
    +	if (unlikely(required_size > (u64) dev_priv->vram_size)) {
    
    Note that both pitch and height are u32.  Their product is still u32 and
    would overflow before being assigned to required_size.  A correct way is
    to convert pitch and height to u64 before the multiplication.
    
    	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;
    
    This patch calls the existing vmw_kms_validate_mode_vram() for
    validation.
    Signed-off-by: default avatarXi Wang <xi.wang@gmail.com>
    Reviewed-and-tested-by: default avatarThomas Hellstrom <thellstrom@vmware.com>
    Signed-off-by: default avatarDave Airlie <airlied@redhat.com>
    8a783896
vmwgfx_kms.c 51.9 KB