• Gerd Hoffmann's avatar
    drm/virtio: fix DRM_FORMAT_* handling · 42fd9e6c
    Gerd Hoffmann authored
    Use DRM_FORMAT_HOST_XRGB8888, so we are using the correct format code
    on bigendian machines.  Also set the quirk_addfb_prefer_host_byte_order
    mode_config bit so drm_mode_addfb() asks for the correct format code.
    
    Both DRM_FORMAT_* and VIRTIO_GPU_FORMAT_* are defined to be little
    endian, so using a different mapping on bigendian machines is wrong.
    It's there because of broken drm_mode_addfb() behavior.  So with
    drm_mode_addfb() being fixed we can fix this too.
    
    While wading through the code I've noticed we have a little issue in
    virtio:  We attach a format to the bo when it is created
    (DRM_IOCTL_MODE_CREATE_DUMB), not when we map it as framebuffer
    (DRM_IOCTL_MODE_ADDFB).  Easy way out:  Support a single format only.
    Pick DRM_FORMAT_HOST_XRGB8888, it is the only one actually used in
    practice.  Drop unused mappings in virtio_gpu_translate_format().
    
    With this patch applied both ADDFB and ADDFB2 ioctls work correctly in
    the virtio-gpu.ko driver on big endian machines.  Without the patch only
    ADDFB (which still seems to be used by the majority of userspace) works
    correctly.
    Signed-off-by: default avatarGerd Hoffmann <kraxel@redhat.com>
    Acked-by: default avatarDaniel Vetter <daniel.vetter@ffwll.ch>
    Link: http://patchwork.freedesktop.org/patch/msgid/20180921134704.12826-6-kraxel@redhat.com
    42fd9e6c
virtgpu_display.c 11.5 KB