media: atomisp: Fix VIDIOC_REQBUFS failing when VIDIOC_S_FMT has not been called yet

camorama calls VIDIOC_REQBUFS(V4L2_MEMORY_MMAP) to test if MMAP support
works (this was added specifically because of the previously broken
MMAP support in atomisp).

Currently this fails because atomisp_get_css_frame_info() fails in this
case. Although it is weird to call VIDIOC_REQBUFS before VIDIOC_S_FMT,
it is allowed to do this.

Fix this not working by doing a S_FMT to V4L2_PIX_FMT_YUV420 + the highest
supported resolution.

Note this will cause camorama to use mmap mode, which means it will also
use libv4l2 to do format conversion. libv4l2 will pick V4L2_PIX_FMT_RGB565
as input format and this will lead to a garbled video display. This is
a libv4lconvert bug, the RGB565 -> RGB24 path in libv4lconvert assumes
that stride == width which is not true on the atomisp.

I've already send out a libv4lconvert fix for this. Also this can be worked
around by passing --dont-use-libv4l2 as argument to camorama.

Link: https://git.linuxtv.org/v4l-utils.git/commit/?id=aecfcfccfc2f78d7531456ffa5465666c6bc641eReviewed-by: Andy Shevchenko <andy@kernel.org>
Signed-off-by: Hans de Goede <hdegoede@redhat.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>

drivers/staging/media/atomisp/pci/atomisp_compat_css20.c

@@ -2688,7 +2688,7 @@ int atomisp_get_css_frame_info(struct atomisp_sub_device *asd,
 
 	if (0 != ia_css_pipe_get_info(asd->stream_env[stream_index]
 		.pipes[pipe_index], &info)) {
-		dev_err(isp->dev, "ia_css_pipe_get_info FAILED");
+		dev_dbg(isp->dev, "ia_css_pipe_get_info FAILED");
 		return -EINVAL;
 	}
 

drivers/staging/media/atomisp/pci/atomisp_fops.c

@@ -50,15 +50,36 @@ static int atomisp_queue_setup(struct vb2_queue *vq,
 	u16 source_pad = atomisp_subdev_source_pad(&pipe->vdev);
 	int ret;
 
+	mutex_lock(&pipe->asd->isp->mutex); /* for get_css_frame_info() / set_fmt() */
+
+	/*
+	 * When VIDIOC_S_FMT has not been called before VIDIOC_REQBUFS, then
+	 * this will fail. Call atomisp_set_fmt() ourselves and try again.
+	 */
+	ret = atomisp_get_css_frame_info(pipe->asd, source_pad, &pipe->frame_info);
+	if (ret) {
+		struct v4l2_format f = {
+			.fmt.pix.pixelformat = V4L2_PIX_FMT_YUV420,
+			.fmt.pix.width = 10000,
+			.fmt.pix.height = 10000,
+		};
+
+		ret = atomisp_set_fmt(&pipe->vdev, &f);
+		if (ret)
+			goto out;
+
 		ret = atomisp_get_css_frame_info(pipe->asd, source_pad, &pipe->frame_info);
 		if (ret)
-		return ret;
+			goto out;
+	}
 
 	atomisp_alloc_css_stat_bufs(pipe->asd, ATOMISP_INPUT_STREAM_GENERAL);
 
 	*nplanes = 1;
 	sizes[0] = PAGE_ALIGN(pipe->pix.sizeimage);
 
+out:
+	mutex_unlock(&pipe->asd->isp->mutex);
 	return 0;
 }