• Monk Liu's avatar
    drm/amdgpu/SRIOV:implement guilty job TDR for(V2) · 65781c78
    Monk Liu authored
    1,TDR will kickout guilty job if it hang exceed the threshold
    of the given one from kernel paramter "job_hang_limit", that
    way a bad command stream will not infinitly cause GPU hang.
    
    by default this threshold is 1 so a job will be kicked out
    after it hang.
    
    2,if a job timeout TDR routine will not reset all sched/ring,
    instead if will only reset on the givn one which is indicated
    by @job of amdgpu_sriov_gpu_reset, that way we don't need to
    reset and recover each sched/ring if we already know which job
    cause GPU hang.
    
    3,unblock sriov_gpu_reset for AI family.
    
    V2:
    1:put kickout guilty job after sched parked.
    2:since parking scheduler prior to kickout already occupies a
    while, we can do last check on the in question job before
    doing hw_reset.
    
    TODO:
    1:when a job is considered as guilty, we should mark some flag
    in its fence status flag, and let UMD side aware that this
    fence signaling is not due to job complete but job hang.
    
    2:if gpu reset cause all video memory lost, we need introduce
    a new policy to implement TDR, like drop all jobs not yet
    signaled, and all IOCTL on this device will return ERROR
    DEVICE_LOST.
    this will be implemented later.
    Signed-off-by: default avatarMonk Liu <Monk.Liu@amd.com>
    Reviewed-by: default avatarChristian König <christian.koenig@amd.com>
    Signed-off-by: default avatarAlex Deucher <alexander.deucher@amd.com>
    65781c78
amdgpu_fence.c 17.9 KB