- 08 Jan, 2021 7 commits
-
-
Chris Wilson authored
If a request is submitted and known to require no preemption, disable arbitration around the batch which prevents the HW from handling a preemption request during the payload. Signed-off-by:
Chris Wilson <chris@chris-wilson.co.uk> Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Lionel Landwerlin <lionel.g.landwerlin@intel.com> Reviewed-by:
Andi Shyti <andi.shyti@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210108204026.20682-7-chris@chris-wilson.co.uk
-
Chris Wilson authored
The reason why we did not enable preemption on Broadwater was due to missing GPGPU workarounds. Since this only applies to rcs0, only restrict rcs0 (and our global capabilities). While this does not affect exposing a preemption capability to userspace, it does affect our internal decisions on whether to use timeslicing and semaphores between individual engines. Signed-off-by:
-
Chris Wilson authored
We use the completion of the last active breadcrumb to retire the requests along a timeline. This is purely opportunistic as nothing guarantees that any particular timeline is terminated by a breadcrumb; except for parking the engine where we explicitly add a breadcrumb so that we park quickly and do an explicit retire upon signaling to reduce the latency dramatically (avoiding a retire worker roundtrip). With scheduling, we anticipate retiring completed timelines as a matter of course. Performing the same action from inside the breadcrumbs is intended to provide similar functionality for legacy ringbuffer submission. Signed-off-by:
-
Chris Wilson authored
Before we mark the virtual engine as no longer inflight, flush any ongoing signaling that may be using the ce->signal_link along the previous breadcrumbs. On switch to a new physical engine, that link will be inserted into the new set of breadcrumbs, causing confusion to an ongoing iterator. This patch undoes a last minute mistake introduced into commit bab0557c ("drm/i915/gt: Remove virtual breadcrumb before transfer"), whereby instead of unconditionally applying the flush, it was only applied if the request itself was going to be reused. v2: Generalise and cancel all remaining ce->signals Fixes: bab0557c ("drm/i915/gt: Remove virtual breadcrumb before transfer") Signed-off-by:
-
Chris Wilson authored
In our tests where we measure the elapsed time on both the CPU and CS using a udelay, our CS results match the udelay much more accurately than the ktime (even when using ktime_get_fast_ns). With preemption disabled, we can go one step lower than ktime and use local_clock. Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2919Signed-off-by:
-
Chris Wilson authored
If any of the perf tests run into 0 time, not only are we liable to divide by zero, but the result would be highly questionable. Nevertheless, let's not have a div-by-zero error. Signed-off-by:
-
Chris Wilson authored
On error we unpin and free the wa_ctx.vma, but do not clear any of the derived flags. During lrc_init, we look at the flags and attempt to dereference the wa_ctx.vma if they are set. To protect the error path where we try to limp along without the wa_ctx, make sure we clear those flags! Reported-by:
Matt Roper <matthew.d.roper@intel.com> Fixes: 604a8f6f ("drm/i915/lrc: Only enable per-context and per-bb buffers if set") Signed-off-by:
-
- 07 Jan, 2021 2 commits
-
-
Chris Wilson authored
In the next^W future patch, we remove the strict priority system and continuously re-evaluate the relative priority of tasks. As such we need to enable the timeslice whenever there is more than one context in the pipeline. This simplifies the decision and removes some of the tweaks to suppress timeslicing, allowing us to lift the timeslice enabling to a common spot at the end of running the submission tasklet. One consequence of the suppression is that it was reducing fairness between virtual engines on an over saturated system; undermining the principle for timeslicing. v2: Commentary v3: Commentary for the right cancel_timer() v4: Add tracing for why we need a timeslice Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2802 Testcase: igt/gem_exec_balancer/fairslice Signed-off-by:
Tvrtko Ursulin <tvrtko.ursulin@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210107132322.28373-1-chris@chris-wilson.co.uk
-
Chris Wilson authored
Refactor our timer_list.expires checking into its own timer_active() helper. Signed-off-by:
-
- 06 Jan, 2021 2 commits
-
-
Chris Wilson authored
Use memset_io() on the iomem, and silence sparse as we copy from the iomem to normal system pages. Signed-off-by:
-
Chris Wilson authored
AFter detecting a register mismatch between the protocontext and the image generated by HW, immediately break out of the double loop. Otherwise we end up with a second confusing error message. Signed-off-by:
-
- 05 Jan, 2021 5 commits
-
-
Chris Wilson authored
In generating the reference LRC, we want a page-aligned address for simplicity in computing the offsets within. This then shares the computation for the HW LRC which is mapped and so page aligned, making the comparison straightforward. It seems that kmalloc(4k) is not always returning from a 4k-aligned slab cache (which would give us a page aligned address) so force alignment by explicitly allocating a page. Reported-by:
"Gote, Nitin R" <nitin.r.gote@intel.com> Signed-off-by:
Matthew Auld <matthew.auld@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20200923114156.17749-1-chris@chris-wilson.co.uk
-
Chris Wilson authored
If another sibling is able to claim the virtual request, by the time we inspect the request under the lock it may no longer match the local engine. Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2877Signed-off-by:
-
Chris Wilson authored
If the engine reset fails, we will attempt to resume with the current inflight submissions. When that happens, we cannot assert that the engine reset cleared the pending submission, so do not. Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2878 Fixes: 16f2941a ("drm/i915/gt: Replace direct submit with direct call to tasklet") Signed-off-by:
-
Chris Wilson authored
Older platforms use rawclk to derive the CS clock. rawclk is being determined during intel_device_info_init(), and so that needs to be pushed slightly earlier. Fixes: f170523a ("drm/i915/gt: Consolidate the CS timestamp clocks") Signed-off-by:
-
Chris Wilson authored
A few missed PTR_ERR() upon create_gang() errors. Signed-off-by:
-
- 04 Jan, 2021 3 commits
-
-
Chris Wilson authored
In the near future, upstream will introduce a SZ_8G macro that is slightly different to our own. Employ a temporary ifndef to avoid compilation failure until we have backmerged. References: 8b0fac44 ("sizes.h: add SZ_8G/SZ_16G/SZ_32G macros") Signed-off-by:
José Roberto de Souza <jose.souza@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210104171511.32684-1-chris@chris-wilson.co.uk
-
Chris Wilson authored
Some rcs0 workarounds were being incorrectly applied to the GT, and so we failed to restore the expected register settings after a reset. Signed-off-by:
Mika Kuoppala <mika.kuoppala@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210104114914.30165-2-chris@chris-wilson.co.uk
-
Chris Wilson authored
Some rcs0 workarounds were being incorrectly applied to the GT, and so we failed to restore the expected register settings after a reset. Signed-off-by:
-
- 03 Jan, 2021 1 commit
-
-
Arnd Bergmann authored
Randconfig builds on 32-bit machines show lots of warnings for the i915 driver for passing a 32-bit value into __const_hweight64(): drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c:2584:9: error: shift count >= width of type [-Werror,-Wshift-count-overflow] return hweight64(VDBOX_MASK(&i915->gt)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ include/asm-generic/bitops/const_hweight.h:29:49: note: expanded from macro 'hweight64' #define hweight64(w) (__builtin_constant_p(w) ? __const_hweight64(w) : __arch_hweight64(w)) Change it to hweight_long() to avoid the warning. Signed-off-by:Arnd Bergmann <arnd@arndb.de> Reviewed-by:
-
- 01 Jan, 2021 2 commits
-
-
Maarten Lankhorst authored
This allows us to remove pin_map from state allocation, which saves us a few retry loops. We won't need this until first pin, anyway. Signed-off-by:
Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Reviewed-by:
Thomas Hellström <thomas.hellstrom@linux.intel.com> Signed-off-by:
-
Matt Roper authored
Let's modify the "workaround lost" error message slightly to make it more clear what the various numbers represent. Also, the 'expected' value needs to be &'d with wa->read so that it doesn't include the mask bits for masked registers (those bits are write-only in the hardware and will usually always read out as 0's). Signed-off-by:
-
- 31 Dec, 2020 3 commits
-
-
Chris Wilson authored
Since we use a flag within i915_request.flags to indicate when we have boosted the request (so that we only apply the boost) once, this can be used as the serialisation with i915_request_retire() to avoid having to explicitly take the i915_request.lock which is more heavily contended. Signed-off-by:
-
Chris Wilson authored
We only need to evaluate the current status of the context when it is scheduled in, we will force a reschedule when the context is closed propagating the change to inflight contexts. Signed-off-by:
-
Chris Wilson authored
Since we process schedule-in of a context after submitting the request, if we decide to reset the context at that time, we also have to cancel the requets we have marked for submission. Signed-off-by:
-
- 30 Dec, 2020 1 commit
-
-
Chris Wilson authored
Declare that, under extreme circumstances, the shrinker may need to wait upon a request, in which case reset must not itself deadlock in order to ensure forward progress of the driver. That is since the shrinker may depend upon a reset, any reset cannot touch the shrinker. Signed-off-by:
-
- 29 Dec, 2020 2 commits
-
-
Chris Wilson authored
If supported by the backend, we can quickly look at the context's inflight engine rather than search along the active list to confirm. Signed-off-by:
-
Chris Wilson authored
When splitting the Coffeelake define to also identify Cometlakes, I missed the double fw_def for Coffeelake. That is only newer Cometlakes use the cml specific guc firmware, older Cometlakes should use kbl firmware. Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2859 Fixes: 5f4ae270 ("drm/i915: Identify Cometlake platform") Signed-off-by:
-
- 24 Dec, 2020 12 commits
-
-
Chris Wilson authored
Pull the individual strands of creating a custom heartbeat requests into a pair of common functions. This will reduce the number of changes we will need to make in future. Signed-off-by:
-
Matthew Auld authored
The reloc batch is short lived but can exist in the user visible ppGTT, and since it's backed by an internal object, which lacks page clearing, we should take care to clear it upfront. Signed-off-by:
-
Matthew Auld authored
The shadow batch is an internal object, which doesn't have any page clearing, and since the batch_len can be smaller than the object, we should take care to clear it. Testcase: igt/gen9_exec_parse/shadow-peek Fixes: 4f7af194 ("drm/i915: Support ro ppgtt mapped cmdparser shadow buffers") Signed-off-by:
-
Chris Wilson authored
Since schedule-in and schedule-out are now both always under the tasklet bitlock, we can reduce the individual atomic operations to simple instructions and worry less. This notably eliminates the race observed with intel_context_inflight in __engine_unpark(). Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2583Signed-off-by:
-
Chris Wilson authored
Now that the tasklet completely controls scheduling of the requests, and we postpone scheduling out the old requests, we can keep a hanging virtual request bound to the engine on which it hung, and remove it from te queue. On release, it will be returned to the same engine and remain in its queue until it is scheduled; after which point it will become eligible for transfer to a sibling. Instead, we could opt to resubmit the request along the virtual engine on unhold, making it eligible for load balancing immediately -- but that seems like a pointless optimisation for a hanging context. Signed-off-by:
-
Chris Wilson authored
Having recognised that we do not change the sibling until we schedule out, we can then defer the decision to resubmit the virtual engine from the unwind of the active queue to scheduling out of the virtual context. This improves our resilence in virtual engine scheduling, and should eliminate the rare cases of gem_exec_balance failing. By keeping the unwind order intact on the local engine, we can preserve data dependency ordering while doing a preempt-to-busy pass until we have determined the new ELSP. This means that if we try to timeslice between a virtual engine and a data-dependent ordinary request, the pair will maintain their relative ordering and we will avoid the resubmission, cancelling the timeslicing until further change. The dilemma though is that we then may end up in a situation where the 'demotion' of the virtual request to an ordinary request in the engine queue results in filling the ELSP[] with virtual requests instead of spreading the load across the engines. To compensate for this, we mark each virtual request and refuse to resubmit a virtual request in the secondary ELSP slots, thus forcing subsequent virtual requests to be scheduled out after timeslicing. By delaying the decision until we schedule out, we will avoid unnecessary resubmission. Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2079 Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/2098Signed-off-by:
-
Chris Wilson authored
Let's only wait for the list iterator when decoupling the virtual breadcrumb, as the signaling of all the requests may take a long time, during which we do not want to keep the tasklet spinning. Signed-off-by:
Matthew Brost <matthew.brost@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20201224135544.1713-6-chris@chris-wilson.co.uk
-
Chris Wilson authored
The issue with stale virtual breadcrumbs remain. Now we have the problem that if the irq-signaler is still referencing the stale breadcrumb as we transfer it to a new sibling, the list becomes spaghetti. This is a very small window, but that doesn't stop it being hit infrequently. To prevent the lists being tangled (the iterator starting on one engine's b->signalers but walking onto another list), always decouple the virtual breadcrumb on schedule-out and make sure that the walker has stepped out of the lists. Signed-off-by:
-
Chris Wilson authored
Inside schedule_out, we do extra work upon idling the context, such as updating the runtime, kicking off retires, kicking virtual engines. However, if we are in a series of processing single requests per contexts, we may find ourselves scheduling out the context, only to immediately schedule it back in during dequeue. This is just extra work that we can avoid if we keep the context marked as inflight across the dequeue. This becomes more significant later on for minimising virtual engine misses. Signed-off-by:
-
Chris Wilson authored
Once a virtual engine has been bound to a sibling, it will remain bound until we finally schedule out the last active request. We can not rebind the context to a new sibling while it is inflight as the context save will conflict, hence we wait. As we cannot then use any other sibliing while the context is inflight, only kick the bound sibling while it inflight and upon scheduling out the kick the rest (so that we can swap engines on timeslicing if the previously bound engine becomes oversubscribed). Signed-off-by:
-
Chris Wilson authored
Rather than going back and forth between the rb_node entry and the virtual_engine type, store the ve local and reuse it. As the container_of conversion from rb_node to virtual_engine requires a variable offset, performing that conversion just once shaves off a bit of code. v2: Keep a single virtual engine lookup, for typical use. Signed-off-by:
-
Chris Wilson authored
Rather than having special case code for opportunistically calling process_csb() and performing a direct submit while holding the engine spinlock for submitting the request, simply call the tasklet directly. This allows us to retain the direct submission path, including the CS draining to allow fast/immediate submissions, without requiring any duplicated code paths, and most importantly greatly simplifying the control flow by removing reentrancy. This will enable us to close a few races in the virtual engines in the next few patches. The trickiest part here is to ensure that paired operations (such as schedule_in/schedule_out) remain under consistent locking domains, e.g. when pulled outside of the engine->active.lock v2: Use bh kicking, see commit 3c53776e ("Mark HI and TASKLET softirq synchronous"). v3: Update engine-reset to be tasklet aware Signed-off-by:
-
