Commit Graph

549 Commits

Author SHA1 Message Date
Kamil Kopryk
56b30d1803 feature: redesign host function workers
Each host function gets its unique ID within a CSR,
uses 1 mi store to write ID - to signal that host function is ready,
and 1 mi semaphore wait will wait for the ID to be cleared,
Use 0th bit from ID as pending/completed flag,
host function ID is incremented by 2, and starts with 1.
So each ID will always have 0bit set.
This is a must have since semaphore wait can wait for 4 bytes only.

Adjust command buffer programming and patching logic to IDs.

Add hostFunction callable class - using invoke method,
which stores required information about callback.

Add host function streamer - stores all host function data
for a given CSR.
All user provided host functions are stored in unordered map,
where key is host function ID.

Add host function scheduler, and a thread pool - under debug flag
Single threaded scheduler loops over all registered host function streamers,
dispatch ready to execute host functions to thread pool.

Allow for out of order host functions execution for OOQ - under debug flag,
each host function has bool isInOrder flag which indicates if it can be
executed Out Of Order - in this mode, ID tag will be cleared immediately,
so semaphore wait will unblock before the host function execution.

Remove Host Function worker CV and atomics based implementation.

Rename classes

Related-To: NEO-14577
Signed-off-by: Kamil Kopryk <kamil.kopryk@intel.com>
2025-11-25 08:05:41 +01:00
Mateusz Hoppe
466f654308 fix: move waitWithKmdNotifyFallback under debug key
Related-To: NEO-2024

Signed-off-by: Mateusz Hoppe <mateusz.hoppe@intel.com>
2025-11-20 09:51:23 +01:00
Mateusz Jablonski
8db92240fc refactor: add wrapper for max product enum value
Signed-off-by: Mateusz Jablonski <mateusz.jablonski@intel.com>
2025-11-20 08:04:57 +01:00
Maciej Plewka
5f819f8212 fix: do not wait with locked csr
Related-To: NEO-16532, HSD-18043732674
Signed-off-by: Maciej Plewka <maciej.plewka@intel.com>
2025-11-13 13:44:39 +01:00
Kamil Kopryk
8757ecf2f7 refactor: reuse tag allocation for host function data
Related-To: NEO-14577
Signed-off-by: Kamil Kopryk <kamil.kopryk@intel.com>
2025-11-05 14:51:20 +01:00
Lukasz Jobczyk
01885fe362 fix: proper lock order when reinitialize context
Signed-off-by: Lukasz Jobczyk <lukasz.jobczyk@intel.com>
2025-11-05 09:51:37 +01:00
Lukasz Jobczyk
498f62d7a0 fix: Reset direct submission when reinitialize context
Resolves: HSD-15018564496
Related-To: NEO-16651

Signed-off-by: Lukasz Jobczyk <lukasz.jobczyk@intel.com>
2025-11-04 13:32:36 +01:00
Kamil Kopryk
f84a5fbee9 feature: add host functions workers
* add common host function worker interface
* add worker as a single thread per csr with 3 modes
* add logic for waiting on internal tag, check gpu hang
* if tag is in pending state, read callback data, run callback
and signal completion
* threads will exit the work loop once stop request
is called in finish
* add multi thread unit tests

Related-To: NEO-14577
Signed-off-by: Kamil Kopryk <kamil.kopryk@intel.com>
2025-11-03 12:11:17 +01:00
Chandio, Bibrak Qamar
421b1861e5 fix: madvise order and reset
Related-To: NEO-16173, NEO-13715

Signed-off-by: Chandio, Bibrak Qamar <bibrak.qamar.chandio@intel.com>
2025-10-27 23:37:57 +01:00
Zbigniew Zdanowicz
17b92df5fb performance: small tweaks of patch preamble encoding
Related-To: NEO-16434

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-10-23 17:21:09 +02:00
Zbigniew Zdanowicz
9474d455b1 refactor: remove not needed patch preamble function
Related-To: NEO-16434

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-10-23 11:23:34 +02:00
Szymon Morek
0696340d3d refactor: remove unused code
Signed-off-by: Szymon Morek <szymon.morek@intel.com>
2025-10-20 08:02:23 +02:00
Zbigniew Zdanowicz
8c522e3097 performance: noop only reminder of patch preamble
Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-10-15 14:53:15 +02:00
Naklicki, Mateusz
0c619299d0 refactor: cleanup header includes in level_zero
Related-To: NEO-15383
Signed-off-by: Naklicki, Mateusz <mateusz.naklicki@intel.com>
2025-10-07 19:58:31 +02:00
Kamil Kopryk
16050126e3 feature: add host function dispatch
- program in place for immediate cmdlist
- add to patch commands for regular cmdlist
- initialize host data allocation for the 1st use in csr

Related-To: NEO-14577
Signed-off-by: Kamil Kopryk <kamil.kopryk@intel.com>
2025-09-24 17:20:40 +02:00
Jaroslaw Warchulski
195bf66a49 refactor: fix typos
Signed-off-by: Jaroslaw Warchulski <jaroslaw.warchulski@intel.com>
2025-09-22 12:46:41 +02:00
Zbigniew Zdanowicz
0b65b86ccb feature: synchronize patch preamble with wait commands before patching
Related-To: NEO-16140

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-09-19 11:21:22 +02:00
Zbigniew Zdanowicz
3380e10492 feature: add saving tag address and task count into command list when required
Related-To: NEO-16140

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-09-18 14:44:10 +02:00
Zbigniew Zdanowicz
138d76a893 feature: add flag to save task count and tag address in command list
Related-To: NEO-16140

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-09-17 16:17:57 +02:00
Slawomir Milczarek
a5f08b5f42 feature: Make priority level in priority extension truly optional
Related-To: NEO-16102

Signed-off-by: Slawomir Milczarek <slawomir.milczarek@intel.com>
2025-09-16 09:34:08 +02:00
Zbigniew Zdanowicz
d8557ff2d3 feature: add scratch address programing to patch preamble
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-19 00:41:01 +02:00
Zbigniew Zdanowicz
216e7a593f feature: add patch preamble support to immediate command lists
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-13 12:56:51 +02:00
Mateusz Jablonski
22e7aa36d1 refactor: remove pre-gen12 code
Signed-off-by: Mateusz Jablonski <mateusz.jablonski@intel.com>
2025-08-11 20:29:37 +02:00
Zbigniew Zdanowicz
ab98dad17e feature: add noop in order info for regular command list to patch preamble
- sum total noop size requirement for both in order and noop patchlist

Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-11 17:30:25 +02:00
Zbigniew Zdanowicz
092f2bb333 feature: get command list information for in order noop data
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-08 15:01:52 +02:00
Zbigniew Zdanowicz
da2fc7a586 feature: add noop support to patch preamble
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-07 16:23:43 +02:00
Bartosz Dunajski
b95ba7cf05 fix: add poll for aub completion on queue synchronize
Signed-off-by: Bartosz Dunajski <bartosz.dunajski@intel.com>
2025-08-06 14:03:30 +02:00
Zbigniew Zdanowicz
b0e9267e62 feature: add support encodng front end command to patch preamble
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-05 12:58:29 +02:00
Zbigniew Zdanowicz
841267ecbd feature: save command buffer gpu address for front end command in command list
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-05 09:59:08 +02:00
Zbigniew Zdanowicz
01889c97a5 feature: add patch preamble to level zero queue handling bb_start commands
Related-To: NEO-15376

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-08-01 14:50:56 +02:00
Mateusz Jablonski
05cf05e10e refactor: remove redundant tracking of media sampler programming
Signed-off-by: Mateusz Jablonski <mateusz.jablonski@intel.com>
2025-07-28 15:17:46 +02:00
Neil R. Spruit
2f6e4d36da fix: Correct all L0 handle definitions to properly handle the DDI ext
Related-To: NEO-15570

- Fixes for missing handle definitions for DDi Ext compliance and
incorrect definitions for several handle definitions requiring handle
translation.

Signed-off-by: Neil R. Spruit <neil.r.spruit@intel.com>
2025-07-17 19:13:51 +02:00
Tomasz Biernacik
2c5cbec033 feature: control post sync completion check
Related-To: NEO-14844

Signed-off-by: Tomasz Biernacik <tomasz.biernacik@intel.com>
2025-07-15 13:51:19 +02:00
Mateusz Jablonski
762b04cf77 feature: add command queue flag to pass copy offload hint
Related-To: NEO-14560
Signed-off-by: Mateusz Jablonski <mateusz.jablonski@intel.com>
2025-07-11 19:42:48 +02:00
Zbigniew Zdanowicz
fc37b98b69 fix: provide fixes for kernel isa mutation of cooperative kernels
- update correctly mutable residency of cooperative kernel barrier allocation
- fix disable noop patching when mutate into non cooperative kernel
- reset mutation variables for not default kernels in mutable kernel group

Related-To: NEO-10492

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-07-09 16:22:00 +02:00
Bartosz Dunajski
fba837a7ca fix: disable in-order counter patching
Signed-off-by: Bartosz Dunajski <bartosz.dunajski@intel.com>
2025-07-03 07:47:02 +02:00
Naklicki, Mateusz
bd80531dea fix: allow scratch to be optional
Related-To: NEO-14130

Signed-off-by: Naklicki, Mateusz <mateusz.naklicki@intel.com>
2025-06-24 11:15:04 +02:00
Chodor, Jaroslaw
46c60290b9 refactor: introducing igfxfmid_wrapper
Signed-off-by: Chodor, Jaroslaw <jaroslaw.chodor@intel.com>
2025-05-30 19:26:04 +02:00
Lukasz Jobczyk
332340b02b refactor: Prework for release fence removal leaving acquire fence
Related-To: NEO-14642

Signed-off-by: Lukasz Jobczyk <lukasz.jobczyk@intel.com>
2025-05-30 12:01:28 +02:00
Maciej Plewka
e6f3ebce5d fix: add pc with stall before barrier with post sync on bmg
Related-To: NEO-14491
Signed-off-by: Maciej Plewka <maciej.plewka@intel.com>
2025-05-28 09:24:21 +02:00
Lukasz Jobczyk
86e7d5b276 performance: Skip fence after PC when no post sync write
Related-To: NEO-14642

Signed-off-by: Lukasz Jobczyk <lukasz.jobczyk@intel.com>
2025-05-23 10:33:28 +02:00
Mateusz Hoppe
0d57dcfdf0 feature: add L0 API querying priority levels - zeDeviceGetPriorityLevels()
Related-To: NEO-14572

Signed-off-by: Mateusz Hoppe <mateusz.hoppe@intel.com>
2025-05-23 09:24:16 +02:00
Szymon Morek
bff8f74de7 refactor: remove unused parameter
Signed-off-by: Szymon Morek <szymon.morek@intel.com>
2025-05-15 12:23:27 +02:00
Lukasz Jobczyk
1d1414febc refactor: remove unused dc flush mitigation
Signed-off-by: Lukasz Jobczyk <lukasz.jobczyk@intel.com>
2025-05-09 08:26:26 +02:00
Mateusz Jablonski
0dd2535674 refactor: change type of stype member in L0 Core driver extensions
use uint32_t-alias type to avoid casting out of ze_structure_type_t enum range
Related-To: NEO-12901

Signed-off-by: Mateusz Jablonski <mateusz.jablonski@intel.com>
2025-04-18 15:02:31 +02:00
Zbigniew Zdanowicz
66c8f86fd8 performance: make reservation in residency before adding to the container
Related-To: NEO-13916

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-04-17 11:02:54 +02:00
Zbigniew Zdanowicz
908cd0abb7 performance: unify all specific non append method calls into single conditional
Related-To: NEO-13916

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-04-16 15:01:21 +02:00
Chandio, Bibrak Qamar
bed2ad1ac0 feature: memAdise for System Allocator
Related-To: NEO-13715

Implements basic functionality for zeCommandListAppendMemoryAdvise for
system allocator

Signed-off-by: Chandio, Bibrak Qamar <bibrak.qamar.chandio@intel.com>
2025-04-16 07:11:14 +02:00
Zbigniew Zdanowicz
09c9b555f9 fix: correct scope for usm allocations indirect access lock
Related-To: NEO-10356

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-04-11 19:39:48 +02:00
Zbigniew Zdanowicz
cf7209ddfe refactor: change queue and immediate command list to accept indirect lock
Related-To: NEO-10356

Signed-off-by: Zbigniew Zdanowicz <zbigniew.zdanowicz@intel.com>
2025-04-11 13:45:29 +02:00