Streams

Similar to CUDA streams, ROCm has HIP streams, which are buffers used to instruct the GPU hardware which kernels to launch. HIP streams are synchronous, like CUDA streams.

Each device has a default stream associated, which is accessible with AMDGPU.stream().

There are several ways to specify which stream to launch a kernel on:

• Using AMDGPU.stream! to change default stream to be used within the same Julia task.

stream = AMDGPU.HIPStream()
AMDGPU.stream!(stream) # Change default stream to be used for subsequent operations.
AMDGPU.ones(Float32, 16) # Will be executed on `stream`.

• Using AMDGPU.stream! to execute given function and reset to the original stream after completion:

stream = AMDGPU.HIPStream()
x = AMDGPU.stream!(() -> AMDGPU.ones(Float32, 16), stream)

• Using stream argument to @roc macro:

stream = AMDGPU.HIPStream()
@roc stream=stream kernel(...)

Streams also have an inherent priority, which allows control of kernel submission latency and on-device scheduling preference with respect to kernels submitted on other streams. There are three priorities: normal (the default), low, and high priority.

Priority of the default stream can be set with AMDGPU.priority!. Alternatively, it can be set at stream creation time:

low_prio = HIPStream(:low)
high_prio = HIPStream(:high)
normal_prio = HIPStream(:normal) # or just omit "priority"

AMDGPU.stream Function

stream()::HIPStream

Get the HIP stream that should be used as the default one for the currently executing task.

Each task gets its own stream, which may be handed to another task once the task has finished and all work on the stream has completed. If you need a stream that outlives the task, or that is never shared, create one with HIPStream() and activate it using AMDGPU.stream!.

source

AMDGPU.stream! Function

stream!(s::HIPStream)

Change the default stream to be used within the same Julia task.

source

stream!(f::Base.Callable, stream::HIPStream)

Change the default stream to be used within the same Julia task, execute f and revert to the original stream.

Returns:

Return value of the function f.

source

AMDGPU.priority! Function

priority!(p::Symbol)

Change the priority of the default stream. Accepted values are :normal (the default), :low and :high.

source

priority!(f::Base.Callable, priority::Symbol)

Change the priority of default stream, execute f and revert to the original priority. Accepted values are :normal (the default), :low and :high.

Returns:

Return value of the function f.

source

AMDGPU.HIP.HIPStream Type

HIPStream(priority::Symbol = :normal)

Arguments:

• priority::Symbol: Priority of the stream: :normal, :high or :low.

Create HIPStream with given priority. Device is the default device that's currently in use.

source

HIPStream(stream::hipStream_t)

Create HIPStream from hipStream_t handle. Device is the default device that's currently in use.

source

Synchronization

By default, AMDGPU.synchronize does not block the calling thread: it briefly polls the stream, and then waits for it on a separate worker thread, so that other tasks can run on the calling thread in the meantime. This is required for Hostcall, whose host side runs as a task. Synchronizing events and HIP.device_synchronize() works the same way. Inside finalizers, which cannot switch tasks, synchronization blocks.

Users, however, can switch to a blocking synchronization globally with nonblocking_synchronization preference or with fine-grained AMDGPU.synchronize(; blocking=true). Blocking synchronization might offer slightly lower latency, but must not be used while hostcalls are running.

You can also perform synchronization of the expression with AMDGPU.@sync macro, which will execute given expression and synchronize afterwards (using AMDGPU.synchronize under the hood).

AMDGPU.@sync begin
    @roc ...
end

Finally, you can perform full device synchronization with AMDGPU.device_synchronize.

AMDGPU.synchronize Function

synchronize(stream::HIPStream = stream(); blocking::Bool = false)

Wait until all kernels executing on stream have completed.

If there are running HostCalls, then blocking must be false. Additionally, if you want to stop host calls afterwards, then provide stop_hostcalls=true keyword argument.

source

synchronize(A::ROCArray)

Wait until the operations on the stream that last used the memory of A have completed.

source

AMDGPU.@sync Macro

@sync ex

Run expression ex on currently active stream and synchronize the GPU on that stream afterwards.

See also: synchronize.

source

AMDGPU.HIP.device_synchronize Function

Blocks until all kernels on all streams have completed. Uses currently active device.

source

Implicit synchronization of arrays

AMDGPU.jl keeps track of the stream that last used the memory of each ROCArray. Using the array on another stream (launching a kernel, copying, or taking its pointer) first waits, on the host, for the stream that used it before, so that operations on different streams are ordered. AMDGPU.synchronize(A) waits for the stream that last used A.

This implicit synchronization can be disabled per array, e.g. to let kernels that work on disjoint parts of an array run concurrently on different streams. Ordering the streams is then up to the user:

s1, s2 = HIPStream(), HIPStream()
AMDGPU.enable_synchronization!(A, false)
@roc stream=s1 kernel_a!(A)  # e.g. computes the interior of `A`
@roc stream=s2 kernel_b!(A)  # e.g. computes the boundary of `A`, concurrently
AMDGPU.synchronize(s1); AMDGPU.synchronize(s2)
AMDGPU.enable_synchronization!(A)

AMDGPU.enable_synchronization! Function

enable_synchronization!(A::ROCArray, enable::Bool=true) -> A

By default, using a ROCArray on another stream than the one that used it last (launching a kernel, copying, or taking its pointer) first synchronizes that stream from the host, so that operations on different streams are ordered. This may be unwanted when, e.g., disjoint parts of an array are processed concurrently on different streams. This function disables (enable=false) or re-enables this implicit synchronization. The memory is still recorded as used by the stream that uses it last.

Warning

This affects all ROCArrays that refer to the same memory (e.g. views and reshapes). While implicit synchronization is disabled, ordering operations on different streams, and synchronizing before accessing the memory from the host or freeing it, is up to the user: unsafe use results in data races. Re-enabling it does not wait for operations that used the memory on other streams in the meantime.

See also AMDGPU.synchronization_enabled, AMDGPU.synchronize.

source

AMDGPU.synchronization_enabled Function

synchronization_enabled(A::ROCArray) -> Bool

Whether the implicit synchronization of the memory of A is enabled (see AMDGPU.enable_synchronization!).

source