The performance issue

[KDr2]

We found that one cause of the performance decline of libtask is that compared to calling a function directly, there's some extra allocation in calling it as an Instruction. Here's a simple demonstration:

using BenchmarkTools

function f(a, b, c)
    return a * b + c
end

# call f directly
function f1() 
    a, b, c = 1, 2, 3
    f(a, b, c)
end

# emulate calling f in an instruction
const data = Dict(:a=>1, :b=>2, :c=>3)
const args = [:a, :b, :c]
function f2()
    as = map(x -> data[x], args)
    f(as...)
end

@btime f1()
@btime f2()

In instruction, we have the arguments in a dictionary (bindings), and we should pull them out and put them into a vector/tuple before we call the original function, this is where the extra allocation is introduced.

Is there a way to avoid this extra allocation?

@yebai @devmotion @rikhuijzer

[rikhuijzer]

The allocation is from map, so maybe we can avoid that via re-using some mutable array or by doing the map only once instead of during processing? I don't know enough of the code to know which one is more feasible.

[devmotion]

Generally, also splatting an array is a bit meh.

The cleanest way would be probably to save not f but data -> f(data[:a], data[:b], data[:c]) in the instruction and avoid map altogether. But this might cause other problems since it differs from the original function. And it would still cause dynamic dispatch since the type of the arguments to f can't be inferred. But this is a problem with all currently discussed approaches (and might be responsible for a non-significant part of the performance issues).

[KDr2]

Methods I tried:

using BenchmarkTools

function f(a, b, c)
    return a * b + c
end

function f1()
    a, b, c = 1, 2, 3
    f(a, b, c)
end

const data = Dict(:a=>1, :b=>2, :c=>3)
const args = [:a, :b, :c]

function f2()
    f(data[:a], data[:b], data[:c])
end

function f3()
    as = map(x -> data[x], args)
    f(as...)
end


@inline _call(f) = f()
@inline _call(f, a) = f(data[a])
@inline _call(f, a, b) = f(data[a], data[b])
@inline _call(f, a, b, c) = f(data[a], data[b], data[c])

function f4()
    _call(f, args...)
end


data2 = Any[0, 0, nothing]
function f5()
    map!(x -> data[x], data2, args)
    f(data2...)
end


@btime for _  in 1:100; f1(); end
@btime for _  in 1:100; f2(); end
@btime for _  in 1:100; f3(); end
@btime for _  in 1:100; f4(); end
@btime for _  in 1:100; f5(); end

The results:

  1.109 ns (0 allocations: 0 bytes)
  527.521 ns (0 allocations: 0 bytes)
  8.353 μs (100 allocations: 7.81 KiB)
  5.268 μs (0 allocations: 0 bytes)
  5.908 μs (0 allocations: 0 bytes)

[rikhuijzer]

@KDr2 is the number of arguments passed to map variable in the code? Yes right? I think I saw that before but I'm not sure.

By the way, @btime already runs a loop so no need to loop it and the @inline docs explicitly say: "Small functions typically do not need the @inline annotation, as the compiler does it automatically."

[KDr2]

is the number of arguments passed to map variable in the code?

Yes.

[yebai]

Implementing the instruction callable object as generated function (i.e. here)can probably help get rid of both map and splattering.

[yebai]

For completeness, the issue that @KDr2 refers to can be found here. We observed in #135 that most of the performance regression seem to result from increased GC time.

[rikhuijzer]

We observed in #135 that most of the performance regression seem to result from increased GC time.

Nice. Some good searching there. GC timing is always hard to spot.

I don't think that I know a solution to this problem. I've experimented a bit more and the allocations do certainly come from the allocation of the array.

const SYMS = [:a, :b, :c]
function f4()
    map(s -> data[s], SYMS)
end

function f5()
    y = Vector{Int}(undef, length(SYMS))
    for i in 1:length(SYMS)
        y[i] = data[SYMS[i]]
    end
    y
end

@btime f5() # 32.414 ns (1 allocation: 80 bytes)
@btime f4() # 32.566 ns (1 allocation: 80 bytes)

I don't see how a generated function could avoid that because, as far as I can tell, the compiler doesn't re-use arrays between generated function calls.

Maybe this question is a good one for the Julia Discourse?

[yebai]

Conceptually, a generated function can produce something like

const SYMS = [:a, :b, :c]
const data = Dict(:a=>1, :b=>2, :c=>3)

function f7()
    y1 = data[SYMS[1]]
    y2 = data[SYMS[2]]
    y3 = data[SYMS[3]]
    # f(y1,y2,y3)
end

@btime f7() # 9.827 ns (0 allocations: 0 bytes)

Then all the variables can be allocated on the stack I think. This is possible because the size of input and output is known.

[devmotion]

Generated functions come at the cost of increased compilation times (usually) but I guess if it works probably in most cases it's worth it. One could also use if @generated blocks to let the compiler decide if it uses the generated or non-generated version.