Only run the "build" job on runner with fast-single-thread

Some of our runners is vey slow and building on them takes +1 hour and
we need to adjust many of the timeouts, so let's just build on the fast
runners.