Only run the "build" job on runner with fast-single-thread

Some of our runners is vey slow and building on them takes +1 hour and
we need to adjust many of the timeouts, so let's just build on the fast
runners.
5 jobs for faster in 28 minutes and 23 seconds (queued for 2 seconds)