This is mostly a backport of issue #21907, but also includes a few
extras necessary to make the bulidbot scripts as thin as possible.