Use dockers new buildkit to build docker images. That should improve speed and space consumption
https://docs.docker.com/build/buildkit/