Unable to do multistage builds using --cache-from and --target

Docker
1

We’ve been using multistage docker builds to help reduce the time it takes to build our apps. This works great locally and on some other CI/CD services we’ve tested, but we haven’t been able to get --target and --cache-from to work as we’d expect it with Codefresh when using a build step.

Our workflow is to try and pull an image of a previously built intermediate stage, and then use that image for the docker layer cache using --cache-from while we target the same stage to be built.

docker pull ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:cf-build-pip || true
docker pull ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:cf-build-pybase || true
docker pull ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:latest || true

docker build -t ${PROJECT_REPONAME}:cf-build-pip -f compose/production/django/Dockerfile \
    --target pip-install \
    --cache-from ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:cf-build-pip

docker build -t ${PROJECT_REPONAME}:cf-build-pybase -f compose/production/django/Dockerfile \
    --target py-base \
    --cache-from ${PROJECT_REPONAME}:cf-build-pip \
    --cache-from ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:cf-build-pybase

docker build -t ${PROJECT_REPONAME}:latest -f compose/production/django/Dockerfile \
    --target py-app \
    --cache-from ${PROJECT_REPONAME}:cf-build-pip \
    --cache-from ${PROJECT_REPONAME}:cf-build-pybase \
    --cache-from ${AWS_ECR_ACCOUNT_URL}/${PROJECT_REPONAME}:latest

#
# More logic to tag/push cf-build-pip, cf-build-pybase and latest to ECR
#

The above is a simplified version of the process. Since we’re using multistage builds, the intermediate stages are discarded after a build completes and aren’t available in the layer cache.

The first build on a new pull request also has no existing local layer cache on CF, and it would be nice to prepopulate with something that’s likely to have some level of shared history.

Any thoughts on how we can make the above work?

Here’s one of the variations we’ve tried in Codefresh:

steps:
  load_cached_images:
    description: Filling docker cache
    image: codefresh/cf-docker-builder:v14
    commands:
      - docker pull r.cfcr.io/account/cfcr/projectname:cfcr-pip || true
      - docker pull r.cfcr.io/account/cfcr/projectname:cfcr-pybase || true
      - docker pull r.cfcr.io/account/cfcr/projectname:latest || true
  pip_image:
    type: build
    description: installing pip requirements
    target: pip-install
    dockerfile: compose/production/django/Dockerfile
    image_name: cfcr/projectname 
    tag: cfcr-pip
    build_arguments:
      - --cache-from=r.cfcr.io/account/cfcr/projectname:cfcr-pip
    stage: build

  base_image:
    type: build
    description: creating base image
    target: py-base
    dockerfile: compose/production/django/Dockerfile
    image_name: cfcr/projectname
    tag: cfcr-pybase
    build_arguments:
      - --cache-from=cfcr/projectname:cfcr-pip
      - --cache-from=r.cfcr.io/account/cfcr/projectname:cfcr-pybase
    stage: build

final_image:
    type: build
    description: Building the final image
    target: py-app
    dockerfile: compose/production/django/Dockerfile
    image_name: cfcr/projectname
    build_arguments:
      - --cache-from=cfcr/projectname:cfcr-pip
      - --cache-from=cfcr/projectname:cfcr-pybase
      - --cache-from=r.cfcr.io/account/cfcr/projectname:latest

Thanks for the help!

2

This is a great request, sharing with the team.

3

We are using a lot of multi-stage builds and don’t find the need to do any manual cache management. The trick is to set no_cf_cache: true to disable the Codefresh build optimizations.

The build optimizations essentially use --cache-from under the hood which breaks with multi-stage builds because

  • the intermediate stages are not part of the final image that will be used in --cache-from
  • using cache-from disables the Docker build cache
    This means that we actually lose caching when doing a multi-stage build with the CF build optimizations enabled.

With no_cf_cache: true, we can rely on the Docker build cache to cache all the layers of all build stages. CF does all the work of persisting the build cache after a pipeline run and making it available to future runs.

Probably something worth mentioning in the documentation. I personally think that no_cf_cache: true should be default. It also leads to problems when a pipeline is building multiple images as the last one built will always be passed to --cache-from. However, this might not be the right image to use.

2 Likes
4

Good to read about this. I was not aware of the implications. Did Codefresh confirm it is so?

5

@evanx9 Yes but it’s only an issue when the default cache isn’t present. So, for example, when you open up a new branch, or a new pull request it will automatically use a new cache. This is done both for reliability and security purposes. The #1 attack vector on CI systems is cache poisoning.

On other CI/CD systems, you usually have to build all the caching components yourself and if you don’t think about the security implications you can end up leaking secrets. Right now, there’s no way to bypass this and manually refill the cache specifically for multi-stage Docker builds.

I think we’ll need to add the cache-from property to the build flow which will require some education to users. In general bringing layer cache should be totally safe because secrets shouldn’t be baked into images anyway.

Edit: There is a way to do this if you’re running an Enterprise account with dedicated nodes because you can enable Docker daemon access.

1 Like
6

@alexjillard if you try @jfahrer’s suggestion would you report back the impact?

7

Can you be more precise what you mean? What is the default cache in this scenario?

8

We’ve done a few tests with no_cf_cache:true on our build steps, and it seems to be picking up layers from the cache like we want, and only invalidates the appropriate layers on code changes.

@jfahrer Thanks for the suggestion!