Shared volumes in freestyle composition

Tests
#1 
   run_integration_tests:
     title: Test pipeline
     stage: test-build
     working_directory: IMAGE_WORK_DIR
     image: '${{build_scheduler}}'
     commands:
       - ls
       - ./integration-test.sh
     services:
       volumes:
         ${{CF_VOLUME}}:
           external: true
       composition:
         postgres:
           image: postgres:9.5
           environment:
             - POSTGRES_DB=airflow
             - POSTGRES_PASSWORD=Password123
             - POSTGRES_USER=postgres
             - TZ=Asia/Singapore
         pipeline_app_1:
           image: '${{build_mock_app}}'
           volumes:
             - ${{CF_VOLUME}}:/app/output
           command: bash -c "pwd && ls && echo 'Finished tests' > test_result.txt"

gives this error:

ERROR: Named volume "pipeline_5d6e1fa1a6c4390f0b463fcc_trigger_5d6e200a2287a2849ea5f4fc:/app/output:rw" is used in service "pipeline_app_1" but no declaration was found in the volumes section.
#2

try:
volumes:
- ${{CF_VOLUME}}:/codefresh/volume/

#3

Cannot start composition with warnings [“Volume mapping is not supported, try use: /app/output”]

#4

Haven’t tried it myself but this page shows using ${{CF_VOUME_NAME}} instead https://codefresh.io/docs/docs/yaml-examples/examples/shared-volumes-of-service-from-composition-step-for-other-yml-steps/

#5

CF_VOLUME_NAME also yields the same error.

#6

I believe you are hitting a similar issue as we did at https://support.codefresh.io/hc/en-us/requests/1356

This was the response from Codefresh:

We have certain volume declarations in compositions that we support and some others that we don’t.

We DO NOT support absolute path mappings:

  • /opt/data:/var/lib/mysql

But we do support relative paths:

  • ./cache:/tmp/cache

Which are relative to the location of the compose file.

This is why the variable referenced used is {{CF_VOLUME_NAME}} in the form of: ${{CF_VOLUME_NAME}}:${{CF_VOLUME_PATH}}'

#7

I wasn’t able to see the linked issue. In any case, i wasn’t able to use relative paths either. Not sure if i did it wrong, but i checked with a Codefresh staff member and engineering is adding the enhancement for a freestyle step to be able to access volumes in service containers.

#8

Our engineering team added automounting on service containers in the setup block.

version: '1.0'
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/cf-example-unit-test'
    revision: 'master'
    git: github
  hello_world_step:
    title: free style step
    image: node:11
    commands:
      - ls -l /codefresh/volume/
      - cat /codefresh/volume/test.txt
    services:
      composition:
        my_custom_service:
          image: 'alpine:latest'  
      setup:
        image: 'alpine:latest'
        commands:
          - "ls -l /codefresh/volume"
          - "echo 'from service' > /codefresh/volume/test.txt"
#9

You can also mount the volume yourself in the service container itself, but I don’t recommend this. I think the setup block is better

version: '1.0'
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/cf-example-unit-test'
    revision: 'master'
    git: github
  hello_world_step:
    title: free style step
    image: node:11
    commands:
      - ls -l /codefresh/volume/
      - cat /codefresh/volume/test.txt
    services:
      composition:
        my_custom_service:
          image: 'alpine:latest'
          volumes:
            - ${{CF_VOLUME_NAME}}:/codefresh/volume
          command: 'ls -l /codefresh/volume'
      setup:
        image: 'alpine:latest'
        commands:
          - "echo 'from service' > /codefresh/volume/test.txt"
1 Like
#10

Basically, this portion is always included then, from now on?

#11

Yes, but only in the setup block (see first example I posted)

If you want it at the composition itself (not recommended) you still have to include it (second example I posted)

The idea is that in the setup block you preload the DB you use for testing or do some other init action, so you grab files from the shared Codefresh volume. And the volume is already there for you with no extra conf :slight_smile:

1 Like
#12

@Kostis I’m still having challenges with shared volume. I’m getting the following error even though I’m following the documentation. The build spec snippet I’m using is below. Please advise.

Error Message:
[SYSTEM]
Message Failed to prepare composition: Run tests
Caused by Cannot start composition with warnings [“Volume mapping is not supported, try use:
/docker-entrypoint-initdb.d/mysql_init.sql”]
Documentation Link https://codefresh.io/docs/docs/codefresh-yaml/steps/composition/
Pipeline executed successfully

Build Spec

  test:
    title: "Run tests"
    stage: Test
    type: composition
    fail_fast: true
    composition:
      version: "3"
      services:
        # Cache
        exos-redis:
          image: redis:3.2.8
          container_name: "redis"
          expose:
            - "6379"
          healthcheck:
            test: ["CMD", "redis-cli", "-h", "localhost", "ping"]
            interval: 10s
            timeout: 10s
            retries: 5
        # DB
        members-mysql:
          image: mysql/mysql-server:8.0.15
          env_file: .env
          ports:
            - 3306
          environment:
            MYSQL_ROOT_PASSWORD:abc
          volumes:
            - '${{CF_VOLUME_NAME}}/members/etc/docker/mysql/mysql_init.sql:/docker-entrypoint-initdb.d/mysql_init.sql'
          healthcheck:
            test: ["CMD", "curl", "-f", "http://localhost:3306"]
            interval: 10s
            timeout: 10s
            retries: 5
          command: --default-authentication-plugin=mysql_native_password

    composition_candidates:
      app:
        image: ${{build_members_test_image}}
        command: infrastructure/codefresh/test/test.sh
        env_file:
          - infrastructure/codefresh/test/test.env
#13

Hey. It is exactly as this error message states. We don’t support volume directives with compositions.

Basically even though we still support compositions, we have since introduced service containers that are much better https://codefresh.io/docs/docs/codefresh-yaml/service-containers/

One of the advantages of service containers, is that you don’t have to mess with volumes any more. The main volume of the pipeline is already mounted to all services so you can freely access yours files from any service.

I looked at your example and you are still using the old composition syntax instead of the new one (service containers)

Here are also some full examples