Shared volumes in freestyle composition

Tests
1 
   run_integration_tests:
     title: Test pipeline
     stage: test-build
     working_directory: IMAGE_WORK_DIR
     image: '${{build_scheduler}}'
     commands:
       - ls
       - ./integration-test.sh
     services:
       volumes:
         ${{CF_VOLUME}}:
           external: true
       composition:
         postgres:
           image: postgres:9.5
           environment:
             - POSTGRES_DB=airflow
             - POSTGRES_PASSWORD=Password123
             - POSTGRES_USER=postgres
             - TZ=Asia/Singapore
         pipeline_app_1:
           image: '${{build_mock_app}}'
           volumes:
             - ${{CF_VOLUME}}:/app/output
           command: bash -c "pwd && ls && echo 'Finished tests' > test_result.txt"

gives this error:

ERROR: Named volume "pipeline_5d6e1fa1a6c4390f0b463fcc_trigger_5d6e200a2287a2849ea5f4fc:/app/output:rw" is used in service "pipeline_app_1" but no declaration was found in the volumes section.
2

try:
volumes:
- ${{CF_VOLUME}}:/codefresh/volume/

3

Cannot start composition with warnings [“Volume mapping is not supported, try use: /app/output”]

4

Haven’t tried it myself but this page shows using ${{CF_VOUME_NAME}} instead Shared volumes of service from composition step for other yml steps · Codefresh | Docs

5

CF_VOLUME_NAME also yields the same error.

6

I believe you are hitting a similar issue as we did at Codefresh

This was the response from Codefresh:

We have certain volume declarations in compositions that we support and some others that we don’t.

We DO NOT support absolute path mappings:

  • /opt/data:/var/lib/mysql

But we do support relative paths:

  • ./cache:/tmp/cache

Which are relative to the location of the compose file.

This is why the variable referenced used is {{CF_VOLUME_NAME}} in the form of: ${{CF_VOLUME_NAME}}:${{CF_VOLUME_PATH}}'

7

I wasn’t able to see the linked issue. In any case, i wasn’t able to use relative paths either. Not sure if i did it wrong, but i checked with a Codefresh staff member and engineering is adding the enhancement for a freestyle step to be able to access volumes in service containers.

8

Our engineering team added automounting on service containers in the setup block.

version: '1.0'
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/cf-example-unit-test'
    revision: 'master'
    git: github
  hello_world_step:
    title: free style step
    image: node:11
    commands:
      - ls -l /codefresh/volume/
      - cat /codefresh/volume/test.txt
    services:
      composition:
        my_custom_service:
          image: 'alpine:latest'  
      setup:
        image: 'alpine:latest'
        commands:
          - "ls -l /codefresh/volume"
          - "echo 'from service' > /codefresh/volume/test.txt"
9

You can also mount the volume yourself in the service container itself, but I don’t recommend this. I think the setup block is better

version: '1.0'
steps:
  main_clone:
    title: Cloning main repository...
    type: git-clone
    repo: 'codefreshdemo/cf-example-unit-test'
    revision: 'master'
    git: github
  hello_world_step:
    title: free style step
    image: node:11
    commands:
      - ls -l /codefresh/volume/
      - cat /codefresh/volume/test.txt
    services:
      composition:
        my_custom_service:
          image: 'alpine:latest'
          volumes:
            - ${{CF_VOLUME_NAME}}:/codefresh/volume
          command: 'ls -l /codefresh/volume'
      setup:
        image: 'alpine:latest'
        commands:
          - "echo 'from service' > /codefresh/volume/test.txt"
1 Like
10

Basically, this portion is always included then, from now on?

11

Yes, but only in the setup block (see first example I posted)

If you want it at the composition itself (not recommended) you still have to include it (second example I posted)

The idea is that in the setup block you preload the DB you use for testing or do some other init action, so you grab files from the shared Codefresh volume. And the volume is already there for you with no extra conf :slight_smile:

1 Like
12

@Kostis I’m still having challenges with shared volume. I’m getting the following error even though I’m following the documentation. The build spec snippet I’m using is below. Please advise.

Error Message:
[SYSTEM]
Message Failed to prepare composition: Run tests
Caused by Cannot start composition with warnings [“Volume mapping is not supported, try use:
/docker-entrypoint-initdb.d/mysql_init.sql”]
Documentation Link Composition · Codefresh | Docs
Pipeline executed successfully

Build Spec

  test:
    title: "Run tests"
    stage: Test
    type: composition
    fail_fast: true
    composition:
      version: "3"
      services:
        # Cache
        exos-redis:
          image: redis:3.2.8
          container_name: "redis"
          expose:
            - "6379"
          healthcheck:
            test: ["CMD", "redis-cli", "-h", "localhost", "ping"]
            interval: 10s
            timeout: 10s
            retries: 5
        # DB
        members-mysql:
          image: mysql/mysql-server:8.0.15
          env_file: .env
          ports:
            - 3306
          environment:
            MYSQL_ROOT_PASSWORD:abc
          volumes:
            - '${{CF_VOLUME_NAME}}/members/etc/docker/mysql/mysql_init.sql:/docker-entrypoint-initdb.d/mysql_init.sql'
          healthcheck:
            test: ["CMD", "curl", "-f", "http://localhost:3306"]
            interval: 10s
            timeout: 10s
            retries: 5
          command: --default-authentication-plugin=mysql_native_password

    composition_candidates:
      app:
        image: ${{build_members_test_image}}
        command: infrastructure/codefresh/test/test.sh
        env_file:
          - infrastructure/codefresh/test/test.env
13

Hey. It is exactly as this error message states. We don’t support volume directives with compositions.

Basically even though we still support compositions, we have since introduced service containers that are much better Service Containers · Codefresh | Docs

One of the advantages of service containers, is that you don’t have to mess with volumes any more. The main volume of the pipeline is already mounted to all services so you can freely access yours files from any service.

I looked at your example and you are still using the old composition syntax instead of the new one (service containers)

Here are also some full examples

14

We don’t support volume directives with compositions.

I want to check that means the instructions from this page on how to access the Docker daemon in Codefresh is completely void? https://support.codefresh.io/hc/en-us/articles/360017901040-How-to-Access-the-Docker-Daemon-in-a-Codefresh-build since the stated preferred way is to use the composition step, and mount the Docker socket /var/run/docker.sock:/var/run/docker.sock.

15

Hello

The support page that you reference is newer than the docs.

However, I cannot guarantee that it will work for you, because over the last months we have locked down several aspects of Codefresh, because of Cryptominer abuse.

I suggest you try the mounting of the socket in a composition and if it doesn’t work, please open a ticket with us so that we can investigate.

May I ask which tool you want to use in your Codefresh pipeline that required direct socket access to the docker daemon?

16

Hi @qbiqing & @Kostis,

Did you eventually manage to get the test-containers up and running in code-fresh??
I think I’m stuck trying to get the service volumes to mount properly.

17

hi @tmgodinho, I opened a separate issue here: Mounting Docker socket in composition step

I was trying to mount a docker socket in composition and it was not allowed