Best practices for config reusability?

Pipeline workflows
1

Hi there,

We have a multi-repo microservices setup with a codefresh.yml config in each repo.
We build a docker image for testing, run tests, build a production image, then commit the new tag to the gitops repo.

Most of the steps are heavily parameterized and identical between services, except for the testing steps in the middle, which vary a lot. We use native Codefresh “service containers” YAML for testing, not an external docker-compose, so we can’t even encapsulate testing steps into a shell script with the same name in each repo.

Of course with this setup, repetition is a huge problem and any change needs to be copy/pasted into every service (which also means n commits because we don’t use a monorepo)

Our problem would be solved instantly if Codefresh YAML allowed importing config snippets from a different path, and merged them before execution. I could put the common stuff in a shared repo somewhere (or at worst in a identical file in each repo, still a small win over a slightly different one). Then import it from the repo’s codefresh.yml which would have only the specific testing steps.

Or the other way round : all pipelines using the same shared codefresh config that only loads the different testing steps from each repo.

This isn’t currently possible is it ?
Is there a chance this could be implemented ?
If not, what do you suggest as an alternative ?

Thanks

2

Great question

Our recommended approach is documented here Building Microservices · Codefresh | Docs or in blog form here CI/CD Pipelines for Microservices | Codefresh. Have you already gone through these?

Basically the plan is to reuse the whole pipeline, instead of importing different parts of it. This is very easy if you have a single programming language in your company. Otherwise you need one pipeline for each programming language.

The blog post also explains how we at Codefresh manage more than 30 applications using just 3 pipelines GitHub - kostis-codefresh/codefresh-on-codefresh-pipelines: Pipelines use to manage Codefresh with Codefresh. All are in a single repo, so every time we change something globally we only need a single commit. Again the article has more details (see the Codefresh-on-Codefresh section).

Other alternatives we have seen people use, is creating reusable typed steps Steps · Codefresh | Docs but I personally don’t think it is necessary.

And of course people have used shared child pipelines from top-level parent pipelines as way to group common functionality.

Would any of these 3 approaches work for you?

3

Hi @Kostis,

First of all thanks for your response. Just so you know, I’m a colleague of @renaud and I’ve been trying to follow your suggestion to define a unique pipeline to be used by our different microservices, but I’ve just reached a blocker when it comes to defining a step for running integration tests, as this would be the step that differs the most between services since we need to link different dependencies (messaging emulator, DB and so on).

Basically, the problem comes when I try to define the services for this step using composition with a local docker-compose.yaml file as one of the examples of Codefresh’s documentation mentions (Service Containers - Docs). My step is defined as follows:

  test:
    title: Running tests
    image: ${{build_test_image}}
    working_directory: ${{clone}}
    services:
      composition: 'docker-compose.yaml'
    stage: test`

and my local docker-compose.yaml looks like this:

version: "3"
accounts-server:
  image: accounts:latest
  environment:
    - CONSOLE=true
    - FIRESTORE_EMULATOR_HOST=firestore:8282
    - FIRESTORE_PROJECT_ID=test
  restart: on-failure
  depends_on:
    - firestore

firestore:
  image: google/cloud-sdk:345.0.0-emulators
  command: gcloud beta emulators firestore start --host-port=0.0.0.0:8282```

For me everything looks normal unless I’m missing something obvious but then when running that step on my pipeline I get the following error

[SYSTEM] .                                                                                                                                                 
 Message             Failed to prepare composition: services                                                                                               
 Caused by           Failed to read file /codefresh/volume/docker-compose.yaml                                                                             
 Documentation Link  https://codefresh.io/docs/docs/codefresh-yaml/steps/freestyle/

I tried multiple combinations of this definition already and none of them worked. Could you give me a hand here?

Thanks in advance

4

Hello

The path /codefresh/volume/docker-compose.yaml looks wrong to me. Isn’t docker-compose.yaml part of your project? I would expect it to be

/codefresh/volume/<your project name>/docker-compose.yaml

As a quick check can you manually modify your step and do this?

 test:
    title: Running tests
    image: ${{build_test_image}}
    working_directory: ${{clone}}/my-project
    services:
      composition: 'docker-compose.yaml'
    stage: test`

or

 test:
    title: Running tests
    image: ${{build_test_image}}
    working_directory: ${{clone}}
    services:
      composition: '/codefresh/volume/my-project/docker-compose.yaml'
    stage: test`

There might be an issue on our side where the “composition” property is not honoring the working_directory property.

You should also open an issue with our support team so that they can test these scenarios instead of you.

5

Hi @Kostis,

I just tried both suggestions and the second one worked like a charm! As you said, it seems that composition property doesn’t point to the correct directory I will raise an issue about this.

Thanks again!

6

Hello again @Kostis,

FYI I work with both Renaud and Miguel above. The solution you recommended mostly works but occasionally we are seeing an error in the tests:

Failed to read file /codefresh/volume/<project name>/docker-compose.yaml

However it works fine after restarting the pipeline. Any ideas why this might be happening?

7

Hello

If you have an approval step in your pipeline make sure that you have enabled this option Approval · Codefresh | Docs

If you don’t have an approval step, then I don’t see why the file should not be there. There are some rare conditions with parallel steps that lead to file issues.

It is best to open a support ticket with the build URL and let our team investigate.

8

We are getting a similar error:

[SYSTEM] .                                                                                                                                          
 Message             Failed to prepare composition: services                                                                                        
 Caused by           Failed to read file /codefresh/volume/<my-project>/docker-compose.yml                                            
 Documentation Link  https://codefresh.io/docs/docs/codefresh-yaml/steps/freestyle/

I’ve checked that the file exists (by debugging the previous step), and is readable by everyone.

9

If the file exists already please open a support ticket with the build URL and let our team investigate.