Running 'n' steps in parallel

Pipeline workflows
#1

We have the feature of running steps in parallel (Advanced Workflows with Parallel steps · Codefresh | Docs). In this case, the number of steps we want to run is known in advance.
I have a scenario where the number of steps I want to run in parallel is dynamic, calculated via an API call to be precise, something like:

version: '1.0'
steps:
  GetInfo:
    title: Get list of files
    image: alpine
    commands: 
          - some logic to get the list_of_files; list might have 1 or many files
          - export list_of_files to be usable in subsequent steps
  WritingInParallel:
    type: parallel
    steps:
      writing_file_n:
        title: Step_n
        image: alpine
        commands: 
          - echo "Step_n" > list_of_files[n]

Somehow, I wish to run n (number of file in list_of_files) parallel steps here. Is there a way of doing it using parallel or scale method or any other method?

#2

Hello @manishjainctr

No, currently this is not supported. Pipelines do not support changing the number of steps in the middle of the pipeline.

Please open a feature request with our support team. They might also offer some workarounds for this.

#3

If ‘n’ nubmer of step is possible in parallel, Is there any way to run ‘n’ step sequentially.

#4

You could use a custom plugin to run a dynamic number of steps sequentially https://codefresh.io/docs/docs/codefresh-yaml/steps/#example-with-step-templating

But they will still appear as a single box in the UI.

#5

I have a similar issue where we have an Nx monorepo and I’m not sure how many “apps” I need to build until a step runs. What I do is setup a new pipeline that gets triggered N number of times based on the output from Nx.

I have a step that builds a yaml file that I feed to Codefresh | codefresh-run-dynamic step

      for APP_NAME in $AFFECTED_APPS;
      do
      {
          echo "- pipeline_id: core/app-pipeline"
          echo "  trigger_id: trigger-as-child"
          echo "  branch: $CF_BRANCH"
          echo "  sha: $CF_REVISION"
          echo "  variables:"
          echo "    APP_NAME: $APP_NAME"
      } >> "/codefresh/volume/$CF_BRANCH_TAG_NORMALIZED_LOWER_CASE/trigger-app-pipeline.yaml";

then

  runPipelines:
    title: Trigger app pipelines
    stage: deploy
    type: codefresh-run-dynamic
    arguments:
      RUN_LIST_YAML_FILE: /codefresh/volume/${{CF_BRANCH_TAG_NORMALIZED_LOWER_CASE}}/trigger-app-pipeline.yaml
      LOG_DIRECTORY: /codefresh/volume/${{CF_BRANCH_TAG_NORMALIZED_LOWER_CASE}}/run-logs
#6

Thanks for the tip @mlebarron

That is also another great approach @manishjainctr that you can use.