Running 'n' steps in parallel

We have the feature of running steps in parallel (Advanced Workflows with Parallel steps · Codefresh | Docs). In this case, the number of steps we want to run is known in advance.
I have a scenario where the number of steps I want to run in parallel is dynamic, calculated via an API call to be precise, something like:

version: '1.0'
    title: Get list of files
    image: alpine
          - some logic to get the list_of_files; list might have 1 or many files
          - export list_of_files to be usable in subsequent steps
    type: parallel
        title: Step_n
        image: alpine
          - echo "Step_n" > list_of_files[n]

Somehow, I wish to run n (number of file in list_of_files) parallel steps here. Is there a way of doing it using parallel or scale method or any other method?

Hello @manishjainctr

No, currently this is not supported. Pipelines do not support changing the number of steps in the middle of the pipeline.

Please open a feature request with our support team. They might also offer some workarounds for this.

If ‘n’ nubmer of step is possible in parallel, Is there any way to run ‘n’ step sequentially.

You could use a custom plugin to run a dynamic number of steps sequentially

But they will still appear as a single box in the UI.

I have a similar issue where we have an Nx monorepo and I’m not sure how many “apps” I need to build until a step runs. What I do is setup a new pipeline that gets triggered N number of times based on the output from Nx.

I have a step that builds a yaml file that I feed to Codefresh | codefresh-run-dynamic step

          echo "- pipeline_id: core/app-pipeline"
          echo "  trigger_id: trigger-as-child"
          echo "  branch: $CF_BRANCH"
          echo "  sha: $CF_REVISION"
          echo "  variables:"
          echo "    APP_NAME: $APP_NAME"
      } >> "/codefresh/volume/$CF_BRANCH_TAG_NORMALIZED_LOWER_CASE/trigger-app-pipeline.yaml";


    title: Trigger app pipelines
    stage: deploy
    type: codefresh-run-dynamic
      RUN_LIST_YAML_FILE: /codefresh/volume/${{CF_BRANCH_TAG_NORMALIZED_LOWER_CASE}}/trigger-app-pipeline.yaml
      LOG_DIRECTORY: /codefresh/volume/${{CF_BRANCH_TAG_NORMALIZED_LOWER_CASE}}/run-logs

Thanks for the tip @mlebarron

That is also another great approach @manishjainctr that you can use.