Child Pipeline and Concurrent Piplelines

Hello,

I am pretty sure I am doing something wrong here - I created child pipelines to publish to Dev/QA/Live, but I noticed when I am running several pipelines, at the max number of concurrent pipelines, the child pipelines end up being blocked (marked as pending).
Is there a way to detect there aren’t enough resources to run the parent and the child pipeline before the parent runs?

Here is a sample of my yaml:

version: '1.0'
stages:
  - init
  - test
  - build
  - deploy
steps:
  main_clone:
    title: Clone repository
    stage: init
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'
  init_vars:
    title: Initialize variables
    stage: init
    image: alpine:latest
    commands:
    - lowerBuildConfigName=`echo ${{PROJECT_NAME}} | tr '[:upper:]' '[:lower:]'`
    - cf_export PROJECT_NAME_LOWER=$lowerBuildConfigName
    - cf_export HELM_CHART_PATH='services/${{PROJECT_NAME}}/.helm/'$lowerBuildConfigName
  warning_count:
    title: Warning report
    stage: test
    image: mcr.microsoft.com/dotnet/core/sdk:2.2
    working_directory: services/${{PROJECT_NAME}}/${{PROJECT_NAME}}
    commands:
    - dotnet build /flp:warningsonly /flp:logfile=warnings.log --no-incremental
    - cf_export WARNING_COUNT=`wc -l < warnings.log`
  test_helm:
    title: Helm Lint
    stage: test
    image: codefresh/cfstep-helm:3.2.3
    working_directory: ${{HELM_CHART_PATH}}
    commands:
    # Connect helm to Codefresh Helm repo
    - export ACTION=auth
    - source /opt/bin/release_chart
    - helm repo add default ${{CF_CTX_CF_HELM_DEFAULT_URL}}
    # Test the chart
    - helm dependency update .
    - helm lint --set global.env=Dev .
  build_dockerfile:
    title: Building Dockerfile
    stage: build
    type: build
    buildkit: true
    image_name: ${{PROJECT_NAME_LOWER}}
    working_directory: services/${{PROJECT_NAME}}
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'
    dockerfile: Dockerfile
    target: production
    build_arguments:
      - PROJECT_NAME=${{PROJECT_NAME}}
    disable_push: true
  push_docker_file:
    title: Pushing image to Google Docker registry
    stage: build
    type: push
    candidate: ${{build_dockerfile}}
    tag: '${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}'
    registry: gcr
    when:
      steps:
      - name: build_dockerfile
  get_helm_version:
    title: Get Helm Version
    stage: build
    image: codefresh/cfstep-helm:3.2.3
    working_directory: ${{HELM_CHART_PATH}}
    commands:
    # Connect helm to Codefresh Helm repo
    - export ACTION=auth
    - source /opt/bin/release_chart
    - helm repo add default ${{CF_CTX_CF_HELM_DEFAULT_URL}}
    # Get the current version of this chart in the Codefresh Helm repo
    - export CURRENT_CHART_VERSION=`helm search repo default/${{PROJECT_NAME_LOWER}} | awk 'FNR==2{print $2}'`
    - echo ${CURRENT_CHART_VERSION}
    # Export the new version as an environment variable
    - cf_export NEW_CHART_VERSION=`echo "${CURRENT_CHART_VERSION}" | awk -F. '{$3 = $3 + 1; print (($1)?$1:1) "." (($2)?$2:0) "." $3}'`
    when:
      steps:
      - name: push_docker_file
  update_helm_version:
    title: Update Helm Chart Version
    stage: build
    image: gksoftware/yq
    working_directory: ${{HELM_CHART_PATH}}
    commands:
    # Update the chart version in the chart metadata
    - yq w -i Chart.yaml version ${NEW_CHART_VERSION}
    # Update the chart values with the image tag used to identify the container image built by this run of the pipeline
    - yq w -i values.yaml image.tag ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}
    when:
      steps:
      - name: get_helm_version
  publish_helm:
    title: Publish Helm chart
    stage: build
    image: codefresh/cfstep-helm:3.2.3
    environment:
    - ACTION=push
    - CHART_REF=${{HELM_CHART_PATH}}
    when:
      steps:
      - name: update_helm_version
  deploy_to_dev:
    title: Deploy to ${{TARGET_ENVIRONMENT}}
    stage: deploy
    type: codefresh-run
    arguments:
      PIPELINE_ID: Shared/Deploy
      VARIABLE:
        - TARGET_ENVIRONMENT=${{TARGET_ENVIRONMENT}}
        - CHART_NAME=${{PROJECT_NAME_LOWER}}
        - CHART_VERSION=${{NEW_CHART_VERSION}}
        - CHANGE_MESSAGE=${{CF_SHORT_REVISION}}
    when:
      steps:
      - name: publish_helm
  approve_for_qa:
    title: Promote to QA?
    stage: deploy
    type: pending-approval
    when:
      condition:
        all:
          prevStepSucceeded: deploy_to_dev.result != 'failure'
          # If we are running a custom build to a dev environment, don't promote
          envIsTrunk: '"${{TARGET_ENVIRONMENT}}" == "Dev"'
  promote_to_qa:
    title: Deploying to QA
    stage: deploy
    type: codefresh-run
    arguments:
      PIPELINE_ID: Shared/Deploy
      VARIABLE:
        - TARGET_ENVIRONMENT=QA
        - CHART_NAME=${{PROJECT_NAME_LOWER}}
        - CHART_VERSION=${{NEW_CHART_VERSION}}
        - CHANGE_MESSAGE='Promotion from ${{TARGET_ENVIRONMENT}}'
    when:
      steps:
      - name: approve_for_qa
        on:
        - approved
  approve_for_staging:
    title: Promote to Staging?
    stage: deploy
    type: pending-approval
    when:
      steps:
      - name: promote_to_qa

Hello and welcome to the Codefresh community.

First of all a quick suggestion regarding your pipeline. Rather than working with parent and child pipelines for this, Codefresh also has the capacity to share the whole pipeline for different applications. So instead of creating shared deploy Prod and shared deploy QA pipelines, you create shared pipelines for the whole process. The concept is explained here Building Microservices · Codefresh | Docs

Now regarding your issue I see two options

  1. Run your child pipelines in “detached” mode. This means that the parent pipeline will not block while the pipeline is running. You need to examine your workflow and the implications of this
  2. Play with the concurrency settings so that you can see if you are launching more pipelines than needed Creating Pipelines · Codefresh | Docs

However in the long run if this is something that happens often it might be beneficial to simply upgrade your plan.

Let us know if you have any more questions on this. You can also open a ticket with our support personnel and they might have different insights on this.

Hi Kostis,

Thank you for your response. I’ll take a look at the Microservices approach as well as I will look at playing with the concurrency settings.
Using Codefresh Runner may be another solution. I noticed that is is possible to autoscale, but it is possible under GCP?

Yes the Codefresh runner is another solution. It is a Kubernetes native application and it will work on all Kubernetes clusters (including GKE).

Note however that builds will run in your infrastructure so while you have more control on capacity you also need to make sure that the runner has the resources it needs.