# Scaling monorepo microservices **URL:** \ **Category:** Pipeline Workflows\ **Created:** [December 26, 2021, 10:11am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696 "2021-12-26T10:11:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Tal\_Faitlov](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/tal_faitlov/32/448_2.png) [@Tal\_Faitlov](https://community.codefresh.io/u/Tal_Faitlov)\ **Post date:** [December 26, 2021, 10:11am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/1 "2021-12-26T10:11:05Z")
Greetings community! We have a Codefresh pipeline that builds images of all miscroservices and deploys to EKS them using a lambda function Building the images looks something like this: ```auto ....build_ms_images: ........title: 'build.docker.images' ........type: 'build' ........stage: 'image' ........working_directory: '${{clone}}' ........image_name: 'image-name-prefix' ........registry: ${{AWS_DOCKER_REGISTRY}} ........dockerfile: 'ci/ci.Dockerfile' ........build_arguments: ............- DEPLOY_ENV=${{DEPLOY_ENV}} ........scale: ............serviceA: ................working_directory: '${{clone}}/path/to/serviceA' ................title: 'service.A' ................tag: 'ms-service-a-${{CF_REVISION}}' ............serviceB: ................working_directory: '${{clone}}/path/to/serviceB' ................title: 'service.B' ................tag: 'ms-service-b-${{CF_REVISION}}' ``` WORKS GREAT! We currently have 20 such services New services are being created on a weekly basis (the system is growing) We want to add a “delta” capability into the pipeline - build and deploy only what changed I already know how to programmatically list the services that have changed since the last deployment (dependencies an all) QUESTIONS: How do we run only the (scaled) build-image steps of the changed services? Can we avoid having to add each service to the pipeline? THOUGHTS: Using a condition on each scale-step… I don’t love it but it’s a solution What I really want is scaling steps dynamically (according to an env var exported from a freestyle step maybe) - is this a sane thought? YOUR THOUGHTS? P.S. I have AgroCD docs opened in the next tab, I’ll read about it ---
**Author:** ![Kostis](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/kostis/32/391_2.png) [@Kostis](https://community.codefresh.io/u/Kostis)\ **Post date:** [December 27, 2021, 8:17am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/2 "2021-12-27T08:17:27Z")
Hello This is similar to this question [Need to build multiple docker images](https://community.codefresh.io/t/need-to-build-multiple-docker-images/691) Basically you need to use our monorepo support and have a single pipeline (with a single service) that automatically gets triggered according to what was changed. See more here [How To Use Codefresh with Mono-Repos](https://codefresh.io/continuous-integration/using-codefresh-with-mono-repos/) Alternatively if you can programmatically find what was changed you can simply call your own pipeline with codefresh-run [Codefresh | codefresh-run step](https://codefresh.io/steps/step/codefresh-run) Finally I see that you are passing the environment to the build. This is an anti-pattern (having docker images that know their environment) See anti-pattern 2 here [Kubernetes Deployment Antipatterns - part 1 | Codefresh](https://codefresh.io/kubernetes-tutorial/kubernetes-antipatterns-1/) or anti-pattern 5 here [Docker Anti Patterns](https://codefresh.io/containers/docker-anti-patterns/) for more details. ---
**Author:** ![Tal\_Faitlov](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/tal_faitlov/32/448_2.png) [@Tal\_Faitlov](https://community.codefresh.io/u/Tal_Faitlov)\ **Post date:** [December 30, 2021, 5:49am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/3 "2021-12-30T05:49:21Z")
Hey Kostis read many of your articles honored - Single pipeline per service: 1. Deploying is a transaction, if one of the services failed to build (typescript, dockerfile) - none should be deployed 2. Managing 50 plus pipelines seems daunting, even if the process of creating new pipelines is automatic 3. we don’t always want to provide backward compatibility - that is one of the perks of using a monorepo - codefresh-run I used it in the past gained major performance improvements by going back to single pipeline Also easier to understand and maintain - env vars in the image Yes, we are aware Currently, services are reading configuration from code (.env files) We will be addressing this in the near future by either: 1. A central configuration service 2. Setting env var in the kube yaml (dev/staging/prod have separate, per-service files) Let’s try to get past the “life style” decisions we took and think about a more dynamic way to handle a multi-service pipeline Tal Faitlov ---
**Author:** ![Kostis](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/kostis/32/391_2.png) [@Kostis](https://community.codefresh.io/u/Kostis)\ **Post date:** [December 30, 2021, 11:50am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/4 "2021-12-30T11:50:42Z")
Hello If you can list the services yourself you can simply create a dynamic step that builds images on the fly. See an example of dynamic git cloning here [Steps · Codefresh | Docs](https://codefresh.io/docs/docs/codefresh-yaml/steps/#example-with-step-templating) You can also parse the Github event yourself for more details [Git Triggers · Codefresh | Docs](https://codefresh.io/docs/docs/configure-ci-cd-pipeline/triggers/git-triggers/#accessing-directly-the-webhook-content-of-the-trigger) (Some people also use git diff commands in a freestyle step) In general however I think it is much easier to simply build all services and let the caching mechanisms do the heavy lifting. Sometimes runtime dependencies and build dependencies are completely different. ---
**Author:** ![Tal\_Faitlov](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/tal_faitlov/32/448_2.png) [@Tal\_Faitlov](https://community.codefresh.io/u/Tal_Faitlov)\ **Post date:** [January 2, 2022, 4:26am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/5 "2022-01-02T04:26:59Z")
Hey The list of changed services that need to be deployed is based on git diff - via pnpm since filter We are aiming for a system where all runtime APIs are described by a code Everything has an SDK, nothing is being “guessed” So runtime dependencies can be statically discoverable at build time in our fully-typed monorepo Caching alone won’t cut down a 30 min build&deploy for 50 services to less than 2 minutes for a single service commit, no sir, no way We have an automatically triggered Test-Only pipeline validating PRs Support for running according to changed packages (and dependents) was added three-week ago The improvement is very noticeable, this is a developer-experience must-have Now, **How to I write a dynamic step to run parallel build image steps (like “scale:” does)?** Because the below times 50 in my pipeline is just not cutting it: ```nohighlight accounts_management: working_directory: '${{clone}}/service/is/here/' title: 'my service' tag: 'my-service-name-${{CF_REVISION}}' when: condition: all: shouldDeploy: 'includes("${{MICRO_SERVICES_TO_DEPLOY}}", "@my/service-name") == true' ``` Ps: a shorthand ‘when’ version would be great, so will being able to base conditions on filesystem queries Tal ---
**Author:** ![Kostis](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/kostis/32/391_2.png) [@Kostis](https://community.codefresh.io/u/Kostis)\ **Post date:** [January 3, 2022, 8:35am UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/6 "2022-01-03T08:35:18Z")
> **How to I write a dynamic step to run parallel build image steps (like “scale:” does)?** You need to create your own Custom step and use a “range” template. See example here [https://github.com/kostis-codefresh/step-examples/blob/master/multi-clone/multi-clone-step.yml#L82](https://github.com/kostis-codefresh/step-examples/blob/master/multi-clone/multi-clone-step.yml#L82) Let me know if you need more clarifications. ---
**Author:** ![Tal\_Faitlov](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/tal_faitlov/32/448_2.png) [@Tal\_Faitlov](https://community.codefresh.io/u/Tal_Faitlov)\ **Post date:** [January 3, 2022, 12:19pm UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/7 "2022-01-03T12:19:12Z")
Yes, please: A. How can the arguments be a dynamic? (according to output from a script called from the pipeline on the cloned repo) B. How can the steps be executed in parallel? C. How can the dynamic, parallel steps be displayed within the pipeline as separate steps? Like this: ![image](https://us1.discourse-cdn.com/flex019/uploads/codefreshtest/original/1X/71357daeccd6ff9f8d7cfb7459200461de73d463.png) ---
**Author:** ![Kostis](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/kostis/32/391_2.png) [@Kostis](https://community.codefresh.io/u/Kostis)\ **Post date:** [January 4, 2022, 2:57pm UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/8 "2022-01-04T14:57:11Z")
Hello First of all you need to read the guide about creating a plugin. See [https://codefresh.io/docs/docs/codefresh-yaml/steps/#creating-a-typed-codefresh-plugin](https://codefresh.io/docs/docs/codefresh-yaml/steps/#creating-a-typed-codefresh-plugin) Then focus on the `stepsTemplate` section that allows you to use the [Gomplate library](https://github.com/hairyhenderson/gomplate) for templating. The example in the docs shows how to dynamically create clone steps according to a list of repositories. You need to do something similar for creating build steps according to a list or variable as you want. I suggest you open a request ticket. Either another customer has already requested this, or our support team will have more pointers. To answer your questions: A. See the gomplate syntax. It can work with dynamic arguments such as lists/arrays/objects B. Just use the “parallel” syntax in your template as you do now C. I am not sure if this is supported or not. Our support team will know ---
**Author:** ![hugo](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/hugo/32/336_2.png) [@hugo](https://community.codefresh.io/u/hugo)\ **Post date:** [January 5, 2022, 5:57pm UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/9 "2022-01-05T17:57:52Z")
We wanted to do exactly the same thing. We ended up writing a script that creates a pipeline steps file, then in another step we run something like: codefresh run myDynamicChildPipeline --yaml=dynamicallyCreatedStepsFile.yaml We previously created `myDynamicChildPipeline` with empty steps like this: ```auto #DO NOT EDIT THIS FILE, THIS IS GENERATED DURING RUNTIME # This is created at runtime via the Parent pipeline. version: 1.0 steps: {} stages: [] ``` ---
**Author:** ![hugo](https://sea1.discourse-cdn.com/flex019/user_avatar/community.codefresh.io/hugo/32/336_2.png) [@hugo](https://community.codefresh.io/u/hugo)\ **Post date:** [January 5, 2022, 6:00pm UTC](https://community.codefresh.io/t/scaling-monorepo-microservices/696/10 "2022-01-05T18:00:55Z")
We tried this approach and we ran into some problems. One of them related to accessing the credentials for the Google Container Registry, which was a bug in Codefresh that was fixed later. I can’t remember what other problems we ran into.