mins read

Switching to Monorepo Architecture

Switching to Monorepo Architecture

November 10, 2022
Green Alert
Last Update posted on
February 3, 2024
Beyond Monitoring: Predictive Digital Risk Protection with CloudSEK

Protect your organization from external threats like data leaks, brand threats, dark web originated threats and more. Schedule a demo today!

Schedule a Demo
Table of Contents
Author(s)
No items found.
  • Author: Rohan Luthra
  • Editor: Benila Susan Jacob

An Obstacle

We started a new project a year ago in CloudSEK, and changed our architecture from monolithic and service-oriented to proper Microservice architecture. With the microservices, each service had its own repository and CI CD pipeline connected to our Kubernetes service mesh.

Everything was going smoothly until the number of services started to increase, and then it became tough to manage the code base. There were as many MRs (Merged Requests) as there were repositories, multiple CICDs to manage, multiple access issues, and a lot of code duplication, which led to increased manual effort. Even though there were common npm packages, updating this package in all repositories was an engineer’s nightmare. This eventually became a bottleneck and started affecting the team’s velocity.

The Solution

Although it should have been anticipated from our microservice approach, one learns from their mistakes. The issue is only with maintainability, hence a monorepo would solve the problem.

Things in the market

Since the tech stack is TS, both frontend and backend, there were a couple of frameworks to look at. The following are the frameworks on which we did a POC before moving forward:

  • Turborepo
  • Lerna
  • Nx

We chose Nx, as it had out-of-the-box capabilities supporting ReactJS and NestJs. Plus it had a couple of advantages over other solutions, specifically with the ‘affected’ code commands.

Setting up NX

With Nx, it’s super simple, just install nx-cli globally and start adding apps.

$ npx create-nx-workspace –pm pnpm –interactive false –defaultBase main –name my-workspace

Since we are using NestJs, let’s create 2 apps and 2 libs

# install NestJs plugin
$ pnpm install -D @nrwl/nest# apps
$ pnx g @nrwl/nest:app test1
$ pnx g @nrwl/nest:app test1# libs
$ pnx g @nrwl/nest:lib lib1
$ pnx g @nrwl/nest:lib lib2# start the apps
$ pnpm run start:all

The folder structure should look like this:

Folder structure
Folder structure

 

Advantages of using Nx

Since maintaining a monorepo is a lot of hassle because of too much code, Nx provides a way to get only changes: Affected

We demonstrate with an example how it was used to our benefit. In the setup, we had created 2 apps and 2 libs and just imported the libraries in these apps. Lets see the overall graph:

Test 1
Test 1

 

Test 2
Test 2

 

$ pnx graph
Now if changes are made to only lib2:
Now if changes are made to only lib2:

 

$ pnx affected:graph –base=main
As expected it only showed changes to test1 app. Similarly, if changes are made to lib2:
As expected it only showed changes to test1 app. Similarly, if changes are made to lib2:

 

$ pnx affected:graph –base=main

Commands with Affected

The changes are reflected in both the apps. Using the affected we need not run lints, tests, builds and deployments for the whole codebase, but just for the affected parts.

Cherry on top

As this was integrated in our CI CD, we were able to lint, test, build and deploy our services. All our pipelines ran only for affected code changes only.

 

But the best finding was that the docker container size was almost half of our previous build. Reason is that Nx build and NestJs builds are very different and Nx makes a smaller build.

Nx Build –

NestJs build –

Conclusion

Nobody can know everything, we all are just learning from our mistakes. And Nx is one of the best monorepo frameworks out there for TS/JS. In this blog, I’ve shown how Nx has helped us in creating a better structure to maintain our code and enhance our pipeline. Some major benefits we got shifting to the Nx –

  • Easier to maintain our code – single repository
  • Scaleable CICD setups
  • Faster CI/CD pipelines – as jobs run only happens for affected code only
  • Smaller docker builds
  • Simple integration with NestJs and ReactJs, easy migration

Other important notes with NX

  • Moving to pnpm ( diff ) ( benchmark ), as pnpm is 3 times faster than npm/yarn
  • Can do e2e testing with Jest and cypress
  • We can also publish our libs or apps (optional)

Author

Predict Cyber threats against your organization

Related Posts
Blog Image
May 19, 2020

How to bypass CAPTCHAs easily using Python and other methods

How to bypass CAPTCHAs easily using Python and other methods

Blog Image
June 3, 2020

What is shadow IT and how do you manage shadow IT risks associated with remote work?

What is shadow IT and how do you manage shadow IT risks associated with remote work?

Blog Image
June 11, 2020

GraphQL 101: Here’s everything you need to know about GraphQL

GraphQL 101: Here’s everything you need to know about GraphQL

Join 10,000+ subscribers

Keep up with the latest news about strains of Malware, Phishing Lures,
Indicators of Compromise, and Data Leaks.

Take action now

Secure your organisation with our Award winning Products

CloudSEK Platform is a no-code platform that powers our products with predictive threat analytic capabilities.

Engineering

min read

Switching to Monorepo Architecture

Switching to Monorepo Architecture

Authors
Co-Authors
No items found.
  • Author: Rohan Luthra
  • Editor: Benila Susan Jacob

An Obstacle

We started a new project a year ago in CloudSEK, and changed our architecture from monolithic and service-oriented to proper Microservice architecture. With the microservices, each service had its own repository and CI CD pipeline connected to our Kubernetes service mesh.

Everything was going smoothly until the number of services started to increase, and then it became tough to manage the code base. There were as many MRs (Merged Requests) as there were repositories, multiple CICDs to manage, multiple access issues, and a lot of code duplication, which led to increased manual effort. Even though there were common npm packages, updating this package in all repositories was an engineer’s nightmare. This eventually became a bottleneck and started affecting the team’s velocity.

The Solution

Although it should have been anticipated from our microservice approach, one learns from their mistakes. The issue is only with maintainability, hence a monorepo would solve the problem.

Things in the market

Since the tech stack is TS, both frontend and backend, there were a couple of frameworks to look at. The following are the frameworks on which we did a POC before moving forward:

  • Turborepo
  • Lerna
  • Nx

We chose Nx, as it had out-of-the-box capabilities supporting ReactJS and NestJs. Plus it had a couple of advantages over other solutions, specifically with the ‘affected’ code commands.

Setting up NX

With Nx, it’s super simple, just install nx-cli globally and start adding apps.

$ npx create-nx-workspace –pm pnpm –interactive false –defaultBase main –name my-workspace

Since we are using NestJs, let’s create 2 apps and 2 libs

# install NestJs plugin
$ pnpm install -D @nrwl/nest# apps
$ pnx g @nrwl/nest:app test1
$ pnx g @nrwl/nest:app test1# libs
$ pnx g @nrwl/nest:lib lib1
$ pnx g @nrwl/nest:lib lib2# start the apps
$ pnpm run start:all

The folder structure should look like this:

Folder structure
Folder structure

 

Advantages of using Nx

Since maintaining a monorepo is a lot of hassle because of too much code, Nx provides a way to get only changes: Affected

We demonstrate with an example how it was used to our benefit. In the setup, we had created 2 apps and 2 libs and just imported the libraries in these apps. Lets see the overall graph:

Test 1
Test 1

 

Test 2
Test 2

 

$ pnx graph
Now if changes are made to only lib2:
Now if changes are made to only lib2:

 

$ pnx affected:graph –base=main
As expected it only showed changes to test1 app. Similarly, if changes are made to lib2:
As expected it only showed changes to test1 app. Similarly, if changes are made to lib2:

 

$ pnx affected:graph –base=main

Commands with Affected

The changes are reflected in both the apps. Using the affected we need not run lints, tests, builds and deployments for the whole codebase, but just for the affected parts.

Cherry on top

As this was integrated in our CI CD, we were able to lint, test, build and deploy our services. All our pipelines ran only for affected code changes only.

 

But the best finding was that the docker container size was almost half of our previous build. Reason is that Nx build and NestJs builds are very different and Nx makes a smaller build.

Nx Build –

NestJs build –

Conclusion

Nobody can know everything, we all are just learning from our mistakes. And Nx is one of the best monorepo frameworks out there for TS/JS. In this blog, I’ve shown how Nx has helped us in creating a better structure to maintain our code and enhance our pipeline. Some major benefits we got shifting to the Nx –

  • Easier to maintain our code – single repository
  • Scaleable CICD setups
  • Faster CI/CD pipelines – as jobs run only happens for affected code only
  • Smaller docker builds
  • Simple integration with NestJs and ReactJs, easy migration

Other important notes with NX

  • Moving to pnpm ( diff ) ( benchmark ), as pnpm is 3 times faster than npm/yarn
  • Can do e2e testing with Jest and cypress
  • We can also publish our libs or apps (optional)