đ CloudSEK has raised $19M Series B1 Round â Powering the Future of Predictive Cybersecurity
Read More
Protect your organization from external threats like data leaks, brand threats, dark web originated threats and more. Schedule a demo today!
Schedule a Demo
There is a proliferation of modern programming languages now more than ever. Nevertheless, development organizations tend to stick to one or two languages, allowing them to manage their stack properly and to keep it clean.Â
Programming languages are essential tools that if not chosen appropriately, could have a crippling effect on the business. For example, developers prefer Python over Java to set up machine learning algorithms, because Python has better libraries required for machine learning and artificial intelligence applications.Â
Every language has a particular set of capabilities and properties. For instance, some languages are faster than others but may take time to write. Some languages consume more resources and yet make development faster. And, there are other languages that fall in between.
While it is not quite common, a microservice architecture can be built to suit multiple languages. It provides developers with the freedom to choose the right tool for their task. In the final phase of the software development lifecycle, if Docker can deploy containers irrespective of its language, its architecture can be replicated in the initial stages of building microservices as well.
However, managing multiple languages can be a huge task and a developer should definitely consider the following aspects when they plan to add more languages to their technology stack:
REST API requires a new framework for every new language such as Express for Node.js, Flask for Python, or Gin for Go. Also, the developer may have to code their models/ data structures over and over for each service as well. This makes the whole process redundant and can result in several errors.
For example, it is very common to have one database model for multiple services. But, when the database is updated, it has to undergo changes for each service. To build a multilingual microservice, we need a common framework that supports multiple languages and platforms and one that is scalable.
gRPC is Googleâs high performing, open-source remote procedure calls framework. Itâs compatible with multiple languages such as Go, Node.js, Python, Java, C-Sharp, and many more. It uses RPC to provide communication capabilities to microservices. Moreover, it is a better alternative to REST. gRPC is much faster than REST. And this framework uses Protocol Buffers as the interface definition language for serialization and communication instead of JSON/ XML. It works based on server-client communication.
An in-depth analysis of gRPC is given here.
Protocol buffers are a method of serializing data that can be transmitted over wire or be stored in files. In short, Protocol buffer or Protobuf is the same as what JSON is with regards to REST. It is an alternative to JSON/ XML, albeit smaller and faster. Protobuf defines services and other data and is then compiled into multiple languages. The gRPC framework is then used to create a service.
gRPC uses the latest transfer protocol – HTTP/2, which supports bidirectional communication along with the traditional request/ response. In gRPC the server is loosely coupled with the client. In practice, the client opens a long-lived connection with the gRPC server and a new HTTP/2 stream will be opened for each RPC call.
We first define our service in Protobuf and compile it to any language. Then, we use the compiled files to create our gRPC framework to create our server and client, as shown in the diagram below.
To explain the process:
Letâs create a simple microservice – HelloWorldService. This is a very basic service that invokes only one method – HelloWorld – which would return the string âhello worldâ.Â
These are the 4 steps to follow while creating a microservice:Â
For this simple service, we are opting 2 languages: Go and Node.js. Since gRPC works on a client-server architecture, we shall use Go on the server (since it is fast and resource efficient) and Node.js for clients (since most of the apps these days are React/ Angular).
**One can also decide to run gRPC servers with REST API too, if they do not want to create clients. They can do this by creating a REST Proxy server. Although it sounds laborious, it is actually pretty simple and will be dealt with in the âCompilingâ section that will follow.
Messages and services are defined using Protobuf in a proto file (.proto file).
The syntax is quite simple; the messages are defined and are used as Request and Response for the RPC calls, for each service. Here is a language guide for Protocol Buffers.
We can create a new folder for each service under the name of that particular service. And, for ease of accessibility, we can store all these services under the folder âservices.â
The service we are defining here is âhelloworld.â And each folder should contain 2 files, namely:
Now, let’s define our service:
As shown above, we have defined an empty Request, a string Response and a HelloWorldService with a single RPC call HelloWorld. This call accepts requests and returns responses. You can see the full repo here.
Once the service has been defined, we will have to choose the languages to compile the services into. This choice is made based on service requirements and usage, and also the developerâs comfort. The different languages that one can choose are Go, Node.js, Python, Java, C, C#, Ruby, Scala, PHP, etc. As mentioned earlier, in this example, weâll be using Go and Node.js. Weâll then add these languages to the .protolangs file, mentioning each language in a new line.
Compiling is the most interesting part of this entire process. In step 3, weâll compile the .proto file to the selected languages, Go and Node.js.
The Protocol Buffer comes with a command line tool called âprotocâ, which compiles the service definition for use. But for each language weâll have to download and install plugins. This can be achieved by using a dockerfile.
Namely is a docker file which includes all of this and is available to all. It has the âproto compilerâ with support for all languages and additional features like documentation, validator, and even a REST Proxy server that we talked about in step-1. Â
For example,
$ docker run -v `pwd`:/defs namely/protoc-all -f myproto.proto -l ruby
It accepts a .proto file and language, both of which we already have. The following is a simple bash script that will loop through our services folder, and pick up the service.proto file to compile to the languages in the .protolangs file.
#!/bin/bash
echo "Starting ... "
set x
REPO="`pwd`/repo"
function enterDir {
echo "Entering $1"
pushd $1 > /dev/null
}
function leaveDir {
echo "Exiting"
popd > /dev/null
}
function complieProto {
for dir in */; do
if ; then
while read lang; do
target=${dir%/*}
mkdir -p $REPO/$lang
rm -rf $REPO/$lang/$target
mkdir -p $REPO/$lang/$target
mkdir -p $REPO/$lang/$target/doc
echo " Compiling for $lang"
docker run -v `pwd`:/defs namely/protoc-all -f $target/service.proto -l $lang --with-docs --lint $( && echo "--with-typescript" || echo "--with-validator")
cp -R gen/pb-$lang/$target/* $REPO/$lang/$target
cp -R gen/pb-$lang/doc/* $REPO/$lang/$target/doc
sudo rm -rf gen
done < $dir/.protolangs
fi
done
}
function complie {
echo "Starting the Build"
mkdir -p $REPO
for dir in services/; do
enterDir $dir
complieProto $dir
leaveDir
done
}
complie
We, then, run the script:
$ chmod +x generate.sh
$ ./genetare.sh
The file generated appears in the /repos folder.
The successfully generated service_pb for both Node.js and Go, along with some docs and validators, constitutes the boiler-plate code for our server and clients that we are about to create.
**As discussed earlier, if you don’t want to use the client and want REST-JSON APIs instead, you can create a REST Proxy by adding a single tag in the namely/protoc-all dockerfile i.e. — with-gateway. For this weâll have to add api paths in our protofiles. Have a look at this for further information. Now, run this gateway and the REST Proxy server will be ready to serve the gRPC server.Â
Tip: You can also host this protorepo on github as a repository. You can have a single repo for all the definitions in your organisation, in the same way as google does.
Now that we have service_pb code for Go and Node.js, we can use it to create a server and a client. For each language the code will be slightly different, because of the differences in the languages. But the concept will remain the same.
For servers: Weâll have to implement RPC functions.
For clients: Weâll have to call RPC functions.
You can see the gRPC code for all languages here. With only a few lines of code we can create a server and with fewer lines of code we can create a client.
package main
import (
"context"
"fmt"
"log"
"net"
helloworld "github.com/rohan-luthra/service-helloworld-go/helloworld"
"google.golang.org/grpc"
)
type server struct {
}
func (*server) HelloWorld(ctx context.Context, request *helloworld.Request) (*helloworld.Response, error) {
response := &helloworld.Response{
Messsage: "hello world from go grpc",
}
return response, nil
}
func main() {
address := "0.0.0.0:50051"
lis, err := net.Listen("tcp", address)
if err != nil {
log.Fatalf("Error %v", err)
}
fmt.Printf("Server is listening on %v ...", address)
s := grpc.NewServer()
helloworld.RegisterHelloWorldServiceServer(s, &server{})
s.Serve(lis)
}
As you can see, we have imported the service.pb.go that was generated by our shell script. Then we implemented the function HelloWorld – which returns the Response âhello world from go grpc.â Thus creating a gRPC server.
var helloword = require('./helloworld/service_pb');
var services = require('./helloworld/service_grpc_pb');
var grpc = require('grpc');
function main() {
var client = new services.HelloWorldServiceClient('localhost:50051',
grpc.credentials.createInsecure());
var request = new helloword.Request();
var user;
if (process.argv.length >= 3) {
user = process.argv[2];
} else {
user = 'world';
}
client.helloWorld(request, function(err, response) {
if(err) console.log('Node Client Error:', err)
else console.log('Node Client Message:', response.getMesssage());
});
}
main();
We have imported the service_pb.js file which was generated by our shell script. Add the gRPC server address and call the HelloWorld function and output response to console.
Run the server and make sure that the code works.Â
Now that our server is running, let’s make a call from our Node.js client:Â
 *Ignore the warning.
When we receive the console output âhello world from go grpc,â we can conclude that everything worked out as expected.
Thus, we have successfully created a gRPC server and client with a single proto definition and a shell script. This was a simple example of an RPC call returning a âhello worldâ text. But you can do almost anything with it. For example, you can create a CRUD microservice that performs add, get, update RPC calls, or an operation of your choice. You just have to define it once and run the shell script. You can also call one service from another, by creating clients wherever you want or in any language you want. This is an example of a perfect microservice architecture.
Hope this blog helped you build a microservice architecture with just Protocol Buffers, gRPC, a docker file, and a shell script. This architecture is language-independent which means that it suits multiple programming languages. Moreover, it is almost 7 times faster than the traditional REST server, with additional benefits of documentation and validations.Â
Not only is the architecture language-independent, this allows you to manage all the data structures in your organisation under a single repository, which can be a game changer. Protocol buffers also support import, inheritance, tags, etc.
You just have to follow these steps when you create a new microservice:
Now you are equipped to create multiple microservices, enable communication between microservices, or just use them as it is and also add a REST Proxy layer to build APIs. Remember that all services use a single framework irrespective of their programming language.
You can find the whole code repo here.
You can also publish the generated proto files of a service as packages for each language, for e.g. Node.js – npm, Python – pip, golang – GitHub, Java – Maven and so on. Run ânpm install helloworld-protoâ to call the .pb files. And if someone updates the Proto definition you just have to run ânpm update helloworld-protoâ.
Discover how CloudSEK's comprehensive takedown services protect your brand from online threats.
How to bypass CAPTCHAs easily using Python and other methods
What is shadow IT and how do you manage shadow IT risks associated with remote work?
Take action now
CloudSEK Platform is a no-code platform that powers our products with predictive threat analytic capabilities.
Digital Risk Protection platform which gives Initial Attack Vector Protection for employees and customers.
Software and Supply chain Monitoring providing Initial Attack Vector Protection for Software Supply Chain risks.
Creates a blueprint of an organization's external attack surface including the core infrastructure and the software components.
Instant Security Score for any Android Mobile App on your phone. Search for any app to get an instant risk score.