Using central repository to manage generated files for gRPC

When you’re working in a large scale gRPC based project with shared proto files among few micro services, problem of versioning arises. Someone might edit and push a proto file to their service but you’re still working on an outdated version. This might lead to huge complications in the long run.

Let me provide a one simple solution to that problem with this sentence, “Treat your proto gen folder as a separate library”. This will help to maintain the proto definition in a single place as well as to maintain the generated files in a single place. I will explain the process in this article with maven.

This is the way

Let’s say we have two services named ServiceA and ServiceB which has two different proto files. We can follow theses steps,

1. Add a pom file to generate and manage the version of the project. You need to define the following plugin section to generate the required files for the proto.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>${proto.version}</version>
  <configuration>
    <source>${java.version}</source>
    <target>${java.version}</target>
    <generatedSourcesDirectory>${project.build.directory}/generated-sources/protobuf</generatedSourcesDirectory>
  </configuration>
</plugin>

Now we can define the project version and it will be treated as a whole library. When we build the project, the target should have the generated files.

2. Now we need to add this to git and manage the versions and releases so that people can download and build the required version to .m2 folder and use it in their projects. Further, you can publish this to maven central depending on your requirements. Most of the cases, building locally would do the job.

3. Make sure to inform your team on the versioning and backward compatibility strategy or this might make a mess in the long run.

Alternatives

For the record, there are alternatives to this library type approach such as,

1. Managing only the proto file updates in a central repository: This can be used to show that there are changes but the real problem is that we have to check it every day and generate the local files. This would be ideal for a structured project where the changes to the proto files are minimum with a low frequency.

2. Maintain the proto file in one repository only: This would be ideal for small scale projects where the functionality of the proto is minimum. Anyhow, this article will be useless for such projects in the first place.

3. Third party versioning system: You can use “protodep” kind of library to manage if you are using Golang.

I have spoken

In my experience working with gRPC, this central repository decision should be taken before the project start. Specially if it is a large scale and rapidly growing project. My advice to you all, “Think twice, code once…”

Happy coding!