Golang's efficient development model

Background

When I was working on Go server development in my company, I found that there was a lack of Go development best practices, which led to the following phenomena

1. confusion when developing with Go, not knowing how to start and how to work more efficiently.
2. the phenomenon of code duplication was more serious.
3. the code quality of the project was uneven, resulting in uneven quality of the delivered product.
4. black box product operation, poor observability, can run on the line.
5. code implementation test the level of R & D staff, but the top is after all a minority, often worse, and the top is not allowed to make mistakes.
6. one person is responsible for the entire feature development, once the staff left, code maintenance will be difficult on the hard.

When I was working on product development, I realized that the subsequent product features would become more and more complex, and if I didn’t pay attention to these problems, I was afraid that the code would become more and more difficult to maintain, the product features would be difficult to change, and the user needs would not be quickly met by the domino effect.

To stop this phenomenon early, explore a set of paradigms that allow for best practices in the Go space that result in the following

1. the code is easy to maintain, at least two people are familiar with the code implementation goals, and it is easy to iterate on code updates.
2. the code is well designed to avoid odd, excessive, and incomplete implementations.
3. product quality assurance, deliverables are highly observable and low bug rate.
4. Reduce the phenomenon of repeated wheel building, focus on business logic development, and improve R&D efficiency.

Therefore, we study the solution in terms of R&D effectiveness, explore the improvement from individual development behavior to team collaboration, and finally practice various methods in two aspects of R&D code quality control and R&D framework, and finally summarize the current model of efficient development, and continue to optimize the precipitation, hoping to verify the effect of the model through practice, and extend it to all scenarios using Go for server-side development to improve the whole team R&D We hope to verify the effectiveness of the model through practice, and extend it to all scenarios using Go for server-side development to improve the R&D efficiency of the whole team.

Introduction to the pattern

This pattern has three parts, which are standardized R&D specification, contract collaboration and unified development framework.

• Standardized R&D specification aims to enable R&D staff to reach a consensus contract and strictly comply with it in terms of code programming style, Git submission specification, and review mechanism when developing code, so as to be responsible for the quality of the product.
• Contract collaboration aims to form a contract-based collaboration model between R&D staff and external partners who need to invoke their services, so that both parties do not rely on each other to carry out their work and reduce their communication costs.
• Unified development framework aims to constrain R&D personnel to use a unified framework for development, precipitate various scenarios for optimization and evolution of the framework, achieve rapid reuse of scenarios in development, continuous iteration to optimize the implementation of each scenario and make solid accumulation for more scenarios in the future.

Pattern Practice-Go Server Side Development

All current practices in real development business.

Standardized development specifications

Code programming specification

When taking over the development of some components, we see that there are very different styles in the implementation of the code, which makes it difficult to read and understand, so we set down the programming specification of the constraint style, which can improve the readability of the code and the ease of maintenance of the code.

In the exploration of Go code programming specifications, there are more well-known programming specifications in the industry such as EffectiveGo and UberGoGuide.

• EffectiveGo, the official Go programming specification, has been seen and benefited from by almost all programmers who develop in Go, and is highly universal.
• UberGoGuide is the industry leader in developing in Go, and has open sourced many high quality open source software, such as Zap; its open source programming specification also extends on EffectiveGo and other programming specifications as a foundation.

From a comprehensive consideration, Uber has a Chinese translation version to get started quickly, and its company open source software quality implementation in accordance with the specification also side by side to prove the superiority of the specification, so choose UberGoGuide as a guide programming specification, but not as the only reference standard, but also consider the best practices according to the actual operation of the corresponding changes.

Git Commit Specification

When working in a team to develop a component, I see that Git commits are very random, with only a few brief descriptions, but no idea what was actually done. In the retrospective code problems, do not know what the commit has done, on the code maintenance caused some obstacles. It is estimated that the industry’s best uniform specification, Conventional Commit 1.0, will be used as a constraint.

Conventional Commit 1.0, a lightweight specification for commit information, provides some easy-to-understand rules for guiding how we write commit information, so this is also easy to be interpreted by machines, and combined with the SemVer versioning specification, it can be easily automated by tools to manage both version numbers and further constrain developers to After all, if the commit message is typed indiscriminately, it will directly affect the versioning.

Therefore, in practice, strict adherence to Conventional Commit 1.0 for commit information is required, and code commits can be rejected for those whose commit information is unclear.

Review Mechanism

In the original teamwork model, most of them developed their own modules and submitted them directly to the master branch, and then released them, which was fast but the quality of delivery was very worrying, after all, it tested the comprehensive level of R&D personnel, but after all, not everyone is a god, so it could not avoid the growth of the number of defects, not only to the overall code quality is worrying, even to the quality of the delivered product The quality of the delivered product is also not guaranteed, and even affects the reputation of the whole team. To solve these problems, CodeReview was introduced to control the process of code merging to code delivery, thus controlling the overall code quality.

The CodeView mechanism involves the roles of a committer and a reviewer. Each time a committer submits code, the code needs to be reviewed by a code reviewer, and the code can be merged into the master branch only after the review is passed, thus achieving the possibility of release.

With such a mechanism in place, the following effects can be achieved.

• Make the R&D staff have other perspectives to verify the code implementation solution and solution ideas in each submission to ensure certain objectivity.
• Reviewr role of the staff, the process needs to identify bugs, whether there are logical problems and whether there is coverage of the full edge of the scene, so as to ensure that the implementation of the idea is comprehensive enough.

The benefits of doing so are as follows.

• Knowledge sharing: The team can know the implementation of the code, the design used, etc. to share when conducting a Review within the team, which is conducive to the improvement of the team’s code level.
• Can find bugs earlier: Avoid finding bugs only after the feature is launched, so as to stop the damage in time.
• Ensure compliance: Different backgrounds of programmers lead to different styles of respective code, but the compliance of project code is constrained by specification standards.
• Enhance security: When professionals with security technology backgrounds participate in reviews, the level of security is very high.
• Enhance teamwork: When team members go together to solve a problem, this helps to enhance their respective OwnerShip thinking and the sense of belonging to the team.

The following conventions are also currently required.

1. Reviewr and Commiter on a submission, in the strict sense can not be the same person.
2. Comments need to be made in good faith, both sides need to discuss the case and rigorous logical reasoning, and only after the Reviewr agrees to terminate the Comment.
3. the code needs to be compiled and tested before submission.
4. a submission try to do not exceed 400 lines of core code.
5. a Review try to be less than 1 hour.
6. need to know to Reviewr clearly know the purpose of this submission to achieve the ultimate expectation is what.
7. if you are afraid to submit the code is not written well, then the code to achieve their own degree of recognition first.
8. single submission if the amount of code is too large or complex logic can set more than two Reviewer for Review.
9. Reviewer and Commiter have joint and several responsibilities, Commiter submitted code in production problems and loss of value, Reviewer also need to assume secondary responsibility.
10. Review mechanism can easily be ignored and bypassed, but the consequences are sometimes tragic, so it needs to be unanimously accepted and followed by the team to do well.

Contractual collaboration

PB documentation both contractual model

In the developer to API development to completion, often need to provide documentation to others to call, such a practice has the following disadvantages.

1. slow delivery of interface documents, interface requirements need to wait for the interface documents to be given in order to develop, which will increase the development risk for the demand side.
2. The interface is not updated in a timely manner, when the developer changes the API, 90% of the probability will forget to update the documentation, resulting in a lot of third-party calls need to communicate the situation, which increases the cost of communication between the two sides, reducing the development efficiency of both sides.

In order to solve the above problems, a model of both contract is implemented for Protobuf files (hereinafter referred to as PB files). This model follows the Google API guidelines, implements the corresponding communication protocol support, and complies with the gRPC API using the HTTP mapping function for JSON/HTTP support. Therefore, it is possible to define REST and RPC APIs by defining PB files, and manage API Schema by means of a repository similar to GoogleAPI.

Furthermore, combined with the rich Protoc plug-in can automate Swagger documents or generate code in different language types, as follows.

• Go HTTP API: generate Go version of Http related code through protoc-gen-go-http plugin
• Go gRPC API: Go version of gPRC-related code generation via the protoc-gen-go-grpc plugin
• Swagger documentation: generated via the protoc-gen-openapiv2 plugin

Therefore, the R&D staff can clearly know the definition of API, parameters, return content, etc. bin through PB files and can generate the corresponding code (Server and Client) based on PB files for their respective development, and can also directly generate Swagger documents to view and use, which greatly reduces the cost of communication.

The model also needs to comply with the following rules:

1. new contract changes need to be notified to each other
2. the old contract has been running for a period of time encountered BreakChange requirements, should consider the new contract, should not be changed in the original basis.
3. the new contract needs to be customized through the requirements side Review before continuing development, to avoid rework, but also to ensure the accuracy of the requirements.
4. contract as far as possible to clear and concise structure definition as the priority, for special needs and then use Map, PBValue, etc.

A unified development framework

Development Framework Selection

In the past programming experience, the accumulation of a framework will help to improve the overall development effectiveness, after all, the former has planted trees, the latter to multiply the cool, the former has figured out the use of the framework, there is enough confidence to guide future generations to develop, optimize the framework code and establish a complete set of surrounding ecology, can have the spare capacity to cope with the future complex business scenarios.

Because of the current team’s limited manpower of only four people, even if they have the will, they can not afford to do self-research framework that requires a lot of brain power and needs to accumulate precipitation work, so they gave up self-research framework, and choose to borrow open source products.

In the framework selection, from the current team demand scenario, the team needs a set of constraints to achieve the specification, ecological perfection, strong user autonomy, background performance requirements are not high and support microservices framework.

In the Go side, we pre-selected GVA, Gin and Kratos as the comparison and practice, and summarized the comparison of GVA, Gin and Kratos as follows

1. Gin, a basic lightweight high-performance Web framework , the implementation of more cases of scenarios , but the code implementation of unconstrained specifications need to rely on the developer to figure out , the universality is strong .
2. GVA, is a front and back-end supporting framework , the main solution is to quickly develop web applications , but the framework code implementation does not have good theoretical guidance and specification constraints , the overall use of the code quality is poor and difficult to maintain , the ability to deal with complex business needs is poor , the most criticized is the abuse of public variables .
3. Kratos, bilibli business validation products , supporting a sound microservices framework , design to follow the neat architecture and DDD ideas , the realization of the module coupling is low , user autonomy , the ability to deal with complex business needs , but the overall design concept tends to standard specification customization , the overall performance of the upper, you need to extend the module to achieve higher performance .

The final use of Kratos as a unified development framework, the specific decision based on the following

1. multi-protocol: default support for HTTP/gRPC, can be customized to extend the specification, support the use of gin to achieve the transport core.
2. API contract concept: support for PB file definition, can generate server-side, client-side code and Swagger documents, and provide online real-time documentation capabilities.
3. DDD idea of layered development: guide developers to develop in accordance with the domain-driven approach to development, so that the function is more focused, more concise implementation, and low coupling, such as replacing ClickHouse for SLS only need to change the infrastructure layer implementation can be.
4. framework structure modular and clear: almost covers the modules involved in the development of microservices framework, such as authentication, registration, self-monitoring , etc., and these modules can be customized to achieve strong adaptability.
5. highly scalable middleware design: extremely scalable middleware design , flexible expansion of various business middleware modules required.
6. framework code quality: the overall code quality of the upper layers, module dependencies decoupled, easy for users to read to get started, independent customization optimization feasible.
7. performance: the business now temporarily or in the future 1 to 2 years are mainly Web back-end applications and microservices, so the performance requirements are not demanding, and such as the need for high performance can also be achieved when the independent implementation of extension modules to achieve.

development framework sedimentation

The current group has a certain accumulation in Kratos framework, which improves a good foundation for the subsequent improvement of back-end development, as follows.

1. Layout rapid development: Relying on Kratos Layout capability, customize Layout templates in line with their own, which can be used quickly in the subsequent development. The current Layout implementation already contains the writing templates of each layer, Prom indicator integration, Opentelemetry full link integration, local cache implementation, etc.
2. Guidance on writing custom API documentation information: it has fully stepped on how to enrich the API documentation, for any custom information needs can be met.
3. guide to quickly use Kratos development: with Layout, to help quickly get started with the framework, rapid business development.
4. Generate front-end SDK through online API documentation: Quickly generate front-end SDK through Swagger-gen, no need to write SDK by hand for front-end, and strong maintainability. Later will also precipitate more scenarios to the framework and documentation to enrich the use of the framework scenarios to help develop more efficient.

Effectiveness in practice

The current practice of this pattern in internal projects has improved the following results

1. Bug situation: After this practice, the number of bugs at code level is less than 10 and the number of bugs at requirement level is less than 15 from the beginning of the project.
2. Front-end and back-end collaboration efficiency: The current communication and consultation between front-end and back-end is basically only in the requirement consultation, agreement definition and actual joint tuning stage, and the overall speed of delivering features is less than 2 weeks.
3. speed of feature completion: back-end service delivery to achieve the features, are less than 5 days a cycle, the project progress stuck point is not in the back-end implementation.
4. team cooperation atmosphere: the addition of review mechanism makes team members more cohesive and more willing to do knowledge sharing, which improves the overall ability of team members.
5. Delivery service self-monitoring perfection: the implementation of the whole link monitoring, improve the observability of the whole link in the front and back ends, locate the problem within less than 5 minutes.

The overall effect is in line with the expected design, but there is still much room for optimization such as API management, Error specification, CICD standardization, and will continue to precipitate optimization of the model to improve the delivery rate of subsequent features and future new projects, enhance developer happiness, and reduce development costs.