Home > Learn > Blog > Contract Testing: An In-Depth Guide

Contract Testing: An In-Depth Guide

Author

Date

Introduction to Contract Testing

In today’s world, if we look at modern applications, microservices implementation is done broadly across multiple systems. Testing these services has become widely important, to maintain reliability and stability across these developed systems. In addition to this, it is equally important to test these interactions across different services or components within a system to ensure that they adhere to their agreed-upon contracts. That is where the term “Contract Testing” comes into the picture. It is a technique that ensures that interactions between two separate systems comply with a mutually agreed specification. In the shift left approach, performing contract testing early in the software development life cycle helps to identify and fix costly and time-consuming issues quickly. This testing approach has proven to be valuable in ensuring that changes done to one component of the system do not negatively impact the other ones.

In the above image, we can actually see where contract tests reside within a test pyramid. They are part of service level tests. Before going deep into contract testing, we should learn about key differences between different service level test types namely API, Integration and Contract tests.

	Contract Tests	Integration Tests	API Tests
Definition	Contract tests bridge the gap between unit and integration testing by validating the agreements and contracts between interacting components or services.	Integration tests validate the interactions between two or more systems. It is kind of a functional testing that ensure all features continue to work as expected after introducing a change. They are instrumental in catching issues that unit tests can’t uncover.	API tests verifies the functionality and behaviour of API by testing its endpoints, request/response formats, authentication, error handling and other API specific aspects.
Objective	To provide faster feedback on API-dependent systems, by removing unnecessary defects and lowering dependency on end-to-end integration tests, as well as reducing the need for sophisticated test environments, and thereby reducing the maintenance work.	It takes holistic perspective, and prevent regression issues at a higher level by assessing how features synergize to create a product and how that product will interact with other software.	To validate the functionality, performance, and reliability of an API ensuring that it behaves as expected by handling various inputs correctly and producing desired outputs consistently.
Mechanism	This testing technique ensures proper collaboration between two applications by independently verifying the compliance of the messages they send or receive according to the shared understanding which is documented in a “contract”.	In an integration test, a number of requests or steps are executed to simulate real user behaviour, and at each one, assertions are employed to verify the expected functionality of the service and the corresponding response.	API tests operate by submitting requests to API end points with different input data and verifying the responses that are returned. This involves creating test scenarios , making API calls and analysing the outcomes to ensure that API adheres to its specifications and requirements.
Scope	Contract test centres around the external collaborator, which is API client interface on consumer side and corresponding receiver on the provider side such as controller.	They examine independent collaboration between two components, disregarding the overall system, with the aim of achieving a particular task or producing a specific outcome.	It includes verifying the functionality, performance, and security of the API endpoints, as well as testing various input scenarios and error conditions. In addition to this, API testing also involves proper handling of data formats, authentication, authorisation and conformance with API specifications.
SDLC Phase	It can be done at following stages Pre-commit phase during the build process. Can also occur at the Post-commit phase.	Performed during Pre-commit, build and post build in deployment environment.	Performed during Pre-commit, build and post build in deployment environment.

Basics

Before going deep into the concepts, let us understand the basic terminology in contract testing which includes Consumer and Provider. Let us see further about them in the coming sections.

Consumer: It is a service that performs the action of initiating a request to another service. The nature or type of the request being made doesn’t matter, as it can take various forms such as HTTP, RPC (Remote Procedure Call), message-broker, and so on. The primary focus here is one service reaching out to another service to make a request, regardless of the specific communication protocol.

Provider: It is a service that plays a crucial role in the testing process by responding to requests made by another service, often referred to as the “Consumer.” The Provider service acts as a simulated or real implementation of the API or service that the Consumer intends to interact with. Provider service is responsible for receiving and processing requests sent by the Consumer, and subsequently returning the expected responses based on predefined contracts or specifications.

Key Concepts in Contract Testing

As we learn about contract testing, it is equally important to know about the principles of contract testing, which includes the following:

Consumer-driven Contracts
Provider-driven Contracts

Consumer-driven Contract: This approach specifically deals with a pattern where the consumer of a service defines the contract that provider must adhere to. This means consumers need to specify a functionality that it expects from the provider. The provider is then responsible for service implementation as per the agreed contract with the consumer. With the consumer driven approach, we can ensure that the provider service is designed with the needs of the consumer in mind and also the communication gap between the consumer and provider can be avoided. This approach offers greater flexibility across distributed systems as they contribute towards independent service development by sticking to the agreed contract.

Provider-driven Contract: In this approach, the provider of service is responsible for defining the contracts that consumers must follow. This means the provider creates a contract that mainly depicts what functionality the consumer can expect from the service, and how the consumer should interact with it. This approach allows providers to play a major role to ensure that their service meets the needs of consumers and it is also reliable.

Types of Contract Testing Tools

Now, it is the time for us to look at the tools that are on offer for us to perform contract testing.

PACT is an open-source contract testing tool that enables consumer-driven contract testing between the services used in developing distributed systems. PACT offers a flexible approach in testing the service interactions and contributes to the services’ evolution independently without breaking each other’s changes. The key advantage of using PACT is it offers a flexible approach to test service interactions. This means engineers can define the contracts that describe the expected behaviour of each service in a way it is appropriate for their specific use case. This approach allows teams to test their services thoroughly and catch any issues that might arise when different services interact with each other.

Coming to Prism, it primarily focuses on testing the APIs by allowing engineers to create and manage contracts that define the expected behaviour of each API. These contracts can be used to check that the API is operating properly and it adheres to the agreed interface. Prism has the capability to generate mock API servers based on the defined contract. These mock servers can be later used to test client applications in isolation without worrying about external dependencies or relying on real API servers. Prism supports OpenAPI specification standards and integrates well with test frameworks like Mocha and Jest, which allows engineering teams to integrate Prism into their test environment. Another important benefit of using prism is, it provides real-time feedback which allows engineering teams to easily identify and resolve the issues. Overall, this is a valuable tool for engineering teams, which can be used to manage API contracts and can be used as a mock server.

Next in the list is Dredd, which is an open-source tool based on the command line, used to test API contracts. Dredd uses API specs defined using OpenAPI or API Blueprint format to perform contract testing. The way dredd works is, it sends requests to the API and verifies the response matches the expected behaviour as described in the contract. Dredd reads API descriptions and creates expectations based on the requests and responses as listed in API description document. Altogether, Dredd is a valuable tool used for testing APIs by providing a way to define and manage contracts, by integrating with popular programming languages.

Another tool that we have explored is Spring Cloud Contract which is java based. This tool helps engineers to implement tests at two layers, namely consumer-driven contracts and provider-driven contracts. The defined contracts set the expected behaviour of the API and the messages that are interchanged between the services. Spring cloud contract provides a way to generate stubs based on the defined contracts, which can be used by API consumers to test the application in isolation without depending on real api implementation.

Redocly CLI is another command line tool that was explored, which was again used to perform API contract testing. This specifically works with OpenAPI specs, to ensure that API is defined as per agreed contract. Redocly CLI provides a way to perform contract testing for the requirements specified in the OpenAPI specification and ensure that API is functioning correctly as per defined contract

Implementing Contract Testing

Implementing contract tests is different for each of the tools that we have explored. In general, the following points need to be considered in writing contract tests, irrespective of the tools being used.

The primary step in this process is to define the contract between the systems. The contract needs to specify what each system’s anticipated inputs, outputs, and behaviour will be. This will operate as a reference to get started with contract testing.

Develop test cases based on the agreed contract by including the input data and anticipated result. Document the system behaviour in the test case, which should cover all potential scenarios and edge cases that the system might encounter

After defining tests, now it is the time to set-up the test environment and run the tests and make sure that verification is happening as per the agreed contract. This is the time where engineers can make use of mocking capabilities provided by the tools and can replicate the real time production environment while running the tests. Whatever tests are defined, they should be automated and should be integrated into the development process and run in the CI pipeline to catch the issues early in the cycle.

Contract Testing Best Practices

It is recommended to use Contract testing when engineering teams go with API first approach. It should be basically implemented ahead of integration testing, to test the integration point between the different apis or microservices. After conducting a feasibility review, teams should focus on finding services that meet the requirements of the systems that are being developed, rather than looking for services based on specific dependencies and information. Take the advantage of contract testing while checking for irregularities in consumers workflow, identifying any issues in the service configuration.

Contract Testing Case Study

We investigated various options for contract testing. Taking the Shift Left methodology and addressing the problems associated with environmental instabilities, the major goal of this assessment is to develop a solution that is acceptable for testing API integration contracts without performing a full integration test suite. The below table gives us the understanding of the tools that we have assessed by considering several factors which includes ease of use , language support, licensing model, CI/CD integration, and community support. Also, we have developed some boilerplate code samples around each of these tools and integrated with CI/CD pipeline to figure out the right candidate. Overall, there was a good learning curve associated with this assessment and each of the tools were great in their area.

The below image gives us detailed information about usage of contract testing tools npm based from the past two years.

Conclusion

By rolling out contract testing into practice, it helps engineering teams to increase communication across different stakeholders, such as frontend and backend developers, or service providers and consumers. Contracts serve as a shared understanding and executable documentation, allowing teams to work independently and reduce coordination efforts. In addition to this, they are instrumental in identifying the issues early in the development lifecycle and also assist in catching service level compatibility issues before deploying the changes to the production environment. Contract testing allows us to perform hassle free deployments, as the scope of introducing breaking changes between the services is very minimal. Moreover, it provides flexibility for engineering teams to replace some extent of expensive end-to-end testing with contract tests wherever applicable. As software systems continue to evolve in complexity, contract testing is becoming an increasingly valuable practice for delivering reliable and robust and scalable applications. When selecting a contract testing tool, engineering teams must consider the factors that are listed in the above table. It is equally important to assess different available tools based on team requirements and select the one that best fits your needs.