Contract Testing: An In-Depth Guide
Introduction to Contract Testing
|Contract Tests||Integration Tests||API Tests|
|Definition||Contract tests bridge the gap between unit and integration testing by validating the agreements and contracts between interacting components or services.||Integration tests validate the interactions between two or more systems. It is kind of a functional testing that ensure all features continue to work as expected after introducing a change. They are instrumental in catching issues that unit tests can’t uncover.||API tests verifies the functionality and behaviour of API by testing its endpoints, request/response formats, authentication, error handling and other API specific aspects.|
|Objective||To provide faster feedback on API-dependent systems, by removing unnecessary defects and lowering dependency on end-to-end integration tests, as well as reducing the need for sophisticated test environments, and thereby reducing the maintenance work.||It takes holistic perspective, and prevent regression issues at a higher level by assessing how features synergize to create a product and how that product will interact with other software.||To validate the functionality, performance, and reliability of an API ensuring that it behaves as expected by handling various inputs correctly and producing desired outputs consistently.|
|Mechanism||This testing technique ensures proper collaboration between two applications by independently verifying the compliance of the messages they send or receive according to the shared understanding which is documented in a “contract”.||In an integration test, a number of requests or steps are executed to simulate real user behaviour, and at each one, assertions are employed to verify the expected functionality of the service and the corresponding response.||API tests operate by submitting requests to API end points with different input data and verifying the responses that are returned. This involves creating test scenarios , making API calls and analysing the outcomes to ensure that API adheres to its specifications and requirements.|
|Scope||Contract test centres around the external collaborator, which is API client interface on consumer side and corresponding receiver on the provider side such as controller.||They examine independent collaboration between two components, disregarding the overall system, with the aim of achieving a particular task or producing a specific outcome.||It includes verifying the functionality, performance, and security of the API endpoints, as well as testing various input scenarios and error conditions. In addition to this, API testing also involves proper handling of data formats, authentication, authorisation and conformance with API specifications.|
|SDLC Phase||It can be done at following stages Pre-commit phase during the build process. Can also occur at the Post-commit phase.||Performed during Pre-commit, build and post build in deployment environment.||Performed during Pre-commit, build and post build in deployment environment.|
Before going deep into the concepts, let us understand the basic terminology in contract testing which includes Consumer and Provider. Let us see further about them in the coming sections.
Consumer: It is a service that performs the action of initiating a request to another service. The nature or type of the request being made doesn’t matter, as it can take various forms such as HTTP, RPC (Remote Procedure Call), message-broker, and so on. The primary focus here is one service reaching out to another service to make a request, regardless of the specific communication protocol.
Provider: It is a service that plays a crucial role in the testing process by responding to requests made by another service, often referred to as the “Consumer.” The Provider service acts as a simulated or real implementation of the API or service that the Consumer intends to interact with. Provider service is responsible for receiving and processing requests sent by the Consumer, and subsequently returning the expected responses based on predefined contracts or specifications.
Key Concepts in Contract Testing
As we learn about contract testing, it is equally important to know about the principles of contract testing, which includes the following:
- Consumer-driven Contracts
- Provider-driven Contracts
Consumer-driven Contract: This approach specifically deals with a pattern where the consumer of a service defines the contract that provider must adhere to. This means consumers need to specify a functionality that it expects from the provider. The provider is then responsible for service implementation as per the agreed contract with the consumer. With the consumer driven approach, we can ensure that the provider service is designed with the needs of the consumer in mind and also the communication gap between the consumer and provider can be avoided. This approach offers greater flexibility across distributed systems as they contribute towards independent service development by sticking to the agreed contract.
Provider-driven Contract: In this approach, the provider of service is responsible for defining the contracts that consumers must follow. This means the provider creates a contract that mainly depicts what functionality the consumer can expect from the service, and how the consumer should interact with it. This approach allows providers to play a major role to ensure that their service meets the needs of consumers and it is also reliable.
Types of Contract Testing Tools
Now, it is the time for us to look at the tools that are on offer for us to perform contract testing.
PACT is an open-source contract testing tool that enables consumer-driven contract testing between the services used in developing distributed systems. PACT offers a flexible approach in testing the service interactions and contributes to the services’ evolution independently without breaking each other’s changes. The key advantage of using PACT is it offers a flexible approach to test service interactions. This means engineers can define the contracts that describe the expected behaviour of each service in a way it is appropriate for their specific use case. This approach allows teams to test their services thoroughly and catch any issues that might arise when different services interact with each other.
Coming to Prism, it primarily focuses on testing the APIs by allowing engineers to create and manage contracts that define the expected behaviour of each API. These contracts can be used to check that the API is operating properly and it adheres to the agreed interface. Prism has the capability to generate mock API servers based on the defined contract. These mock servers can be later used to test client applications in isolation without worrying about external dependencies or relying on real API servers. Prism supports OpenAPI specification standards and integrates well with test frameworks like Mocha and Jest, which allows engineering teams to integrate Prism into their test environment. Another important benefit of using prism is, it provides real-time feedback which allows engineering teams to easily identify and resolve the issues. Overall, this is a valuable tool for engineering teams, which can be used to manage API contracts and can be used as a mock server.
Next in the list is Dredd, which is an open-source tool based on the command line, used to test API contracts. Dredd uses API specs defined using OpenAPI or API Blueprint format to perform contract testing. The way dredd works is, it sends requests to the API and verifies the response matches the expected behaviour as described in the contract. Dredd reads API descriptions and creates expectations based on the requests and responses as listed in API description document. Altogether, Dredd is a valuable tool used for testing APIs by providing a way to define and manage contracts, by integrating with popular programming languages.
Another tool that we have explored is Spring Cloud Contract which is java based. This tool helps engineers to implement tests at two layers, namely consumer-driven contracts and provider-driven contracts. The defined contracts set the expected behaviour of the API and the messages that are interchanged between the services. Spring cloud contract provides a way to generate stubs based on the defined contracts, which can be used by API consumers to test the application in isolation without depending on real api implementation.
Redocly CLI is another command line tool that was explored, which was again used to perform API contract testing. This specifically works with OpenAPI specs, to ensure that API is defined as per agreed contract. Redocly CLI provides a way to perform contract testing for the requirements specified in the OpenAPI specification and ensure that API is functioning correctly as per defined contract
Implementing Contract Testing
Implementing contract tests is different for each of the tools that we have explored. In general, the following points need to be considered in writing contract tests, irrespective of the tools being used.
The primary step in this process is to define the contract between the systems. The contract needs to specify what each system’s anticipated inputs, outputs, and behaviour will be. This will operate as a reference to get started with contract testing.
Develop test cases based on the agreed contract by including the input data and anticipated result. Document the system behaviour in the test case, which should cover all potential scenarios and edge cases that the system might encounter
After defining tests, now it is the time to set-up the test environment and run the tests and make sure that verification is happening as per the agreed contract. This is the time where engineers can make use of mocking capabilities provided by the tools and can replicate the real time production environment while running the tests. Whatever tests are defined, they should be automated and should be integrated into the development process and run in the CI pipeline to catch the issues early in the cycle.
Contract Testing Best Practices
Contract Testing Case Study
The below image gives us detailed information about usage of contract testing tools npm based from the past two years.
By rolling out contract testing into practice, it helps engineering teams to increase communication across different stakeholders, such as frontend and backend developers, or service providers and consumers. Contracts serve as a shared understanding and executable documentation, allowing teams to work independently and reduce coordination efforts. In addition to this, they are instrumental in identifying the issues early in the development lifecycle and also assist in catching service level compatibility issues before deploying the changes to the production environment. Contract testing allows us to perform hassle free deployments, as the scope of introducing breaking changes between the services is very minimal. Moreover, it provides flexibility for engineering teams to replace some extent of expensive end-to-end testing with contract tests wherever applicable. As software systems continue to evolve in complexity, contract testing is becoming an increasingly valuable practice for delivering reliable and robust and scalable applications. When selecting a contract testing tool, engineering teams must consider the factors that are listed in the above table. It is equally important to assess different available tools based on team requirements and select the one that best fits your needs.