Author: Brett Henderson

DigIO prides itself on being a learning organisation, striving to promote technology vitality and prepare our clients so they can evolve and be successful. Each year we run a series of Tech Radar workshops with a range of engineers across web, mobile, API, QA and platform domains.  They provide insights and perspectives from their diverse consulting experiences and research.

If you’d like to learn more about the DigIO Tech Radar and how it fits into our Tech Vitality process then Sangeeta Vishwanath, one of our Principal Engineers, describes it in detail in her article – Tech radar – Discover emerging tech trends.

In this piece, we will review some of the key themes to come out of our most recent API Tech Radar.

Key API Themes

There is a steady trend away from containers to serverless style deployments.  This continues to influence our technology choices.  The developer experience with serverless tooling leaves plenty of room for improvement.  Golang is becoming more compelling compared to Java and Kotlin and forcing us to think about the JVM-centric nature of Kafka.  We’re increasingly forced to choose a path with open standards on one side and proprietary cloud and SaaS offerings on the other.

API tools and techniques are evolving.  REST isn’t going anywhere any time soon but it has competition on its hands in the form of GraphQL and gRPC.  Each has different tradeoffs with discussions usually centred around varying requirements for flexibility, compatibility, versioning, performance and ease of use.

Investment Priorities

The primary output of our tech radar sessions is a prioritised list of technologies that we wish to invest in.  The following technologies were rated highly during these sessions.

Golang (Adopt)

Golang is one of our primary languages and a great fit for many of our solutions.  Advantages include great performance, pleasant developer experience, ease of learning, and good compatibility with serverless runtimes.  It’s increasingly taking over from JVM-based solutions.  We have reached a critical mass of skills availability and customer demand.

As the top-voted Adopt entry this year we will continue to focus on learning and development opportunities for the team.  We use a combination of publicly available training content to develop base skills as well as an internal training program to develop more advanced skills related to our line of work.  Given the level of interest we will also investigate whether it makes sense to invest in additional areas such as solution accelerators.

gRPC (Adopt)

We like gRPC due to its strong API contract management, approach to versioning and excellent performance.  We use it wherever possible but for pragmatic reasons tend not to use it as much as we’d like.  Not all clients are familiar with it or comfortable using it and GraphQL often makes more sense for client facing APIs.  Service to service communications provide the best use cases.

This year we will aim to adopt gRPC more broadly.  We have multiple success stories with it and think it should be more widely promoted.  We will provide additional opportunities for team members to learn the technology, use it effectively, and promote its adoption.

Kafka (Adopt) / AWS MSK Serverless (Assess)

We like Kafka and the event driven architecture it promotes.  Treating events as first class citizens and a long-term representation of the exposed state makes it much easier to unlock the value of data in an organisation.  However, while we use Kafka on many of our engagements, it can be difficult to introduce due the complexity of standing up and managing the platform.  This limits its use to larger engagements that can justify the effort and costs involved.  Both AWS MSK and Confluent Cloud are good products but neither are trivial to setup and Confluent Cloud in particular can be expensive.

We will spend more time assessing AWS MSK Serverless to understand if it reduces the barriers to adoption and can be widely recommended, this may lead to an accelerator if outcomes are positive.  We will continue to offer internal training for Kafka and this is becoming more attractive to team members as Kafka adoption increases across our projects.  We’re cautious about investing in solution accelerators until our concerns about barriers to adoption impacting future growth are addressed.

Kotlin Flow for APIs (Assess)

We regularly develop Kotlin APIs using asynchronous IO for improved performance under high load.  We usually use Project Reactor included in the Spring Framework to achieve this but it can be difficult to learn, implement effectively, and debug.  Kotlin Flow is already used widely for Android development and Spring includes support for it.

Some of the team have built proof of concept APIs using Kotlin Flow and think it should be used in favour of Project Reactor so we’ll socialise those learnings and assuming feedback is positive we’ll help the broader team to adopt it.

AWS Lambda local dev loop (Assess)

AWS Lambda is one of our primary deployment targets for implementing APIs.  However, the developer experience is not as mature as we’d like.  We are still finding that it’s difficult to implement a fast and efficient inner dev loop that allows developers to quickly iterate through dev/build/test cycles and there is no clear consensus on the right solution.  We are open to using cloud infrastructure for development but still find that running all tooling locally is the most efficient.

Assuming the use of CDK for application deployments, we will explore our options to see if there’s a common approach we can easily employ across all engagements.

Federated GraphQL (Assess)

With the majority of our solutions utilising microservice architectures in some form or another, and GraphQL becoming increasingly popular, we have a common need to aggregate many domains/services into a single GraphQL endpoint.  While we can do this using a GraphQL gateway talking to REST endpoints, there can be benefits to using GraphQL “all the way down”.  GraphQL federation is an evolving space and one we’re keen to explore more fully.

We plan to run a tech assessment activity using a realistic proof of concept to evaluate the options available and design considerations involved.  We’ll then share those learnings and ensure that our team is equipped to utilise it effectively.

Open API testing/verification (Assess)

OpenAPI is one of our sensible defaults and used almost without exception when building REST APIs.  We typically use contract first API development and want to ensure that all of our APIs remain conformant with the specification.  We have used multiple approaches such as relying on code generation, using contract testing, and automated schema validation to achieve this but don’t have a clear recommendation.

We want to spend more time sharing our project experiences in achieving this with a view to aligning on a common approach and possibly evolving existing solution accelerators such as https://github.com/digio/oaat to streamline implementation.

GraalVM & Spring Native (Assess)

JVM solutions are becoming increasingly difficult to recommend as we move to serverless runtimes requiring fast startup and efficient memory usage.

We are currently performing tech assessments on GraalVM and Spring Native to understand how it addresses those issues and any downsides they introduce.

API expand-contract pattern (Trial)

When it comes to API versioning, opinions are plentiful and we are unlikely to land on a one size fits all solution.  However, right now we do have some consensus amongst the team that the expand-contract pattern should be considered for use due to its ability to effectively sidestep the need to version APIs at all.  We have had promising experiences with it to date but are still cautious because we haven’t yet seen it used over a long period of time.  One concern is how we identify what fields a client is still dependent on, this is relatively easy to determine with GraphQL with its explicit field-level queries but more difficult with REST and gRPC.

We will closely monitor the projects that are currently using this pattern to see if we can recommend this more broadly.

What’s next?

These investment recommendations are a key input to our technology strategy. This strategy informs our recommendations to our clients and specifies the activities that we will perform for the next 6 months.  These activities include:

  • Identification, creation and delivery of training using a mix of online, study group and facilitated formats.  Revise communities of practice around technology domains.
  • Definition and execution of technology assessments as short focused projects which deliver findings and recommendations for adoption.
  • Nurturing of solution accelerators by providing owners with time and support to develop ideas and test them with regular checkpoints to decide whether further investment is required and justified.

Over the course of the year we’ll apply the outputs to our solutions before repeating the process again.