Is Kalix the next step to speed up and simplify software development?

Krzysztof Borowski

What is Kalix?

High-performance microservices and APIs with no operations required

The promise

https://go.lightbend.com/kalix-executive-briefing

Diagram showing the parts of a mature flower. In this example the perianth is separated into a calyx (sepals) and corolla (petals)

brew install kalix

kalix auth login

kalix projects new <projectname> "project description" --region=<region>

kalix config set project <projectname>

kalix docker add-credentials --docker-server <my-server> \ 
  --docker-username <my-username> \ 
  --docker-email <my-email> \ 
  --docker-password <my-password>

Setup the project

PhotoTrip App

Create personal map
Add new visited places
Add photo links to the place
Display system stats

Architecture oversimplified

#!/bin/sh

sbt Docker/publish -Ddocker.username=liosedhel &&
docker tag liosedhel/phototrip:$1 &&
docker push liosedhel/phototrip:$1 &&

kalix service deploy phototrip liosedhel/phototrip:$1
kalix services expose my-service

Deploy!

Management

kalix projects list
kalix logs --raw phototrip
kalix svc list
kalix svc components list phototrip

Creating a PhotoTrip App

Aggregates = Entities

Entity

Can be referenced from outside (ID).
A cluster of domain objects that can be treated as a single unit.
Aggregates domain objects and theirs behaviour.
Every change of the state within aggregate is transactional.

Kalix Entities

https://docs.kalix.io/concepts/state-model.html#_choosing_a_state_model

Value Entity

Value Entity - Scaling

Value Entity - WorldMap

message CreateWorldMap {
  string map_id = 1 [(kalix.field).entity_key = true];
  string creator_id = 2;
  string description = 3;
}

service WorldMapService {
  option (kalix.codegen) = {
    value_entity: {
      name: "com.virtuslab.phototrip.worldmap.domain.WorldMapValueEntity"
      entity_type: "worldmap"
      state: "com.virtuslab.phototrip.worldmap.domain.WorldMapState"
    }
  };
  option (kalix.service).acl.allow = { principal: ALL };

  rpc Create (CreateWorldMap) returns (google.protobuf.Empty);
  rpc Get (GetWorldMap) returns (CurrentWorldMap) {
    option (google.api.http) = {
      get: "/worldmaps/{worldmap_id}"
    };
  };
}

Value Entity - WorldMap

class WorldMapValueEntity(context: ValueEntityContext) 
  extends AbstractWorldMapValueEntity {
    
  override def emptyState: WorldMap = WorldMap()

  override def create(
    currentState: WorldMap, 
    createWorldMap: CreateWorldMap
  ): ValueEntity.Effect[Empty] = {
    if(!isValid(createWorldMap)) {
      effects.error(s"Invalid $createWorldMap command")
    } else {
      effects
      	.updateState(toWorldMap(createWorldMap))
        .thenReply(Empty.defaultInstance)
    }
  }

  override def get(
    currentState: WorldMap, 
    getWorldMap: GetWorldMap
  ): ValueEntity.Effect[CurrentWorldMap] = {
    if (currentState == emptyState) {
      effects.error(s"Map ${getWorldMap.worldmapId} does not exist")
    } else {
      effects.reply(toCurrentWorldMap(currentState))
    }
  }
  ...
}

Event Sourced Entity

Event Sourced Entity - Place

service PlaceService {
  option (kalix.codegen) = {
    event_sourced_entity: {
      name: "com.virtuslab.phototrip.place.domain.PlaceEventSourcedEntity"
      entity_type: "place"
      state: "com.virtuslab.phototrip.place.domain.Place"
      events: [
        "com.virtuslab.phototrip.place.domain.PlaceCreated",
        "com.virtuslab.phototrip.place.domain.PhotoLinkAdded"
      ]
    }
  };
  option (kalix.service).acl.allow = { principal: ALL };

  rpc CreatePlace (CreateNewPlace) returns (google.protobuf.Empty);
  rpc AddPhotoLink (AddPhotoLinkUrl) returns (google.protobuf.Empty);
  rpc Get (GetPlace) returns (CurrentPlace) {
    option (google.api.http) = {
      get: "/places/{place_id}"
    };
  };
}

Event Sourced Entity - Place

class PlaceEventSourcedEntity(context: EventSourcedEntityContext)
  extends AbstractPlaceEventSourcedEntity {

  override def emptyState: Place = Place()

  override def createPlace(
    currentState: Place, 
    createNewPlace: CreateNewPlace
  ): EventSourcedEntity.Effect[Empty] =
    effects.emitEvent(
      PlaceCreated(
        createNewPlace.placeId, 
        createNewPlace.mapId, 
        createNewPlace.description, 
        createNewPlace.coordinates
      )
    ).thenReply(_ => Empty.defaultInstance)

  override def placeCreated(
    currentState: Place, 
    placeCreated: PlaceCreated
  ): Place =
    Place(
      placeCreated.placeId, 
      placeCreated.mapId, 
      placeCreated.description, 
      placeCreated.coordinates, 
      Nil
    )

Replicated Entity

Replicated Entity - User

service UserService {
  option (kalix.codegen) = {
    replicated_entity: {
      name: "com.virtuslab.phototrip.user.domain.UserReplicatedEntity"
      entity_type: "user"
      replicated_register: {
        value: "com.virtuslab.phototrip.user.domain.User"
      }
    }
  };
  option (kalix.service).acl.allow = { principal: ALL };
  rpc Create (CreateNewUser) returns (google.protobuf.Empty) {
    option (google.api.http) = {
      post: "/users"
    };
  };
  rpc Get (GetUser) returns (com.virtuslab.phototrip.user.domain.User) {
    option (google.api.http) = {
      get: "/users/{user_id}"
    };
  };
}

Replicated Entity - User

class UserReplicatedEntity(context: ReplicatedEntityContext)
  extends AbstractUserReplicatedEntity {

  override def emptyValue: User = User()

  override def create(
    currentData: ReplicatedRegister[User],
    createNewUser: CreateNewUser
  ): ReplicatedEntity.Effect[Empty] = {
    effects
      .update(
        currentData.set(
          User(createNewUser.userId, createNewUser.email, createNewUser.nick)
        )
      ).thenReply(Empty.defaultInstance)
  }

  def get(
    currentData: ReplicatedRegister[User],
    getUser: api.GetUser
  ): ReplicatedEntity.Effect[User] = {
    currentData.get match {
      case Some(user) => effects.reply(user)
      case None =>
        effects.error("User does not exist", io.grpc.Status.Code.NOT_FOUND)
    }
  }
}

Views

View - World map by user

service WorldMapByUserId {
  option (kalix.codegen) = {
    view: {}
  };

  rpc UpdateWorldMap(domain.WorldMapState) returns (WorldMapView) {
    option (kalix.method).eventing.in = {
      value_entity: "worldmap"
    };
    option (kalix.method).view.update = {
      table: "worldmaps"
      transform_updates: true
    };
  }

  rpc GetWorldMaps(ByUserIdRequest) returns (stream WorldMapView) {
    option (kalix.method).view.query = {
      query: "SELECT * FROM worldmaps WHERE creator_id = :user_id"
    };
    option (google.api.http) = {
      get: "/worldmap/user/{user_id}"
    };
  }

}

View - World map by user

class WorldMapByUserIdView(context: ViewContext) 
  extends AbstractWorldMapByUserIdView {

  override def emptyState: WorldMapView = WorldMapView.defaultInstance

  override def updateWorldMap(
    state: WorldMapView,
    worldMapState: WorldMapState
  ): UpdateEffect[WorldMapView] = {
    effects.updateState(
      WorldMapView(
        worldMapState.mapId,
        worldMapState.creatorId,
        worldMapState.description
      )
    )
  }

}

PubSub

aka glue it together

PubSub Configuration

$ kalix projects config set broker \
	--broker-service kafka \
    --broker-config-file <kafka-broker-config-file>
    
    
ccloud kafka topic create TOPIC_ID

Publishing to PubSub

service WorldMapAndPlacesEventsToTopic {
  option (kalix.codegen) = {
    action: {}
  };

  rpc PlaceCreation (com.virtuslab.phototrip.place.domain.PlaceCreated) 
  	returns (PlaceCreatedMessage) {
    option (kalix.method).eventing.in = {
      event_sourced_entity: "place"
      ignore_unknown: true
    };
    option (kalix.method).eventing.out = {
      topic: "analytics-events"
    };
  }

  rpc WorldMapUpdate (com.virtuslab.phototrip.worldmap.domain.WorldMapState) 
  	returns (WorldMapUpdatedMessage) {
    option (kalix.method).eventing.in = {
      value_entity: "worldmap"
    };
    option (kalix.method).eventing.out = {
      topic: "analytics-events"
    };
  }

}

Publishing to PubSub

class WorldMapAndPlacesEventsToTopicAction(creationContext: ActionCreationContext)
  extends AbstractWorldMapAndPlacesEventsToTopicAction {

  override def placeCreation(
    placeCreated: PlaceCreated
  ): Action.Effect[PlaceCreatedMessage] = {
    effects.reply(
      PlaceCreatedMessage(
        placeCreated.placeId,
        placeCreated.mapId,
        placeCreated.description,
        placeCreated.coordinates.map(c => Coordinates(c.latitude, c.latitude))
      )
    )
  }
  
  override def worldMapUpdate(
   worldMapState: WorldMapState
  ): Action.Effect[WorldMapUpdatedMessage] = {
    effects.reply(
      WorldMapUpdatedMessage(
        worldMapState.mapId,
        worldMapState.creatorId,
        worldMapState.description
      )
    )
  }
}

Reading from PubSub

service ReadFromAnalyticsTopic {
  option (kalix.codegen) = {
    action: {}
  };

  rpc PlaceCreation (com.virtuslab.phototrip.worldmap.actions.PlaceCreatedMessage) 
      returns (google.protobuf.Empty) {
    option (kalix.method).eventing.in = {
      topic: "analytics-events"
    };
  }

  rpc WorldMapUpdate (com.virtuslab.phototrip.worldmap.actions.WorldMapUpdatedMessage) 
      returns (google.protobuf.Empty) {
    option (kalix.method).eventing.in = {
      topic: "analytics-events"
    };
  }

}

Reading from PubSub

class ReadFromAnalyticsTopicAction(creationContext: ActionCreationContext) 
  extends AbstractReadFromAnalyticsTopicAction {

  private val log = LoggerFactory.getLogger(classOf[ReadFromAnalyticsTopicAction])

  override def placeCreation(
    placeCreatedMessage: PlaceCreatedMessage
  ): Action.Effect[Empty] = {
    log.info(s"PubSub:Read $placeCreatedMessage")
    effects.forward(
      components.statsValueEntity.placeCreation(
        NewPlace(StatsValueEntity.key, placeCreatedMessage.placeId)
      )
    )
  }
  override def worldMapUpdate(
   worldMapUpdatedMessage: WorldMapUpdatedMessage
  ): Action.Effect[Empty] = {
    log.info(s"PubSub:Read $worldMapUpdatedMessage")
    effects.forward(
      components.statsValueEntity.mapUpdate(
        NewMap(StatsValueEntity.key, worldMapUpdatedMessage.mapId)
      )
    )
  }
}

Managing Components

kalix svc c list-entity-ids phototrip \
    com.virtuslab.phototrip.place.api.PlaceService

kalix svc c get-state phototrip \
    com.virtuslab.phototrip.place.api.PlaceService Place1 --output json

kalix svc c list-events phototrip \
    com.virtuslab.phototrip.place.api.PlaceService Place1 --output json

kalix svc views list phototrip

Developer Experience

Developer Experience - Writing the code

Protobuf first approach

Pros

All built components have clearly stated APIs.
Code stubs and test stubs are auto-generated based on the proto definition.
gRPC and REST endpoints “for free”.

Cons

You need to learn proto and gRPC DSL first.
Poor experience when refactoring.
Lost Scala-ish typesafety.
Runtime errors.
Protobuf serde only (or some workaround with proto and Any type).

Abstract high-level components

Pros

You can jump straight to the business logic (no infra layer).
Scalable, following the Single-Writer principle and optionally event sourced, entities available out of the box.
Glue it with configuration.

Cons

Quite specific components, different to what most programmers are used to.
Some useful features are under development (like view joins, sagas).
Limited configuration options or performance tweaking.

Writing tests

Pros

Easy to start and extend pregenerated unit tests.
Easy to start and test manually particular Kalix service.

Cons

Integration tests only on dev environment (costly in the long run) (* soon it should be addressed by LB).
Local testing is done with in-memory database only.

Developer Experience - managing the Kalix services

Running services locally

Pros

Relatively straightforward to run one service locally.
It can help to catch most of the potential runtime errors.

Cons

Stateless only - can’t help with testing backward compatibility.
Tricky to run multiple services or configure and test the service-to-service connection.

docker-compose up -d

sbt run

Managing Kalix projects and services via CLI

Pros

One tool to invoke any Kalix related operations.
Allows to view your data.
Under active development.

Cons

Yet another CLI syntax to learn.
Under active development.
Views have to be browsed. through gRPC calls directly.

kalix svc start phototrip
kalix svc expose phototrip
kalix svc components list phototrip
kalix svc views list phototrip
kalix svc delete phototrip

Prod deployment

Pros

Easy to deploy a new version.
Zero downtime deployment, scaling, whole provisioning is out of the box.
Extremaly low time to market.

Cons

No schema backward compatibility check mechanism.
Limited visibility into the platform - in case of cryptic error the ticket is required.

kalix service deploy \
	phototrip liosedhel/phototrip:55

Managing persistent data

Pros

You don’t have to manage the database.
You don’t have to implement infrastructure level (both database and messaging related).
Infra solution is well adjusted to Kalix programming model.

Cons

You don’t have direct access to your data.
~~You can’t do your own backups~~ (feature in progress).
You can break your data consistency (no schema check).
Data migration only through protobuf schema adjustment way.

Performance and scalability

Pros

Polished and proven akka technology wrapped in convenient high-level API deployed with Kubernetes capabilities.
You get automatic horizontal scalability without any manual setup or integrations.
Each entity state is loaded to the memory - effectively serve as cache layer.

Cons

Each recently used entity state is loaded to the memory - services can easily become memory intensive.
To achieve optimal performance, you need to know the underlying computation technology (e.g. to not block inside entity logic as it might block underlying dispatcher).
~~There is no way to tweak autoscaling capabilities~~.

Integrating external services

Pros

Integration through high level contract first API.
External services can be accessed and integrated through Kalix actions.
Kalix services by default can call any publically available service

Cons

Only possible through additional abstraction level (wrapping external calls with effects).
Has to be integrated over a public internet connection.

Other considerations

Pricing

You can get a custom deal, see https://discuss.kalix.io/t/minimum-per-service-cost-on-pay-as-you-go/163.

Additionally, the feature to put some upper bound or tweak autoscaling service capability is on the way - this should reduce unexpected costs in case of unpredicted service traffic.

Subjective: most important Kalix missing capabilities

IAM integration - no way to integrate directly with existing organisation accounts.
~~No direct integration network (e.g. VPC Network Peering) between Kalix platform and client infrastructure~~ (available with custom deal).
~~Limited views capabilities~~ (features in progress).
~~Scalability only within a particular region~~ (feature in progress).
~~No external data backup~~ (feature in progress)

Is Kalix a next step in software development?

Useful materials

Source code: https://github.com/liosedhel/kalix-phototrip
https://www.kalix.io/
https://docs.kalix.io/index.html or kalix docs
https://discuss.kalix.io/