Jakub Heidtke logoJakub Heidtke
back to All projects

Professional

Camunda Sink

Overview

Camunda Sink connected event-driven business systems with Camunda workflows. It consumed configured events from the company messaging platform, transformed them into Camunda-specific integration patterns, and handled correlation, retries, timeouts, tenant rules and failed-event reprocessing. I contributed extensively to its architecture and evolution, including project restructuring, dynamic configuration, handler registration, multi-tenant behavior, back-office tooling, asynchronous global-message processing, reliability improvements and platform modernization.

My Role

  • Restructured the service into separate API, application, consumer, data and worker projects and introduced a Web API surface for operational configuration.
  • Implemented Azure Blob Storage-backed event configuration with automatic polling and reload-on-change behavior, allowing event routing changes without redeploying the service.
  • Built dynamic event handler registration and a handler factory capable of routing configured event types to correlation-message, signal or global-message processing.
  • Extended the Camunda integration with asynchronous global-message processing that updates variables across subscribed process instances and monitors Camunda batch completion.
  • Implemented tenant-aware event processing and configurable tenant suppression, including API, back-office and audit behavior.
  • Improved reliability through differentiated Polly retry policies, timeout handling, circuit-breaker behavior and safer failed-event storage and reprocessing.
  • Built a Blazor back-office application and generated API client for inspecting and managing handled-event configuration.
  • Implemented a maintenance service for cleaning expired Dead Events Storage records and associated blobs, including Azurite-backed integration tests and CI/CD support.
  • Migrated the multi-project service and deployment tooling to .NET 7.

Architecture & Technology

  • .NET worker services consuming business events and commands through Ori.Messaging over Azure Service Bus
  • ASP.NET Core Web API for viewing and updating handled-event configuration
  • Dynamic event handler factory supporting Camunda correlation messages, signals and global variable updates
  • Camunda OpenAPI client integration for message correlation, signal delivery and asynchronous process-variable batch updates
  • Azure Blob Storage-backed runtime configuration with ETag-based reload detection and IOptionsMonitor change propagation
  • Azure Table Storage, Blob Storage and Queue Storage for failed-event persistence and reprocessing
  • Polly retry and circuit-breaker policies with integration-specific timeout and retry behavior
  • Server-side Blazor back-office UI with generated API client
  • Application Insights telemetry, audit logging, Docker and Azure DevOps CI/CD

Engineering Challenges

  • Route many event types into different Camunda integration patterns while keeping behavior configurable and avoiding large amounts of event-specific infrastructure code.
  • Apply configuration changes at runtime without redeploying the service or leaving consumers registered with stale handler definitions.
  • Handle Camunda operations with very different execution characteristics, from synchronous correlation messages to long-running signals and asynchronous batch updates.
  • Avoid unsafe retries for operations where Camunda could still be processing a timed-out request and retrying could create duplicate side effects.
  • Support multi-tenant event processing while allowing selected tenants to be suppressed dynamically at configuration level.
  • Preserve failed-event data for investigation and reprocessing while safely managing the lifecycle of metadata and associated blob payloads.

Decisions & Trade-offs

  • Model event handling as configurable handler types rather than implementing separate hard-coded pipelines for every event. A handler factory resolved event and handler type into correlation-message, signal or global-message processing.
  • Store handled-event configuration in Azure Blob Storage and monitor ETags for changes. This enabled operational routing changes without rebuilding or redeploying the service.
  • Restart and re-register event consumers when configuration changed instead of attempting to mutate active handler registrations in place. This kept runtime behavior aligned with the latest configuration.
  • Treat long-running Camunda signals and global-message operations differently from standard correlation messages. Some timeout scenarios were deliberately moved to Dead Events Storage rather than retried automatically to reduce the risk of duplicate Camunda-side work.
  • Run global-message delivery asynchronously and monitor Camunda batch completion because updating multiple subscribed process instances could outlive the normal message-handler execution window.
  • Keep failed-event cleanup as a separate maintenance process and delete blob payloads only after the corresponding storage metadata had been removed successfully, reducing the risk of inconsistent cleanup.

Results & Impact

  • Enabled configuration-driven routing of business events into multiple Camunda integration patterns.
  • Allowed handled-event configuration to be changed at runtime through Blob Storage-backed configuration and dynamic consumer re-registration.
  • Extended the integration beyond direct message correlation to tenant-aware signals and asynchronous process-variable updates.
  • Improved production reliability with tailored retry, timeout and failed-event handling for different Camunda operations.
  • Provided internal operational tooling for inspecting and managing event configuration through a Web API and Blazor back office.
  • Added lifecycle management for failed-event storage through a dedicated maintenance service with integration-test coverage.
  • Modernized the service architecture and runtime through project restructuring and migration to .NET 7.