KB 244 - PowerConnect Cloud Scalability, Resiliency and Delivery Semantics

KB 244 (CLOUD): PowerConnect Cloud Scalability, Resiliency and Delivery Semantics

Overview

PowerConnect Cloud is designed to collect observability data from multiple SAP Cloud applications and API endpoints concurrently and forward that data to observability platforms.

This article explains how PowerConnect Cloud handles concurrency, scaling, backpressure, failure recovery, persistence and delivery semantics in standalone and clustered deployments.

Concurrent SAP endpoint processing

A single PowerConnect Cloud installation can be configured to monitor many SAP Cloud applications, OData services and other SAP API endpoints.

Connections are handled concurrently using multithreading and connection pooling.

Each collection job runs independently using its own scheduled worker. As a result, a slow or temporarily unavailable SAP endpoint does not typically affect processing of other configured endpoints.

Failures, retries and timeouts associated with one endpoint are isolated to the corresponding collection job.

Concurrency and resource management

Concurrency is controlled through the PowerConnect Cloud scheduler and its worker threads.

The number of scheduler threads can be increased as the number of configured endpoints and overall collection volume grows.

The appropriate level of concurrency depends on factors including:

• number of configured collection jobs

• collection frequency

• SAP API response time

• volume of data returned

• available CPU and memory

• network throughput

• downstream throughput

As with any concurrent application, increasing thread counts does not provide unlimited scaling. The available resources of the host and the characteristics of the source and destination systems ultimately determine the practical capacity of an individual node.

Horizontal scaling and clustering

Where a single PowerConnect Cloud instance is not sufficient, the recommended scaling approach is to deploy PowerConnect Cloud as a cluster and add additional nodes.

The cluster coordinates job ownership between its members.

If a node leaves the cluster, the remaining members can take ownership of and execute the jobs previously assigned to that node. When the node rejoins the cluster, work is redistributed according to the cluster’s normal scheduling behaviour.

This provides both horizontal scalability and improved availability without requiring administrators to manually assign individual SAP endpoints to specific PowerConnect instances.

For deployment and configuration details, refer to the PowerConnect Cloud clustering documentation.

Backpressure and downstream outages

PowerConnect Cloud uses disk-backed queues to decouple collection from downstream delivery.

If the downstream observability platform becomes slow or temporarily unavailable, collected telemetry can be retained in the persistent queue and delivered when the destination becomes available again.

This prevents short downstream outages from requiring PowerConnect to discard data immediately or retain an unlimited amount of telemetry in process memory.

Queue capacity remains finite and should therefore be sized appropriately for the expected telemetry volume and the length of downstream outages the deployment is expected to tolerate.

Checkpointing and restart recovery

Collection jobs maintain persistent checkpoints (where applicable) representing their collection progress.

If the PowerConnect Cloud process is restarted, jobs resume from their stored checkpoints rather than simply beginning collection from the current time.

This allows collection to recover from planned restarts and many temporary failures without unnecessarily creating gaps in the collected data.

Cluster node failure

In a clustered deployment, the loss of an individual PowerConnect Cloud node does not cause all collection to stop.

The remaining cluster members continue operating and can execute the jobs that were previously owned by the unavailable node.

This provides resilience against individual process or host failures, assuming sufficient capacity remains within the surviving members of the cluster.

SAP endpoint failure

If an individual SAP endpoint becomes unavailable, the corresponding collection job may fail or time out.

Other collection jobs continue to operate independently.

PowerConnect applies its normal retry and checkpointing behaviour to the affected collection. Once the SAP endpoint becomes available again, collection can continue.

The exact data available after recovery is ultimately dependent on the historical retrieval and retention capabilities of the individual SAP API.

Destination failure

If the configured destination becomes unavailable, PowerConnect Cloud continues to use its disk-backed queues to retain telemetry awaiting delivery.

When the destination becomes available again, queued telemetry can be retried and forwarded downstream.

The duration of outage that can be tolerated depends on the size of the persistent queue and the rate at which telemetry is being generated.

Data-loss mitigation

PowerConnect Cloud uses several complementary mechanisms to minimise data loss:

• persistent collection checkpoints

• disk-backed delivery queues

• retries

• clustered job failover

• manual replay functionality

These mechanisms address different parts of the collection pipeline.

Checkpoints allow collection to resume from previously recorded progress.

Persistent queues protect telemetry that has already been collected but has not yet been successfully delivered downstream.

Clustering allows collection work to continue if an individual PowerConnect node becomes unavailable.

Replay functionality provides an additional recovery mechanism where a period of historical data needs to be collected again.

Replay

PowerConnect Cloud supports replay of a specified historical time window for supported endpoints.

If an operator believes data may have been missed during an outage or other failure scenario, the relevant time period can be manually replayed where the underlying SAP API supports historical retrieval.

Persistent state

A standalone PowerConnect Cloud deployment uses both persistent checkpoints and disk-backed queues.

These serve different purposes.

Checkpoints retain the progress of collection jobs so that processing can resume following restart or failure.

Disk-backed queues retain telemetry that PowerConnect has already collected but which has not yet been successfully delivered downstream.

This separation allows source collection and downstream delivery to recover independently.

Delivery semantics

PowerConnect Cloud does not claim strict end-to-end at-most-once, at-least-once or exactly-once delivery semantics across the complete SAP-to-observability pipeline.

For telemetry that PowerConnect has successfully retrieved from SAP, persistent queues, retries and checkpointing provide behaviour broadly consistent with at-least-once delivery. This means duplicate telemetry may be possible following retry or recovery scenarios.

However, the complete pipeline also depends on the behaviour of the source SAP APIs.

PowerConnect collects data using HTTP-based SAP APIs whose consistency, retention periods, pagination behaviour and historical-query capabilities are controlled by the corresponding SAP service.

PowerConnect cannot guarantee that every source event will always be exposed by an SAP API or remain available for subsequent retrieval.

For this reason, PowerConnect Cloud should be considered a best-effort observability collection system with durable recovery mechanisms rather than a transactional data replication system.

Is PowerConnect Cloud suitable for reconciliation or CDC use cases?

No.

PowerConnect Cloud is designed for observability and operational monitoring.

It is not a Change Data Capture (CDC), transactional replication or reconciliation system and should not be used where the business requirement is to prove that every source record has been captured exactly once.

PowerConnect includes retries, persistent queues, checkpointing, clustering and replay functionality to minimise the possibility of telemetry loss as far as reasonably possible.

These mechanisms cannot provide guarantees beyond those offered by the underlying SAP APIs.

Failure behaviour summary

Summary

PowerConnect Cloud is designed to tolerate common transient failures across both source SAP systems and downstream observability platforms.

Its resilience model combines independent concurrent collection jobs, configurable concurrency, clustering, persistent checkpoints, disk-backed queues, retries and replay.

These mechanisms provide strong practical protection against telemetry loss while recognising an important architectural limitation: PowerConnect cannot provide stronger guarantees than the SAP APIs from which it retrieves data.