Microsoft.ServiceFabricApps.ClusterObserver.Windows.FrameworkDependent 2.1.14

ClusterObserver

ClusterObserver (CO) is a stateless singleton Service Fabric service that runs on one node in a cluster. CO observes cluster health (aggregated) and sends telemetry when a cluster is in Error or Warning. CO shares a very small subset of FabricObserver's (FO) code. It is designed to be completely independent from FO sources, but lives in this repo (and SLN) because it is very useful to have both services deployed, especially for those who want cluster-level health observation and reporting in addition to the node-level user-defined resource monitoring, health event creation, and health reporting done by FO. FabricObserver is designed to generate Service Fabric health events based on user-defined resource usage Warning and Error thresholds which ClusterObserver sends to your log analytics and alerting service.

By design, CO will send an Ok health state report when a cluster goes from Warning or Error state to Ok.

CO only sends telemetry when something is wrong or when something that was previously wrong recovers. This limits the amount of data sent to your log analytics service. Like FabricObserver, you can implement whatever analytics backend you want by implementing the IObserverTelemetryProvider interface. As stated, this is already implemented for both Azure ApplicationInsights and Azure LogAnalytics.

The core idea is that you use the aggregated cluster error/warning/Ok health state information from ClusterObserver to fire alerts and/or trigger some other action that gets your attention and/or some SF on-call's enagement via auto-creating a support incident (and an Ok signal would mean auto-mitigate the related incident/ticket).

You can change ClusterObserver configuration parameters by doing a versionless Application Parameter Upgrade. This means you can change settings for CO without having to redeploy the application or any packages.

Application Parameter Upgrade Example:

• Open an Admin Powershell console.

• Connect to your Service Fabric cluster using Connect-ServiceFabricCluster command.

• Create a variable that contains all the settings you want update:

$appParams = @{ "RunInterval" = "00:10:00"; "MaxTimeNodeStatusNotOk" = "04:00:00"; }

Then execute the application upgrade with

Start-ServiceFabricApplicationUpgrade -ApplicationName fabric:/ClusterObserver -ApplicationTypeVersion 2.1.12 -ApplicationParameter $appParams -Monitored -FailureAction rollback

Example Configuration:

<?xml version="1.0" encoding="utf-8" ?>
<Settings xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Section Name="ObserverManagerConfiguration">
    
    <Parameter Name="ObserverLoopSleepTimeSeconds" Value="30" />
    
    <Parameter Name="ObserverExecutionTimeout" Value="3600" />
    
    <Parameter Name="AsyncOperationTimeoutSeconds" Value="120" />
    
    <Parameter Name="ObserverLogPath" Value="cluster_observer_logs" />
    
    <Parameter Name="EnableVerboseLogging" Value="false" />
    <Parameter Name="EnableETWProvider" Value="true" />
    
    <Parameter Name="EnableTelemetry" Value="true" />
    
    <Parameter Name="TelemetryProvider" Value="AzureLogAnalytics" />
    
    <Parameter Name="AppInsightsInstrumentationKey" Value="" />
    
    <Parameter Name="LogAnalyticsWorkspaceId" Value="" />
    
    <Parameter Name="LogAnalyticsSharedKey" Value="" />
    
    <Parameter Name="LogAnalyticsLogType" Value="ClusterObserver" />
    
    <Parameter Name="ObserverShutdownGracePeriodInSeconds" Value="1" />
  </Section>
  
  <Section Name="ClusterObserverConfiguration">
    
    <Parameter Name="AsyncOperationTimeoutSeconds" Value="" MustOverride="true" />
    
    <Parameter Name="Enabled" Value="" MustOverride="true" />
    
    <Parameter Name="EnableVerboseLogging" Value="" MustOverride="true" />
    
    <Parameter Name="EmitHealthWarningEvaluationDetails" Value="" MustOverride="true" />
    
    <Parameter Name="MaxTimeNodeStatusNotOk" Value="" MustOverride="true" />
    
    <Parameter Name="RunInterval" Value="" MustOverride="true" />
    
    <Parameter Name="MonitorRepairJobs" Value="" MustOverride ="true" />
    
    <Parameter Name="EnableOperationalTelemetry" Value="" MustOverride="true" />
  </Section>
</Settings>

Example LogAnalytics Query

You should configure FabricObserver to monitor ClusterObserver, of course. 😃