Czy wszyscy zdrowi?

Badanie kondycji aplikacji ASP.NET Core

Mateusz
Turzyński

SRE

Badanie kondycji?

Jak szybko zdiagnozować awarię?

Badanie kondycji pozwala

  • Odpowiednio reagować w przypadku awarii
  • Budować systemy, które potrafią naprawić się same

Health Checks

na ratunek

https://example.com/healthz

Status: 200 OK

https://example.com/healthz

{
   "status":"Unhealthy",
   "totalDuration":"00:00:01.1946607",
   "entries":{
      "sqlserver":{
         "data": {
         
         },
         "description":"Cannot open server (...) requested by the login.",
         "duration":"00:00:01.0638395",
         "exception":"Cannot open server (...)",
         "status":"Unhealthy",
         "tags":[
            
         ]
      }
   }
}

Sonda zwracająca szczegółowe dane

Implementacja w ASP.NET Core

Podstawowa konfiguracja

using Microsoft.AspNetCore.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers();

builder.Services.AddHealthChecks();	

var app = builder.Build();

app.MapHealthChecks("/healthz");

// Configure the HTTP request pipeline.

app.UseAuthorization();

app.MapControllers();

app.Run();

Dodawanie sond

Dodawanie sond

builder.Services.AddHealthChecks()
  .AddSqlServer(builder.Configuration["Data:ConnectionStrings:Sql"])
  .AddSqlServer(
     connectionString: Configuration["Data:ConnectionStrings:Sql"],
     healthQuery: "SELECT 1;",
     name: "sql",     
     tags: new string[] { "db", "sql", "sqlserver" })
  .AddRedis(builder.Configuration["Data:ConnectionStrings:Redis"])
  .AddAzureKeyVault()
  .AddElasticsearch()
  .AddMongoDb();

Dostępne sondy

Co jeśli to nie wystarczy?

public class NotFridayHealthcheck : IHealthCheck
{        
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        if (DateTime.Today.DayOfWeek == DayOfWeek.Friday)
        {
            return Task.FromResult(HealthCheckResult.Unhealthy("It's friday!"));
        }

        if (DateTime.Today.DayOfWeek == DayOfWeek.Monday)
        {
            return Task.FromResult(HealthCheckResult.Degraded("Could be better..."));
        }

        return Task.FromResult(HealthCheckResult.Healthy("All goood"));
    }
}

Własna sonda

builder.Services.AddHealthChecks()
    .AddCheck<NotFridayHealthcheck>("not-friday")
    .AddCheck("lambda-check", () => DateTime.Today.DayOfWeek == DayOfWeek.Friday
      ? HealthCheckResult.Unhealthy("It's friday!")
      : HealthCheckResult.Healthy("All good"));

Rejestracja własnej sondy

public static class NotFridayHealthCheckBuilderExtensions
{
    private const string DefaultName = "not-friday";

    public static IHealthChecksBuilder AddNotFridayHealthCheck(
        this IHealthChecksBuilder healthChecksBuilder,
        string? name = null,
        HealthStatus? failureStatus = null,
        IEnumerable<string>? tags = default)
    {
        return healthChecksBuilder.Add(
            new HealthCheckRegistration(
                name ?? DefaultName,
                _ => new NotFridayHealthcheck(),
                failureStatus,
                tags));
    }
}

builder.Services.AddHealthChecks()
    .AddNotFridayHealthCheck();

Rejestracja własnej sondy - extension method

Model PULL

vs

Model PUSH

Model PULL

Model PUSH

Własny publisher

public class SampleHealthCheckPublisher : IHealthCheckPublisher
{
    public Task PublishAsync(HealthReport report, CancellationToken cancellationToken)
    {
        if (report.Status == HealthStatus.Healthy)
        {
            // ...
        }
        else
        {
            // ...
        }

        return Task.CompletedTask;
    }
}

builder.Services.Configure<HealthCheckPublisherOptions>(options =>
{
    options.Delay = TimeSpan.FromSeconds(2);
    options.Period = TimeSpan.FromSeconds(60);
    options.Timeout = TimeSpan.FromSeconds(60)
    options.Predicate = healthCheck => healthCheck.Tags.Contains("sample");
});

builder.Services.AddSingleton<IHealthCheckPublisher, SampleHealthCheckPublisher>();

Gotowe integracje

services
    .AddHealthChecks()    
    .AddApplicationInsightsPublisher()
    .AddDatadogPublisher()
    .AddPrometheusGatewayPublisher();

Co z tym wszystkim zrobić?

Źródło: https://status.uptimerobot.com/
Źródło: https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks
builder.Services.AddHealthChecksUI(opt =>
    {
        opt.AddHealthCheckEndpoint("api", "/healthz");
    })
.AddInMemoryStorage();

app.MapHealthChecks("/healthz", new HealthCheckOptions
{
    Predicate = _ => true,
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});

app.MapHealthChecksUI();

Konfiguracja HealthChecksUI

HealthChecks a orkiestracja usług

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app

COPY *.csproj ./
RUN dotnet restore

COPY . ./
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=build-env /app/out .

RUN apt-get update && apt-get install -y curl

HEALTHCHECK CMD curl --fail http://localhost/healthz || exit 1

ENTRYPOINT ["dotnet", "HealthCheckSample.Web.dll"]

Kontener startuje

Kontener nie działa

Kontener działa prawidłowo

Jak uleczyć wadliwy kontener?

Docker Autoheal

docker run -d \
    --name autoheal \
    --restart=always \
    -e AUTOHEAL_CONTAINER_LABEL=all \
    -v /var/run/docker.sock:/var/run/docker.sock \
    willfarrell/autoheal

Integracja z Kubernetesem

Kubernetes Probes

  • Liveness probe
  • Readiness probe
  • Startup probe

Kubernetes Probes - konfiguracja

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-healthcheck-app
spec:
  template:
    metadata:
      labels:
        app: sample-healthcheck-app
    spec:
      containers:
      - name: sample-healthcheck-app
        image: my-repo/sample-healthcheck-app:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 0
          periodSeconds: 10
          timeoutSeconds: 1
          failureThreshold: 3

Różne endpointy na różne sondy

endpoints.MapHealthChecks("/healthz/startup");
        
endpoints.MapHealthChecks("/healthz", 
    new HealthCheckOptions 
    { 
        Predicate = x => x.Tags.Contains("redis") 
    });
        
endpoints.MapHealthChecks("/readyz", 
    new HealthCheckOptions 
    {
        Predicate = _ => x.Tags.Contains("sql") 
    });        

Azure App Service

Azure Traffic Manager

Zabawa bez końca: integracja z Polly

Retry policy

var retry = Policy
  .Handle<SomeExceptionType>()
  .WaitAndRetry(new[]
  {
    TimeSpan.FromSeconds(1),
    TimeSpan.FromSeconds(2),
    TimeSpan.FromSeconds(3)
  });
  
retry.Execute(() > {} );

Circuit Breaker

Circuit Breaker

// WeatherForecastController.cs
public static CircuitBreakerPolicy CircuitBreaker = Policy
    .Handle<Exception>()
    .CircuitBreaker(3, TimeSpan.FromSeconds(30));
                                                          
// Get() method
CircuitBreaker.Execute(() =>
{
    GetWeather();  
});

// Program.cs
builder.Services.AddHealthChecks()
  .AddCheck("circuit-breaker", 
    () => CircuitBreaker.CircuitState == Polly.CircuitBreaker.CircuitState.Closed
       ? HealthCheckResult.Healthy()
       : HealthCheckResult.Unhealthy())

Do zapamiętania:

  • Healthchecks wspierają monitoring systemu
  • Pozwalają odpowiednio reagować
  • Mogą wspomóc samonaprawę systemu
  • Wspomagają load balancing
  • W ASP.NET Core mamy gotową implementację

justjoin.it/sniadanie-z-programowaniem

mateusz.turzynski@aspiresys.com

Czy wszyscy

By Mateusz Turzyński

Czy wszyscy

  • 469