إنتقل إلى المحتوى الرئيسي

📊 Observability & Monitoring

BayanCore implements a comprehensive observability stack based on OpenTelemetry, ensuring complete visibility across container health, database performance, transactional latency, and AI model quality.

1. The Three Pillars of Observability​

All telemetry data is collected, aggregated, and stored strictly inside Saudi Arabia (OCI Riyadh).

Metrics Collection (Prometheus & OCI Monitoring)​

  • Infrastructure: Monitors container CPU shares, RAM utilization, block storage IOPS, and Kubernetes node health.
  • Application: Tracks request counts, error rates, and P95 latency metrics across REST/GraphQL endpoints.
  • Business & Compliance Key Performance Indicators (KPIs):
    • FWCR (First-Workflow-Completion-Rate): Target >80%.
    • ZATCA Latency: Target <2 seconds.
    • AI OCR Extraction Accuracy: Target >95%.

Structured Logging (Grafana Loki)​

  • Structured Format: All services output logs as standard structured JSON blocks: 
    {
      "timestamp": "2026-06-02T01:16:39Z",
      "service": "guardian-validator",
      "level": "INFO",
      "trace_id": "82a71f092b3a1",
      "message": "Invoice UBL 2.1 validation passed",
      "metadata": { "invoice_id": "INV-2026-0102", "company_id": "comp-12" }
    }
  • Log Isolation: Debug and application logs are stored in OCI Object Storage with an automated lifecycle policy that purges records after 2 years.

Distributed Tracing (OpenTelemetry & Jaeger)​

  • Span Tracking: Every incoming API call is assigned a unique trace_id by the API Gateway. This trace ID propagates across background tasks, Celery queues, database queries, and AI inferences, allowing engineers to isolate latency bottlenecks.

2. AI Model Drift & Telemetry​

Observing generative AI pipelines requires specific monitoring parameters:

  • Feedback Loops: Captures instances where users manually correct AI suggestions (such as modifying fields populated via receipt OCR). A correction rate exceeding 5% triggers alerts for model retraining.
  • Semantic Shift: Analyzes shifts in user input query distributions (e.g. new vocabulary or regional dialects), updating RAG retrieval parameters accordingly.
  • LLM Hallucinations: Tracks confidence scores and deterministic post-validation failures. If the validation engine blocks LLM outputs due to data mismatches at a rate >1%, it triggers alerts to fallback to rule-based execution.

3. Incident Alerting Matrix​

Alerts are routed to operations teams based on severity:

Severity LevelTrigger ThresholdPrimary Alert ChannelEscalation Policy
P1 (Critical)ZATCA API failure rate >5%, DB Connection Pool exhausted, site offlineSMS / WhatsApp CallPage on-call engineer; escalate to CTO if unresolved in 15m
P2 (High)API error rate >1%, P95 latency >3s, disk usage >85%Slack & EmailRoute to active dev team channel; resolve within 4 hours
P3 (Medium)AI OCR correction rate >10%, Redis cache hit rate <80%Slack AlertLog to backlog; review during weekly maintenance sprints
P4 (Low)UI layout alignment warning, non-critical package updatesDashboard NotificationReview during bi-weekly release cycles