Realtime Heart Rate Estimators

NOTE

Web SDKs Only Realtime estimation is currently available for web-based SDKs only (JavaScript, React, Vue). It is not available for mobile SDKs (iOS, Android, Flutter, React Native).

Overview

The Realtime Estimators provide instant heart rate feedback during video capture, showing results in as little as 3-8 seconds instead of waiting for the full 30-second scan. This creates a more engaging user experience and helps users adjust their positioning and lighting for optimal measurements.

Quick Start

import { createVitalSignCamera, RealtimeEstimatorType } from 'ts-vital-sign-camera';

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.Panoptic // Default
  },
  onVideoFrameProcessed: (event) => {
    // Realtime estimation is available in the event
    if (event.realtimeEstimation) {
      console.log(`Heart Rate: ${event.realtimeEstimation.heartRate} BPM`);
      console.log(`Confidence: ${event.realtimeEstimation.confidence}`);
      console.log(`Stable: ${event.realtimeEstimation.isStable}`);
    }
  }
});

Simple Setup

For the fastest setup with Panoptic estimator defaults, use the enableRealtimeEstimation boolean:

import { createVitalSignCamera } from 'ts-vital-sign-camera';

const camera = createVitalSignCamera({
  enableRealtimeEstimation: true,
  onVideoFrameProcessed: (event) => {
    if (event.realtimeEstimation) {
      console.log(`Heart Rate: ${event.realtimeEstimation.heartRate} BPM`);
    }
  }
});

Vue SDK:

<VitalSignCamera
  :enableRealtimeEstimation="true"
  @onVideoFrameProcessed="onVideoFrameProcessed"
/>

This activates the Panoptic estimator with sensible defaults. Use realtimeEstimationConfig when you need custom configuration.

Available Estimators

ME-rPPG Estimator 🤖

AI-powered neural network approach

• Best For: Consumer apps, varying conditions, modern devices
• Accuracy: State-of-the-art (±2-3 BPM)
• Speed: First result in ~3 seconds
• Motion Tolerance: Excellent
• Lighting Tolerance: Excellent
• Licensing: Open-source
• Requirements: ~10 MB model files

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.MeRppg
  }
});

Panoptic Estimator (Default) 📈

Signal processing approach

• Best For: Medical apps, instant initialization, minimal bundle size
• Accuracy: Medical-grade (±2-3 BPM)
• Speed: First result in ~8 seconds
• Motion Tolerance: Good
• Lighting Tolerance: Good
• Licensing: Proprietary (PanopticAI)
• Requirements: None (instant initialization)

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.Panoptic
  }
});

Configuration Options

Basic Configuration

interface RealtimeEstimationConfig {
  // Which estimator to use
  estimatorType?: RealtimeEstimatorType;
  
  // Minimum seconds before computing quality metrics
  signalQualityDelay?: number;
  
  // Minimum RGB samples before producing an estimate
  minSamples?: number;
  
  // Total scan duration in seconds
  scanDuration?: number;
  
  // Type of visual signal (Panoptic only: Raw | Normalized)
  visualSignalType?: VisualSignalType;
  
  // Enable debug logging
  debug?: boolean;
}

ME-rPPG Specific Options

{
  // Model file paths (optional, defaults provided)
  modelPath?: string;
  statePath?: string;
  welchPath?: string;
  hrPath?: string;
  
  // Temporal normalization parameter
  lambda?: number; // Default: 1.0
}

Estimation Results

RealtimeEstimation Interface

interface RealtimeEstimation {
  // Heart rate in BPM
  heartRate: number;
  
  // Confidence level (0-1)
  confidence: number;
  
  // Whether result is stable enough to display
  isStable: boolean;
  
  // Signal-to-noise ratio (optional)
  snr?: number;
  
  // Signal quality score 0-100 (optional)
  signalQuality?: number;
}

Using Estimation Results

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  
  if (!estimation) return;
  
  if (estimation.isStable && estimation.confidence > 0.6) {
    // High confidence - display prominently
    displayHeartRate(estimation.heartRate);
  } else if (estimation.confidence > 0.3) {
    // Medium confidence - show with indicator
    displayHeartRate(estimation.heartRate, 'Refining...');
  } else {
    // Low confidence - guide user
    showMessage('Adjusting... Please stay still');
  }
};

Signal Visualization

For detailed documentation on real-time signal visualization (PPG and respiratory waveforms), see the Signal Visualizer Guide.

The Signal Visualizer displays scrolling PPG/respiratory waveform data alongside the camera feed. It is a standalone component created separately and connected via the camera's visualizer property or the onVideoFrameProcessed callback.

For the full list of visualization options including heatmap, bounding box, and face mesh overlays, see the Visualization Options Overview.

SDK Behavior

Timeline

0s ────────► 3s ────────► 8s ────────► 15s ────────► 30s
│            │            │            │             │
│            │            │            │             └─ Full scan complete
│            │            │            └─ Optimal accuracy (Panoptic)
│            │            └─ First realtime estimate (Panoptic after warmup)
│            └─ First estimate (ME-rPPG)
└─ Scan starts

Estimation Lifecycle

1. Initialization (0-3s)

   • Camera starts capturing
   • Face detection active
   • Signal buffer filling

2. Early Estimation (3-8s)

   • First results available (ME-rPPG at ~3s, Panoptic at ~8s)
   • Lower confidence
   • Continuous refinement

3. Stable Estimation (15s+)

   • High confidence results
   • Optimal accuracy
   • Ready for display

4. Scan Complete (30s)

   • Server-side validation
   • Final results available
   • Realtime estimator can be reset

Event Flow

// Scan lifecycle
camera.onVideoFrameProcessed = (event) => {
  // Realtime estimation available here
  if (event.realtimeEstimation) {
    console.log('Realtime update:', event.realtimeEstimation);
  }
};

camera.startScanning(); // Start the scan

// Final results come from the health result
camera.onVideoFrameProcessed = (event) => {
  if (event.healthResult) {
    console.log('Final results:', event.healthResult);
  }
};

Best Practices

1. Choose the Right Estimator

// For consumer apps with varying conditions
const config = {
  estimatorType: RealtimeEstimatorType.MeRppg
};

// For medical apps requiring instant init
const config = {
  estimatorType: RealtimeEstimatorType.Panoptic
};

2. Provide User Feedback

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  if (!estimation) return;
  
  // Show confidence indicator
  updateConfidenceBar(estimation.confidence);
  
  // Guide user based on quality
  if (estimation.confidence < 0.4) {
    showTip('Improve lighting or stay still');
  }
  
  // Display when stable
  if (estimation.isStable) {
    displayHeartRate(estimation.heartRate);
  }
};

3. Handle Edge Cases

let noEstimationTimeout;
let hasReceivedEstimation = false;

camera.startScanning();

// Set timeout for no estimation
noEstimationTimeout = setTimeout(() => {
  if (!hasReceivedEstimation) {
    showError('Unable to detect heart rate. Please adjust lighting.');
  }
}, 15000); // 15 seconds

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  
  if (estimation) {
    hasReceivedEstimation = true;
    clearTimeout(noEstimationTimeout);
    
    if (estimation.confidence > 0.5) {
      hideError();
    }
  }
};

4. Optimize Performance

// Throttle UI updates
let lastUpdate = 0;
const UPDATE_INTERVAL = 500; // 500ms

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  if (!estimation) return;
  
  const now = Date.now();
  if (now - lastUpdate > UPDATE_INTERVAL) {
    updateUI(estimation);
    lastUpdate = now;
  }
};

Common Use Cases

Fitness App

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.MeRppg
  },
  onVideoFrameProcessed: (event) => {
    if (event.realtimeEstimation) {
      updateWorkoutDisplay(event.realtimeEstimation.heartRate);
      updateHeartRateZone(event.realtimeEstimation.heartRate);
    }
  }
});

Medical Application

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.Panoptic,
    signalQualityDelay: 10  // Longer for medical use
  },
  onVideoFrameProcessed: (event) => {
    const estimation = event.realtimeEstimation;
    if (estimation?.isStable && estimation.confidence > 0.7) {
      recordMeasurement(estimation);
    }
  }
});

With Signal Visualization

For a complete example, see the Signal Visualizer Guide.

Troubleshooting

No Realtime Estimates

Problem: Not receiving realtime estimation in events

Solutions:

1. Check face is detected properly
2. Verify lighting conditions are adequate
3. Wait at least 15 seconds for stable Panoptic results
4. Check event.realtimeEstimation is not null
5. Panoptic estimator has a warmup period (~8s) — no estimates are emitted until the buffer fills

Low Confidence Scores

Problem: Confidence consistently below threshold

Solutions:

1. Improve lighting (natural light is best)
2. Ensure user stays still
3. Check face is centered and stable
4. Use ME-rPPG for better motion tolerance

ME-rPPG Models Not Loading

Problem: Models fail to load

Solutions:

1. Verify model files are in correct directory
2. Check CORS headers allow model loading
3. Ensure correct file paths (use absolute paths)
4. Check browser console for errors

API Reference

RealtimeEstimator Interface

interface RealtimeEstimator {
  readonly estimation: RealtimeEstimation | null;
  readonly signalData: SignalData | null;
  
  configure(config?: RealtimeEstimationConfig): void;
  reset(): void;
}

SignalVisualizer Class

class SignalVisualizer {
  constructor(container: HTMLElement, config?: SignalVisualizerConfig);
  
  updateSignal(
    signalData: SignalData | null,
    realtimeEstimation?: RealtimeEstimation | null
  ): void;
  
  resetScale(): void;
  destroy(): void;
}

VideoFrameProcessedEvent

interface VideoFrameProcessedEvent {
  videoFrameInfo: VideoFrameInfo;
  videoFrame: VideoFrame;
  facebox?: NormalizedFacebox;
  healthResult?: ScanResult;
  scanConditions?: ScanConditions;
  landmarks?: NormalizedLandmarkList;
  
  // Realtime estimation data
  realtimeEstimation?: RealtimeEstimation | null;
  signalData?: SignalData | null;
}

Next Steps

• ME-rPPG Estimator Details
• Panoptic Estimator Details
• Selection Guide
• Full SDK Documentation