Realtime Heart Rate Estimators

NOTE

Web SDKs Only Realtime estimation is currently available for web-based SDKs only (JavaScript, React, Vue). It is not available for mobile SDKs (iOS, Android, Flutter, React Native).

Overview

The Realtime Estimators provide instant heart rate feedback during video capture, showing results in as little as 3-5 seconds instead of waiting for the full 30-second scan. This creates a more engaging user experience and helps users adjust their positioning and lighting for optimal measurements.

Quick Start

import { createVitalSignCamera, RealtimeEstimatorType } from 'ts-vital-sign-camera';

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.MeRppg, // Default: AI-powered
    earlyEstimation: true,
    minDuration: 3,
    minConfidence: 0.3
  },
  onVideoFrameProcessed: (event) => {
    // Realtime estimation is available in the event
    if (event.realtimeEstimation) {
      console.log(`Heart Rate: ${event.realtimeEstimation.heartRate} BPM`);
      console.log(`Confidence: ${event.realtimeEstimation.confidence}`);
      console.log(`Stable: ${event.realtimeEstimation.isStable}`);
    }
  }
});

Available Estimators

ME-rPPG Estimator (Default) 🤖

AI-powered neural network approach

• Best For: Consumer apps, varying conditions, modern devices
• Accuracy: State-of-the-art (±2-3 BPM)
• Speed: First result in ~3 seconds
• Motion Tolerance: Excellent
• Lighting Tolerance: Excellent
• Licensing: Open-source
• Requirements: ~10 MB model files

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.MeRppg
  }
});

FDA Estimator 📈

Signal processing approach

• Best For: Medical apps, instant initialization, minimal bundle size
• Accuracy: Medical-grade (±2-3 BPM)
• Speed: First result in ~5 seconds
• Motion Tolerance: Good
• Lighting Tolerance: Good
• Licensing: Proprietary (PanopticAI)
• Requirements: None (instant initialization)

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.Fda
  }
});

Configuration Options

Basic Configuration

interface RealtimeEstimationConfig {
  // Which estimator to use
  estimatorType?: RealtimeEstimatorType;
  
  // Show results before full scan completes
  earlyEstimation?: boolean;
  
  // Minimum seconds before showing result
  minDuration?: number;
  
  // Minimum confidence threshold (0-1)
  minConfidence?: number;
  
  // Enable debug logging
  debug?: boolean;
}

ME-rPPG Specific Options

{
  // Model file paths (optional, defaults provided)
  modelPath?: string;
  statePath?: string;
  welchPath?: string;
  hrPath?: string;
  
  // Temporal normalization parameter
  lambda?: number; // Default: 1.0
}

Estimation Results

RealtimeEstimation Interface

interface RealtimeEstimation {
  // Heart rate in BPM
  heartRate: number;
  
  // Confidence level (0-1)
  confidence: number;
  
  // Whether result is stable enough to display
  isStable: boolean;
  
  // Signal-to-noise ratio (optional)
  snr?: number;
  
  // Signal quality score 0-100 (optional)
  signalQuality?: number;
}

Using Estimation Results

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  
  if (!estimation) return;
  
  if (estimation.isStable && estimation.confidence > 0.6) {
    // High confidence - display prominently
    displayHeartRate(estimation.heartRate);
  } else if (estimation.confidence > 0.3) {
    // Medium confidence - show with indicator
    displayHeartRate(estimation.heartRate, 'Refining...');
  } else {
    // Low confidence - guide user
    showMessage('Adjusting... Please stay still');
  }
};

Signal Visualization

Display the raw PPG signal for debugging or user feedback.

Basic Usage

import { SignalVisualizer } from 'ts-vital-sign-camera';

// Create visualizer
const container = document.getElementById('signal-container');
const visualizer = new SignalVisualizer(container, {
  width: 600,
  height: 200,
  visualizationType: 'PPG', // or 'ECG' for simulation
  lineColor: '#00ff00',
  backgroundColor: '#000000'
});

// Update with signal data from video frame event
camera.onVideoFrameProcessed = (event) => {
  if (event.signalData) {
    visualizer.updateSignal(event.signalData, event.realtimeEstimation);
  }
};

// Cleanup
visualizer.destroy();

Configuration Options

interface SignalVisualizerConfig {
  width?: number;              // Canvas width (default: 600)
  height?: number;             // Canvas height (default: 200)
  maxPoints?: number;          // Max data points (default: 300)
  lineColor?: string;          // Signal line color
  backgroundColor?: string;    // Canvas background
  gridColor?: string;          // Grid line color
  visualizationType?: 'PPG' | 'ECG';  // Signal type
  debug?: boolean;             // Debug logging
}

Visualization Types

PPG Mode (Default):

• Shows actual photoplethysmography signal from camera
• Real-time scrolling waveform
• Useful for debugging signal quality

ECG Mode:

• Simulated electrocardiogram animation
• Visual feedback during measurement
• Matches estimated heart rate

// PPG visualization (actual signal)
const ppgVisualizer = new SignalVisualizer(container, {
  visualizationType: 'PPG'
});

// ECG simulation (animated)
const ecgVisualizer = new SignalVisualizer(container, {
  visualizationType: 'ECG',
  lineColor: '#ff6b6b'
});

SDK Behavior

Timeline

0s ────────► 3s ────────► 10s ────────► 30s
│            │            │             │
│            │            │             └─ Full scan complete
│            │            └─ Optimal accuracy achieved
│            └─ First realtime estimate (ME-rPPG)
└─ Scan starts

Estimation Lifecycle

1. Initialization (0-3s)

   • Camera starts capturing
   • Face detection active
   • Signal buffer filling

2. Early Estimation (3-5s)

   • First results available
   • Lower confidence
   • Continuous refinement

3. Stable Estimation (10s+)

   • High confidence results
   • Optimal accuracy
   • Ready for display

4. Scan Complete (30s)

   • Server-side validation
   • Final results available
   • Realtime estimator can be reset

Event Flow

// Scan lifecycle
camera.onVideoFrameProcessed = (event) => {
  // Realtime estimation available here
  if (event.realtimeEstimation) {
    console.log('Realtime update:', event.realtimeEstimation);
  }
};

camera.startScanning(); // Start the scan

// Final results come from the health result
camera.onVideoFrameProcessed = (event) => {
  if (event.healthResult) {
    console.log('Final results:', event.healthResult);
  }
};

Best Practices

1. Choose the Right Estimator

// For consumer apps with varying conditions
const config = {
  estimatorType: RealtimeEstimatorType.MeRppg,
  earlyEstimation: true,
  minDuration: 3,
  minConfidence: 0.3
};

// For medical apps requiring instant init
const config = {
  estimatorType: RealtimeEstimatorType.Fda,
  earlyEstimation: false,
  minDuration: 10,
  minConfidence: 0.7
};

2. Provide User Feedback

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  if (!estimation) return;
  
  // Show confidence indicator
  updateConfidenceBar(estimation.confidence);
  
  // Guide user based on quality
  if (estimation.confidence < 0.4) {
    showTip('Improve lighting or stay still');
  }
  
  // Display when stable
  if (estimation.isStable) {
    displayHeartRate(estimation.heartRate);
  }
};

3. Handle Edge Cases

let noEstimationTimeout;
let hasReceivedEstimation = false;

camera.startScanning();

// Set timeout for no estimation
noEstimationTimeout = setTimeout(() => {
  if (!hasReceivedEstimation) {
    showError('Unable to detect heart rate. Please adjust lighting.');
  }
}, 15000); // 15 seconds

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  
  if (estimation) {
    hasReceivedEstimation = true;
    clearTimeout(noEstimationTimeout);
    
    if (estimation.confidence > 0.5) {
      hideError();
    }
  }
};

4. Optimize Performance

// Throttle UI updates
let lastUpdate = 0;
const UPDATE_INTERVAL = 500; // 500ms

camera.onVideoFrameProcessed = (event) => {
  const estimation = event.realtimeEstimation;
  if (!estimation) return;
  
  const now = Date.now();
  if (now - lastUpdate > UPDATE_INTERVAL) {
    updateUI(estimation);
    lastUpdate = now;
  }
};

Common Use Cases

Fitness App

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.MeRppg,
    earlyEstimation: true,
    minDuration: 3,
    minConfidence: 0.3
  },
  onVideoFrameProcessed: (event) => {
    if (event.realtimeEstimation) {
      updateWorkoutDisplay(event.realtimeEstimation.heartRate);
      updateHeartRateZone(event.realtimeEstimation.heartRate);
    }
  }
});

Medical Application

const camera = createVitalSignCamera({
  realtimeEstimationConfig: {
    estimatorType: RealtimeEstimatorType.Fda,
    earlyEstimation: false,
    minDuration: 10,
    minConfidence: 0.7
  },
  onVideoFrameProcessed: (event) => {
    const estimation = event.realtimeEstimation;
    if (estimation?.isStable && estimation.confidence > 0.7) {
      recordMeasurement(estimation);
    }
  }
});

With Signal Visualization

const visualizer = new SignalVisualizer(container, {
  visualizationType: 'PPG',
  width: 800,
  height: 300
});

camera.onVideoFrameProcessed = (event) => {
  if (event.signalData) {
    visualizer.updateSignal(event.signalData, event.realtimeEstimation);
  }
};

Troubleshooting

No Realtime Estimates

Problem: Not receiving realtime estimation in events

Solutions:

1. Ensure earlyEstimation: true is set
2. Check face is detected properly
3. Verify lighting conditions are adequate
4. Wait at least minDuration seconds
5. Check event.realtimeEstimation is not null

Low Confidence Scores

Problem: Confidence consistently below threshold

Solutions:

1. Improve lighting (natural light is best)
2. Ensure user stays still
3. Check face is centered and stable
4. Lower minConfidence threshold
5. Use ME-rPPG for better motion tolerance

ME-rPPG Models Not Loading

Problem: Models fail to load

Solutions:

1. Verify model files are in correct directory
2. Check CORS headers allow model loading
3. Ensure correct file paths (use absolute paths)
4. Check browser console for errors

API Reference

RealtimeEstimator Interface

interface RealtimeEstimator {
  readonly estimation: RealtimeEstimation | null;
  readonly signalData: SignalData | null;
  
  configure(config?: RealtimeEstimationConfig): void;
  reset(): void;
}

SignalVisualizer Class

class SignalVisualizer {
  constructor(container: HTMLElement, config?: SignalVisualizerConfig);
  
  updateSignal(
    signalData: SignalData | null,
    realtimeEstimation?: RealtimeEstimation | null
  ): void;
  
  resetScale(): void;
  destroy(): void;
}

VideoFrameProcessedEvent

interface VideoFrameProcessedEvent {
  videoFrameInfo: VideoFrameInfo;
  videoFrame: VideoFrame;
  facebox?: NormalizedFacebox;
  healthResult?: ScanResult;
  scanConditions?: ScanConditions;
  landmarks?: NormalizedLandmarkList;
  
  // Realtime estimation data
  realtimeEstimation?: RealtimeEstimation | null;
  signalData?: SignalData | null;
}

Next Steps

• ME-rPPG Estimator Details
• FDA Estimator Details
• Selection Guide
• Full SDK Documentation