Search Architecture

This document describes the internal architecture of OctoFHIR’s search parameter system, including how search parameters are loaded, registered, and used during search operations.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                     FHIR REST API Layer                          │
│  POST /SearchParameter → Create & Auto-Register                 │
│  GET /Patient?name=...  → Query using Registry                  │
└──────────────────┬──────────────────────────────────────────────┘
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
┌───────────────────┐  ┌────────────────────┐
│ Postprocess Hook  │  │  Search Handler    │
│ - Validate FHIRPath│  │ - Parse Query      │
│ - Upsert Registry │  │ - Get Config       │
│ - Clear Cache     │  │ - Execute Search   │
└─────────┬─────────┘  └──────────┬─────────┘
          │                       │
          ▼                       ▼
┌─────────────────────────────────────────────┐
│      ReloadableSearchConfig (Arc-Swap)       │
│  ┌───────────────────────────────────────┐  │
│  │    SearchConfig (Lock-Free Reads)     │  │
│  │  ┌─────────────────────────────────┐  │  │
│  │  │  SearchParameterRegistry        │  │  │
│  │  │  - by_resource (DashMap)        │  │  │
│  │  │  - by_url (DashMap)             │  │  │
│  │  │  - common (DashMap)             │  │  │
│  │  └─────────────────────────────────┘  │  │
│  │  ┌─────────────────────────────────┐  │  │
│  │  │  QueryCache (Optional)          │  │  │
│  │  └─────────────────────────────────┘  │  │
│  └───────────────────────────────────────┘  │
└─────────────────────────────────────────────┘
                   │
                   ▼
         ┌──────────────────┐
         │ Canonical Manager│
         │ (PostgreSQL)     │
         └──────────────────┘

Core Components

1. SearchParameterRegistry

Thread-safe registry using DashMap for lock-free concurrent access.

Location: crates/octofhir-search/src/registry.rs

Data Structures:

pub struct SearchParameterRegistry {
    /// Parameters indexed by (resource_type, code)
    by_resource: DashMap<(String, String), Arc<SearchParameter>>,
    /// All parameters by canonical URL
    by_url: DashMap<String, Arc<SearchParameter>>,
    /// Common parameters (base = "Resource" or "DomainResource")
    common: DashMap<String, Arc<SearchParameter>>,
}

Key Methods:

• register(&self, param: SearchParameter) - Register a new parameter
• upsert(&self, param: SearchParameter) - Update or insert (alias for register)
• remove_by_url(&self, url: &str) -> bool - Remove by canonical URL
• get(&self, resource_type: &str, code: &str) -> Option<Arc<SearchParameter>> - Lookup parameter

Why DashMap?

• Lock-free concurrent HashMap
• Multiple readers can access simultaneously
• Writers don’t block readers
• O(1) lookup performance

2. ReloadableSearchConfig

Wrapper providing lock-free configuration access using atomic pointer swapping.

Location: crates/octofhir-search/src/reloadable.rs

Implementation:

pub struct ReloadableSearchConfig {
    /// Inner config with atomic pointer swap (lock-free reads!)
    inner: Arc<ArcSwap<SearchConfig>>,
    options: Arc<RwLock<SearchOptions>>,
    cache: Option<Arc<QueryCache>>,
}

Key Operations:

Get Config (Lock-Free Read):

pub fn config(&self) -> Arc<SearchConfig> {
    self.inner.load_full()  // Single atomic load, ~1-5 nanoseconds
}

Reload Registry (Atomic Swap):

pub async fn reload_registry(&self, canonical_manager: &CanonicalManager)
    -> Result<(), LoaderError>
{
    // 1. Load new registry (background, doesn't block readers)
    let new_registry = Arc::new(load_search_parameters(canonical_manager).await?);

    // 2. Create new config
    let current = self.inner.load_full();
    let new_config = SearchConfig {
        default_count: current.default_count,
        max_count: current.max_count,
        registry: new_registry,
        cache: self.cache.clone(),
    };

    // 3. Atomic swap (old readers keep old config, new readers get new config)
    self.inner.store(Arc::new(new_config));

    // 4. Clear query cache
    if let Some(cache) = &self.cache {
        cache.clear();
    }

    Ok(())
}

Why Arc-Swap?

• Zero lock contention - Readers never wait for writers
• Atomic updates - Pointer swap is atomic (single CPU instruction)
• Graceful transition - Old readers finish with old config
• Automatic cleanup - Old config freed when last reader releases it

3. Search Parameter Loading

Location: crates/octofhir-search/src/loader.rs

Loading Process:

1. Register Common Parameters (built-in)

   • _id, _lastUpdated, _tag, _profile, _security, _source
   • Apply to all resource types

2. Query Canonical Manager

   • Load all SearchParameter resources from PostgreSQL
   • High limit (10,000) to get all parameters in one query

3. Parse and Validate

   • Extract required fields (code, url, type, base)
   • Parse FHIRPath expression
   • Build internal SearchParameter struct

4. Register in Registry

   • Index by URL
   • Index by (resource_type, code) for each base
   • Mark as common if base includes “Resource”

Performance:

• Initial load: ~100-200ms for ~1000 parameters
• Incremental update: ~100-500 nanoseconds per parameter

4. Postprocessing Hook

Location: crates/octofhir-server/src/handlers.rs

Automatically called after successful create/update/delete operations.

Implementation:

async fn postprocess_resource(
    resource_type: &str,
    _resource_id: &str,
    payload: &Value,
    state: &AppState,
) -> Result<(), ApiError> {
    if resource_type == "SearchParameter" {
        // 1. Validate FHIRPath expression syntax
        if let Some(expression) = payload.get("expression").and_then(|v| v.as_str()) {
            validate_fhirpath_expression(expression)?;
        }

        // 2. Parse SearchParameter resource
        match octofhir_search::parse_search_parameter(payload) {
            Ok(param) => {
                // 3. Get current config snapshot
                let config = state.search_config.config();

                // 4. Incrementally update registry (thread-safe)
                config.registry.upsert(param);

                // 5. Clear query cache to prevent stale results
                if let Some(cache) = &config.cache {
                    cache.clear();
                }

                tracing::info!(
                    code = payload.get("code").and_then(|v| v.as_str()),
                    "Search parameter registered incrementally"
                );
            }
            Err(e) => {
                tracing::error!(error = %e, "Failed to parse SearchParameter");
            }
        }
    }

    Ok(())
}

FHIRPath Validation:

fn validate_fhirpath_expression(expression: &str) -> Result<(), ApiError> {
    use octofhir_fhirpath::parse;

    let result = parse(expression);
    if !result.success {
        let error_msg = result.error_message
            .unwrap_or_else(|| "Unknown parse error".to_string());
        return Err(ApiError::bad_request(format!(
            "Invalid FHIRPath expression: {}",
            error_msg
        )));
    }

    Ok(())
}

Validation Scope:

• ✅ Syntax correctness (parentheses, operators, function names)
• ✅ Basic structure (paths, predicates, function calls)
• ❌ Semantic correctness (doesn’t check against resource schema)
• ❌ Type checking (doesn’t validate return types)

Performance Characteristics

Lock-Free Architecture

Traditional Approach (RwLock):

┌─────────────────────────────────────────────┐
│ RwLock<SearchConfig>                        │
│                                             │
│  Readers:  [Wait] [Wait] [Wait] [Wait]      │
│  Writer:   [LOCK - BLOCKING ALL READERS]    │
│                                             │
│  Problem: All searches blocked during       │
│           registry reload (1-2 seconds)     │
└─────────────────────────────────────────────┘

OctoFHIR Approach (Arc-Swap):

┌─────────────────────────────────────────────┐
│ ArcSwap<SearchConfig>                       │
│                                             │
│  Readers:  [Read Old] [Read Old] [Read New] │
│  Writer:   [Create New] → [Atomic Swap]     │
│                                             │
│  Benefit: Zero blocking, seamless           │
│           transition between configs        │
└─────────────────────────────────────────────┘

Performance Metrics

Operation	Time	Impact on Searches
Registry lookup (read)	1-5 ns	None (lock-free)
Incremental update (upsert)	100-500 ns	None (lock-free)
Full registry reload	100-200 ms	None (atomic swap)
Query cache lookup	10-50 ns	None (concurrent)

Memory Usage

Per SearchParameter:

• SearchParameter struct: ~200 bytes
• Arc overhead: 16 bytes
• DashMap entry overhead: ~40 bytes
• Total: ~256 bytes per parameter

Full Registry (1000 parameters):

• Parameters: ~256 KB
• Three indices (by_resource, by_url, common): ~768 KB
• Total: ~1 MB

Config Snapshot:

• Arc<SearchConfig>: 8 bytes (pointer)
• Reference counting: atomic operations
• Zero copy on read

Concurrency Model

Read Operations (Search Queries)

// Multiple threads can read simultaneously
let config = state.search_config.config();  // Arc<SearchConfig>
let param = config.registry.get("Patient", "name");  // Option<Arc<SearchParameter>>

// All operations are lock-free:
// 1. config() - single atomic load
// 2. registry.get() - DashMap concurrent read
// 3. No blocking, no waiting

Write Operations (SearchParameter CRUD)

// Incremental update (after POST/PUT SearchParameter)
let config = state.search_config.config();
config.registry.upsert(new_param);  // DashMap concurrent insert

// Full reload (after package installation)
state.search_config.reload_registry(&canonical_manager).await?;
// 1. Loads new registry in background
// 2. Swaps pointer atomically
// 3. Old config stays alive until readers release it

Thread Safety Guarantees

1. No Data Races - All data structures are thread-safe
2. No Deadlocks - No locks used in read path
3. No Starvation - Readers never wait for writers
4. Eventual Consistency - All readers eventually see updates

Query Cache

Location: crates/octofhir-search/src/query_cache.rs

Purpose: Cache parsed search queries to avoid re-parsing identical queries.

Implementation:

pub struct QueryCache {
    cache: Arc<DashMap<QueryCacheKey, QueryCacheEntry>>,
    capacity: usize,
}

struct QueryCacheKey {
    resource_type: String,
    query_string: String,
}

Cache Invalidation:

• Cleared when SearchParameter is created/updated/deleted
• Cleared when registry is reloaded
• Prevents stale query results

Performance:

• Cache hit: ~10-50 nanoseconds
• Cache miss: ~1-10 microseconds (parse + cache)
• Hit rate: Typically 60-90% in production

Multi-Instance Behavior

Challenge

Multiple OctoFHIR instances sharing the same PostgreSQL database:

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Instance A  │    │  Instance B  │    │  Instance C  │
│              │    │              │    │              │
│  Registry A  │    │  Registry B  │    │  Registry C  │
└──────┬───────┘    └──────┬───────┘    └──────┬───────┘
       │                   │                   │
       └───────────────────┴───────────────────┘
                           │
                 ┌─────────▼──────────┐
                 │   PostgreSQL       │
                 │   (Canonical Mgr)  │
                 └────────────────────┘

Current Behavior

1. SearchParameter Created on Instance A:

   • Stored in PostgreSQL (shared)
   • Registered in Instance A’s registry (local)
   • Instance B and C don’t know about it yet

2. Instance B/C Eventually Receive Update:

   • When they reload from canonical manager
   • Manual reload trigger
   • Server restart

Recommendations

For Development/Single Instance:

• Use REST API to create SearchParameters
• Changes are immediate on that instance

For Production/Multi-Instance:

• Use FHIR packages for SearchParameters
• Install packages on all instances
• Ensures consistency across cluster

Future Enhancement:

┌──────────────┐         ┌──────────────┐
│  Instance A  │ ◄─────► │    Redis     │
│  POST /SP    │ Pub/Sub │ (Message Bus)│
└──────────────┘         └──────┬───────┘
                                │
                    ┌───────────┴───────────┐
                    │                       │
            ┌───────▼──────┐       ┌───────▼──────┐
            │  Instance B  │       │  Instance C  │
            │  (Reload)    │       │  (Reload)    │
            └──────────────┘       └──────────────┘

Configuration

Config File: octofhir.toml

[search]
default_count = 10      # Default result count
max_count = 100         # Maximum allowed count
cache_capacity = 1000   # Query cache size (0 = disabled)

Environment Variables:

OCTOFHIR__SEARCH__DEFAULT_COUNT=20
OCTOFHIR__SEARCH__MAX_COUNT=500
OCTOFHIR__SEARCH__CACHE_CAPACITY=5000

Rust Configuration:

use octofhir_search::SearchOptions;

let options = SearchOptions {
    default_count: 10,
    max_count: 100,
    cache_capacity: 1000,
};

let config = ReloadableSearchConfig::new(&canonical_manager, options).await?;

Debugging and Monitoring

Logging

Enable debug logging to see search parameter operations:

RUST_LOG=octofhir_search=debug cargo run

Log Examples:

DEBUG octofhir_search::loader: Loaded search parameter code=name bases=["Patient"]
INFO  octofhir_search::loader: Loaded search parameters loaded=1247 skipped=3 total=1247
INFO  octofhir_server::handlers: Search parameter registered incrementally code="custom-field"
DEBUG octofhir_search::reloadable: Cleared query cache after registry reload

Metrics

Track performance metrics:

// Registry size
let param_count = state.search_config.config().registry.len();

// Cache statistics (if caching enabled)
if let Some(stats) = state.search_config.cache_stats() {
    println!("Cache hits: {}", stats.hits);
    println!("Cache misses: {}", stats.misses);
    println!("Hit rate: {:.2}%", stats.hit_rate() * 100.0);
}

Common Issues

High memory usage:

• Check registry size: registry.len()
• Review query cache capacity
• Monitor for memory leaks in cache

Slow search queries:

• Enable query cache if disabled
• Check for complex FHIRPath expressions
• Review database indices

Inconsistent results across instances:

• Verify all instances use same FHIR packages
• Check canonical manager synchronization
• Consider Redis pub/sub for real-time updates

Best Practices

For Developers

1. Use Lock-Free APIs - Always call config() to get snapshot, never hold locks
2. Incremental Updates - Prefer upsert() over full reload when possible
3. Clear Caches - Always clear query cache after registry changes
4. Atomic Operations - Use atomic swap for config updates

For Production

1. Package-Based Parameters - Use FHIR packages for production SearchParameters
2. Monitor Cache Hit Rate - Target >70% hit rate for optimal performance
3. Size Query Cache Appropriately - 1000-5000 entries typical for production
4. Log Registry Changes - Enable INFO logging to track parameter updates

For Testing

1. Use Test Helpers - ReloadableSearchConfig::with_registry() for unit tests
2. Mock Canonical Manager - Avoid database dependencies in tests
3. Verify Thread Safety - Test concurrent reads and writes
4. Check Memory Cleanup - Ensure old configs are freed

Future Enhancements

Planned Features

1. Redis Pub/Sub Integration

   • Broadcast registry updates across instances
   • Real-time synchronization in multi-instance deployments

2. Selective Cache Invalidation

   • Only clear cached queries affected by parameter change
   • Preserve unrelated cached queries

3. Parameter Versioning

   • Support multiple versions of same parameter
   • Gradual migration between versions

4. Performance Monitoring

   • Built-in metrics endpoint
   • Prometheus integration
   • Query performance tracking

Research Topics

1. Semantic FHIRPath Validation

   • Validate expressions against resource schemas
   • Type checking at registration time
   • Prevent runtime errors

2. Distributed Registry

   • Shared registry across all instances
   • Consistent reads with local caching
   • Eventually consistent writes

3. Parameter Optimization

   • Analyze query patterns
   • Suggest missing indices
   • Identify unused parameters

References

• FHIR Search Specification
• SearchParameter Resource
• FHIRPath Specification
• Arc-Swap Documentation
• DashMap Documentation