Services¶

Deep dive into BigBrotr's six independent services: how relays are discovered, validated, monitored, how events are archived, and how analytics views are refreshed.

Overview¶

BigBrotr uses six independent async services that share a PostgreSQL database. Each service runs as its own process and can be started, stopped, and scaled independently:

flowchart TD
    DB[("PostgreSQL")]

    SE["Seeder<br/><small>Bootstrap</small>"]
    FI["Finder<br/><small>Discovery</small>"]
    VA["Validator<br/><small>Verification</small>"]
    MO["Monitor<br/><small>Health checks</small>"]
    SY["Synchronizer<br/><small>Event collection</small>"]
    RE["Refresher<br/><small>View refresh</small>"]

    SE --> DB
    FI --> DB
    VA --> DB
    MO --> DB
    SY --> DB
    RE --> DB

    style SE fill:#7B1FA2,color:#fff,stroke:#4A148C
    style FI fill:#7B1FA2,color:#fff,stroke:#4A148C
    style VA fill:#7B1FA2,color:#fff,stroke:#4A148C
    style MO fill:#7B1FA2,color:#fff,stroke:#4A148C
    style SY fill:#7B1FA2,color:#fff,stroke:#4A148C
    style RE fill:#7B1FA2,color:#fff,stroke:#4A148C
    style DB fill:#311B92,color:#fff,stroke:#1A237E

Service	Role	Mode
Seeder	Bootstraps initial relay URLs from a seed file	One-shot
Finder	Discovers new relays from events and external APIs	Continuous
Validator	Verifies URLs are live Nostr relays via WebSocket	Continuous
Monitor	Runs NIP-11 + NIP-66 health checks, publishes kind 10166/30166 events	Continuous
Synchronizer	Collects events from relays using cursor-based pagination	Continuous
Refresher	Refreshes materialized views for analytics queries	Continuous

Services communicate exclusively through the shared PostgreSQL database. There is no direct inter-service communication or dependency ordering.

Service-Database Interactions¶

The following diagram shows which database tables each service reads from and writes to:

flowchart TD
    subgraph Services
        SE["Seeder"]
        FI["Finder"]
        VA["Validator"]
        MO["Monitor"]
        SY["Synchronizer"]
        RE2["Refresher"]
    end

    subgraph Database
        SS["service_state<br/><small>candidates, cursors</small>"]
        RE["relay<br/><small>validated URLs</small>"]
        MD["metadata<br/><small>NIP-11/NIP-66 docs</small>"]
        RM["relay_metadata<br/><small>time-series snapshots</small>"]
        EV["event<br/><small>Nostr events</small>"]
        ER["event_relay<br/><small>event-relay junction</small>"]
        MV["materialized views<br/><small>11 analytics views</small>"]
    end

    SE -->|"write candidates"| SS
    SE -->|"write relays"| RE
    FI -->|"read events"| ER
    FI -->|"write candidates"| SS
    VA -->|"read candidates"| SS
    VA -->|"promote valid"| RE
    VA -->|"update failures"| SS
    MO -->|"read relays"| RE
    MO -->|"write metadata"| MD
    MO -->|"write snapshots"| RM
    SY -->|"read relays"| RE
    SY -->|"read/write cursors"| SS
    SY -->|"write events"| EV
    SY -->|"write junctions"| ER
    RE2 -->|"refresh"| MV

    style SE fill:#7B1FA2,color:#fff,stroke:#4A148C
    style FI fill:#7B1FA2,color:#fff,stroke:#4A148C
    style VA fill:#7B1FA2,color:#fff,stroke:#4A148C
    style MO fill:#7B1FA2,color:#fff,stroke:#4A148C
    style SY fill:#7B1FA2,color:#fff,stroke:#4A148C
    style RE2 fill:#7B1FA2,color:#fff,stroke:#4A148C
    style SS fill:#311B92,color:#fff,stroke:#1A237E
    style RE fill:#311B92,color:#fff,stroke:#1A237E
    style MD fill:#311B92,color:#fff,stroke:#1A237E
    style RM fill:#311B92,color:#fff,stroke:#1A237E
    style EV fill:#311B92,color:#fff,stroke:#1A237E
    style ER fill:#311B92,color:#fff,stroke:#1A237E
    style MV fill:#311B92,color:#fff,stroke:#1A237E

Seeder¶

Purpose: Bootstrap the system by loading relay URLs from a static seed file.

Mode: One-shot (--once flag). Runs once and exits.

Reads: Seed file (static/seed_relays.txt) Writes: service_state (candidates) or relay (direct insert)

How It Works¶

Read the seed file (one URL per line, # comments skipped)
Parse each URL into a Relay object (validates URL format, detects network type)
Insert as candidates via insert_candidates() (default) or directly to the relay table

Tip

Set to_validate: false in the Seeder config to skip validation and insert relays directly. This is useful when seeding with a trusted, pre-validated relay list.

Configuration¶

Field	Type	Default	Description
`seed.file_path`	string	`static/seed_relays.txt`	Path to seed relay URLs file
`seed.to_validate`	bool	`true`	Insert as candidates (true) or directly as relays (false)

API Reference

See bigbrotr.services.seeder for the complete Seeder API.

Finder¶

Purpose: Discover new relay URLs from stored Nostr events and external HTTP APIs.

Mode: Continuous (run_forever, default interval 1 hour)

Reads: event (stored Nostr events), external HTTP APIs Writes: service_state (candidates)

How It Works¶

flowchart TD
    A["Finder.run()"] --> B["_find_from_events()"]
    A --> C["_find_from_api()"]

    B --> D["Scan kind 3<br/><small>contact lists</small>"]
    B --> E["Scan kind 10002<br/><small>NIP-65 relay lists</small>"]
    B --> F["Scan r tags<br/><small>relay references</small>"]

    C --> G["HTTP GET<br/><small>nostr.watch API</small>"]
    C --> H["HTTP GET<br/><small>custom sources</small>"]

    D --> I["Collect URLs"]
    E --> I
    F --> I
    G --> I
    H --> I

    I --> J["insert_candidates()"]

    style A fill:#7B1FA2,color:#fff,stroke:#4A148C
    style J fill:#311B92,color:#fff,stroke:#1A237E

Discovery sources:

Event scanning -- extracts relay URLs from:
- Kind 3 (contact list): content field contains JSON with relay URLs as keys
- Kind 10002 (NIP-65 relay list): r tags contain relay URLs
- Any event with r tags
API fetching -- HTTP requests to external sources:
- Default: nostr.watch online/offline relay list endpoints
- Configurable timeout, SSL verification, delay between requests

Configuration¶

Field	Type	Default	Description
`interval`	float	`3600.0`	Seconds between discovery cycles
`concurrency.max_parallel`	int	`5`	Concurrent API requests
`events.enabled`	bool	`true`	Enable event-based relay discovery
`events.batch_size`	int	`1000`	Events per scanning batch
`events.kinds`	list[int]	`[2, 3, 10002]`	Nostr event kinds to scan
`api.enabled`	bool	`true`	Enable API-based discovery
`api.sources[].url`	string	--	API endpoint URL
`api.delay_between_requests`	float	`1.0`	Delay between API calls

API Reference

See bigbrotr.services.finder for the complete Finder API.

Validator¶

Purpose: Test candidate relay URLs via WebSocket and promote valid ones to the relay table.

Mode: Continuous (run_forever, default interval 8 hours)

Reads: service_state (candidates) Writes: relay (promoted valid relays), service_state (updated failure counts)

How It Works¶

flowchart TD
    A["Validator.run()"] --> B["delete_stale_candidates()"]
    B --> C["delete_exhausted_candidates()"]
    C --> D["fetch_candidate_chunk()"]
    D --> E{Candidates?}
    E -->|No| F["Cycle complete"]
    E -->|Yes| G["Validate in parallel<br/><small>per-network semaphores</small>"]
    G --> H["is_nostr_relay()<br/><small>WebSocket test</small>"]
    H --> I{Valid?}
    I -->|Yes| J["promote_candidates()"]
    I -->|No| K["Increment failure count"]
    J --> D
    K --> D

    style A fill:#7B1FA2,color:#fff,stroke:#4A148C
    style J fill:#1B5E20,color:#fff,stroke:#0D3B0F
    style K fill:#B71C1C,color:#fff,stroke:#7F0000

Delete stale candidates (URLs already in the relay table)
Delete exhausted candidates (exceeded max_failures threshold)
Fetch a chunk of candidates ordered by failure count (ASC) then age (ASC)
Validate in parallel with per-network semaphores via is_nostr_relay(relay, timeout, proxy_url)
Promote valid candidates to the relay table; increment failure count for invalid ones
Repeat until all candidates are processed

Note

The Validator uses is_nostr_relay() which performs a WebSocket handshake and checks for a valid Nostr protocol response. It does not verify event storage or relay policies.

Configuration¶

Field	Type	Default	Description
`interval`	float	`28800.0`	Seconds between validation cycles
`processing.chunk_size`	int	`100`	Candidates per fetch batch
`processing.max_candidates`	int or null	`null`	Max candidates per cycle
`cleanup.enabled`	bool	`false`	Enable stale candidate cleanup
`cleanup.max_failures`	int	`100`	Failure threshold for removal
`networks`	NetworkConfig	--	Per-network timeouts and concurrency

API Reference

See bigbrotr.services.validator for the complete Validator API.

Monitor¶

Purpose: Perform NIP-11 and NIP-66 health checks on all validated relays and publish results as Nostr events.

Mode: Continuous (run_forever, default interval 1 hour)

Reads: relay (validated relays) Writes: metadata, relay_metadata (health check results); publishes Nostr kind 0, 10166, 30166 events

How It Works¶

The Monitor is the most complex service (services/monitor/service.py), handling health checks, event publishing, and NIP-66 tag building in a single module.

Orchestration flow:

run() -- setup (geo, publish profile/announcement), delegate to monitor()
monitor() -- count relays, process chunks, emit metrics
check_chunks() -- async generator yielding (successful, failed) per chunk
check_relay(relay) -- run NIP-11 + all NIP-66 checks, return CheckResult
_persist_results(successful, failed) -- insert metadata to DB
publish_relay_discoveries(successful) -- build and broadcast kind 30166 events
_publish_announcement() -- kind 10166 (monitor capabilities)
_publish_profile() -- kind 0 (monitor profile metadata)

CheckResult (what each relay check produces):

class CheckResult(NamedTuple):
    nip11: RelayMetadata | None
    nip66_rtt: RelayMetadata | None
    nip66_ssl: RelayMetadata | None
    nip66_geo: RelayMetadata | None
    nip66_net: RelayMetadata | None
    nip66_dns: RelayMetadata | None
    nip66_http: RelayMetadata | None

Published Nostr events:

Kind	Type	Content
0	Profile	Monitor name, about, picture (NIP-01)
10166	Announcement	Monitor capabilities, check frequency, supported checks (NIP-66)
30166	Discovery	Per-relay health data: RTT, SSL, DNS, Geo, Net, NIP-11 (addressable, `d` tag = relay URL)

NIP-66 tags produced by monitor/service.py:

Method	Tags Produced
`_add_rtt_tags()`	`rtt-open`, `rtt-read`, `rtt-write`
`_add_ssl_tags()`	`ssl`, `ssl-expires`, `ssl-issuer`
`_add_net_tags()`	`net-ip`, `net-ipv6`, `net-asn`, `net-asn-org`
`_add_geo_tags()`	`g` (geohash), `geo-country`, `geo-city`, `geo-lat`, `geo-lon`, `geo-tz`
`_add_nip11_tags()`	`N` (NIPs), `t` (topics), `l` (languages), `R` (requirements), `T` (types)

Configuration¶

Field	Type	Default	Description
`interval`	float	`3600.0`	Seconds between check cycles
`processing.chunk_size`	int	`100`	Relays per batch
`processing.max_relays`	int or null	`null`	Max relays per cycle
`processing.compute.*`	bool	`true`	Enable computation per metadata type
`processing.store.*`	bool	`true`	Enable persistence per metadata type
`discovery.enabled`	bool	`true`	Publish kind 30166 events
`announcement.enabled`	bool	`true`	Publish kind 10166 events
`networks`	NetworkConfig	--	Per-network timeouts and concurrency

Warning

The Monitor requires the PRIVATE_KEY environment variable for signing published Nostr events and performing NIP-66 write tests.

API Reference

See bigbrotr.services.monitor for the complete Monitor API.

Synchronizer¶

Purpose: Connect to relays, subscribe to events, and archive them to PostgreSQL.

Mode: Continuous (run_forever, default interval 15 minutes)

Reads: relay (validated relays), service_state (cursors) Writes: event, event_relay (archived events and junctions), service_state (updated cursors)

How It Works¶

flowchart TD
    A["Synchronizer.run()"] --> B["Fetch relays from DB"]
    B --> C["Load cursors from service_state"]
    C --> D["_sync_all_relays(relays)<br/><small>TaskGroup + semaphore</small>"]
    D --> E["Per relay:"]
    E --> F["Connect via WebSocket"]
    F --> G["Subscribe with filter<br/><small>kinds, since, until, limit</small>"]
    G --> H["Collect events into EventBatch"]
    H --> I{Batch full?}
    I -->|Yes| J["insert_event_relay<br/>(cascade=True)"]
    I -->|No| K{EOSE received?}
    K -->|Yes| J
    K -->|No| H
    J --> L["Update cursor"]
    L --> M{More events?}
    M -->|Yes| G
    M -->|No| N["Next relay"]

    style A fill:#7B1FA2,color:#fff,stroke:#4A148C
    style J fill:#311B92,color:#fff,stroke:#1A237E

run() -- fetch relays from DB, load cursors, distribute work
_sync_all_relays(relays) -- TaskGroup with semaphore coordination
For each relay: connect via WebSocket, subscribe with filter, collect events
Per-relay cursor tracking via ServiceState with ServiceStateType.CURSOR
Batch insert events + relay junctions via insert_event_relay(cascade=True)
Flush cursor updates periodically

EventBatch -- bounded event buffer:

class EventBatch:
    since: int           # filter start timestamp
    until: int           # filter end timestamp
    limit: int           # max events
    events: list[Event]  # collected events

    def append(event) -> None   # raises OverflowError if full
    def is_full() -> bool
    def is_empty() -> bool

Configuration¶

Field	Type	Default	Description
`interval`	float	`900.0`	Seconds between sync cycles
`filter.kinds`	list[int] or null	`null`	Event kinds to sync (null = all)
`filter.limit`	int	`500`	Events per REQ request
`time_range.use_relay_state`	bool	`true`	Use per-relay incremental cursors
`time_range.lookback_seconds`	int	`86400`	Lookback window from cursor position
`concurrency.max_parallel`	int	`10`	Concurrent relays
`source.from_database`	bool	`true`	Fetch relay list from database
`networks`	NetworkConfig	--	Per-network timeouts and concurrency

API Reference

See bigbrotr.services.synchronizer for the complete Synchronizer API.

Refresher¶

Purpose: Refresh materialized views that power analytics queries.

Mode: Continuous (run_forever, default interval 1 hour)

Reads: Base tables (indirectly, via REFRESH MATERIALIZED VIEW CONCURRENTLY) Writes: 11 materialized views

How It Works¶

Iterate over the configured list of materialized views
Refresh each view individually via its stored function (e.g., relay_metadata_latest_refresh())
Log per-view timing and success/failure
A failure on one view does not prevent subsequent views from refreshing

The Refresher calls views in dependency order: relay_metadata_latest first (because relay_software_counts and supported_nip_counts depend on it), then all remaining views.

Configuration¶

Field	Type	Default	Description
`interval`	float	`3600.0`	Seconds between refresh cycles
`views`	list[string]	all 11 views	Materialized views to refresh

API Reference

See bigbrotr.services.refresher for the complete Refresher API.

Service Lifecycle¶

All services share a common lifecycle managed by BaseService:

statediagram-v2
    [*] --> Created: __init__()
    Created --> Running: __aenter__()
    Running --> Cycling: run_forever()

    state Cycling {
        [*] --> RunCycle: run()
        RunCycle --> WaitInterval: wait(interval)
        WaitInterval --> RunCycle: interval elapsed
        WaitInterval --> [*]: shutdown requested
        RunCycle --> FailureTracking: exception
        FailureTracking --> WaitInterval: consecutive < max
        FailureTracking --> [*]: consecutive >= max
    }

    Cycling --> Cleanup: __aexit__()
    Cleanup --> [*]

run() vs run_forever()¶

Method	Behavior	Use Case
`run()`	Execute a single cycle, return	Testing, one-shot (`--once` flag)
`run_forever()`	Loop: `run()` -> `wait(interval)` -> repeat	Production continuous operation

Failure Handling¶

Each service tracks consecutive failures
After max_consecutive_failures (default 5), the service stops
Set max_consecutive_failures: 0 to disable the limit (never auto-stop)
A successful cycle resets the consecutive failure counter

Graceful Shutdown¶

SIGTERM or SIGINT triggers request_shutdown()
The current cycle completes before exiting
wait() is interruptible -- no waiting for the full interval
stop_grace_period: 60s in Docker Compose ensures time for cleanup

Configuration Reference¶

For complete configuration details including all fields, defaults, constraints, and YAML examples, see the Configuration reference. Key tuning parameters per service:

Service	Key Config	Impact
Seeder	`seed.to_validate`	Skip validation for trusted seed lists
Finder	`events.kinds`, `api.sources`	Control discovery breadth
Validator	`processing.chunk_size`, `cleanup.max_failures`	Throughput vs resource usage
Monitor	`processing.compute.*`, `discovery.enabled`	Which checks to run and publish
Synchronizer	`concurrency.max_parallel`, `filter.kinds`	Archival throughput and scope
Refresher	`views`, `interval`	Which views to refresh and how often

Architecture -- Diamond DAG layer structure and design patterns
Configuration -- Complete YAML configuration reference
Database -- PostgreSQL schema and stored functions
Monitoring -- Prometheus metrics, alerting, and dashboards

Services¶

Overview¶

Service-Database Interactions¶

Seeder¶

How It Works¶

Configuration¶

Finder¶

How It Works¶

Configuration¶

Validator¶

How It Works¶

Configuration¶

Monitor¶

How It Works¶

Configuration¶

Synchronizer¶

How It Works¶

Configuration¶

Refresher¶

How It Works¶

Configuration¶

Service Lifecycle¶

run() vs run_forever()¶

Failure Handling¶

Graceful Shutdown¶

Configuration Reference¶

Related Documentation¶