find providers update plan

[xcolwell]

The goals of this update are to:

1. modernize the find providers algorithm to produce higher quality matches
2. remove dependency on external ip meta data and derive all ip meta data from local data sources/ephemerally

Progress

• Phase 1
• Phase 1.b
• Phase 2
• Phase 3

Phase 1

For each connection, store the locally derived hosting status as net_type_hosting2. This will allow comparison to the current data.

Phase 1.b.

For each match making, export anonymous debug statistics about the client ranks and the chosen window. Expose an API to download the stats.

Store the raw stats in a protobuf dump in a redis list, with expiring key per time block. Merge current and previous block per lookup.

Store an anonymous exported id per client_id.

Phase 2

For each connection, measure the latency. Compute the expected latency, and store both latency and expected_latency

Phase 3

Adjust the market algorithm as follows:

For each connection, load quality, latency, and reliability metrics:

• quality metrics (different combo for quality and speed targets)
• latency
• expected_latency
• client_reliability_weight

Shuffle clients in multiple passes by:

1. Latency ascending
2. Reliability weight descending (or unreliability = 1 - reliability ascending)

Note that reliability weight already factors in duplicate ip hashes by distributing the weight evenly across clients with the same hash.

In this way the highest precedence will be reliability with secondary precedence latency.

Compute the tier as sum:

• quality metrics
• case proxy_latency_threshold<=latency-expected_latency then 1
• latency/latency_band

Stable sort by tier to compute the final selection ordering.