Refreshing the Feature flag SDK cache which is loading from CDN is not working properly

[sathishkumarkaliavaradhan]

Hi All,

We are using the Azure Feature flag in WPF application, which gets connected via CDN connection. (WPF -> CDN -> Azure Functions -> App Configuration)

I have three methods one is initialize and another is refresher and third to read the feature flag value.

Initialize - Method called only once
Refresher - Called periodically
Read Value - Called always while reading the value.

/// <inheritdoc/>
public override async Task<bool> InitAsync(CancellationToken cancellationToken)
{
    this.logger.WriteInfoEx($"Initializing the remote configuration client, loading the feature configuration from CDN source.");
    var configurationBuilder = new ConfigurationBuilder();            
    configurationBuilder.AddAzureAppConfiguration(options =>
    {
        // Disabling replica discovery to avoid potential edge-region inconsistencies and ensure stable feature flag resolution.
        // For more details, refer to the documentation: [Insert relevant documentation link here].
        options.ReplicaDiscoveryEnabled = false;
        options.Connect(connectionString)
        .ConfigureStartupOptions(startupOptions =>
        {
            // Set the time-out for the initial configuration load
            startupOptions.Timeout = TimeSpan.FromSeconds(DefaultFeatureFlagConnectionTimeoutInSeconds);
        })
        .UseFeatureFlags(config => config.SetRefreshInterval(Configuration.ConfigurationRefreshInterval));
        this.configurationRefresher = options.GetRefresher();
    }, true);

    this.logger.WriteInfoEx($"Completed loading the feature configuration from CDN source.");

    var configuration = configurationBuilder.Build();

    // Initialize Feature Management
    this.logger.WriteInfoEx($"Initializing the feature management service collection.");
    var serviceCollection = new ServiceCollection();
    serviceCollection.Configure<FeatureManagementOptions>(options =>
    {
        options.IgnoreMissingFeatures = false;
        // The 'IgnoreMissingFeatureFilters' flag cannot be used in combination with a feature of requirement type 'All'.
        options.IgnoreMissingFeatureFilters = false;                
    });

    serviceCollection.AddSingleton<IConfiguration>(configuration)
        .AddFeatureManagement()
        .AddFeatureFilter<ContextualTargetingFilter>()
        .AddFeatureFilter<DeviceFilter>();

    var serviceProvider = serviceCollection.BuildServiceProvider();
    this.featuremanager = serviceProvider.GetRequiredService<IFeatureManager>();

    this.logger.WriteInfoEx($"Completed the feature management service collection.");

    this.RefreshFeatureFlagConfigurationAsync(cancellationToken).AsBackgroundTask(
            "RemoteConfigurationFeatureProvider background refresh",
            options: TaskContinuationOptions.LongRunning, cancellationToken: cancellationToken);

    return await base.InitAsync(cancellationToken);
}

private async Task RefreshFeatureFlagConfigurationAsync(CancellationToken cancellationToken)
{
    while (!cancellationToken.IsCancellationRequested)
    {
        try
        {
            this.logger.WriteInfoEx("Refreshing the remote configuration");

            var refreshResult = await this.configurationRefresher.TryRefreshAsync(cancellationToken).ConfigureAwait(false);

            this.logger.WriteInfoEx($"Remote configuration refresh completed. Changes applied: {refreshResult}");

            this.logger.WriteInfoEx($"Background task going to wait for {this.Configuration.ConfigurationRefreshInterval} interval.");
            await Task.Delay(this.Configuration.ConfigurationRefreshInterval, cancellationToken).ConfigureAwait(false);
        }
        catch (Exception ex) when (!ex.IsCritical())
        {
            this.logger.WriteErrorEx($"Exception occurred during background refresh of remote configuration in RemoteConfigurationFeatureProvider. Exception: {ex.Message}");
        }
    }
}

/// <inheritdoc/>
protected override async Task<object> ReadProviderValue<T>(string featureId)
{
    try
    {                
        bool remoteFeatureValue = await this.featuremanager.IsEnabledAsync(featureId, DeviceContext).ConfigureAwait(false);
        return (T)Convert.ChangeType(remoteFeatureValue, typeof(T));
    }
    catch (FeatureManagementException ex) when (ex.Error == FeatureManagementError.MissingFeature)
    {
        // Null indicates that feature is not configured to the framework, logging handled by SDK
        this.logger.WriteWarningEx($"Feature '{featureId}' is not configured in the remote configuration.");                
        return null;
    }
}

Below is our SDK version

Microsoft.FeatureManagement - 4.0.0
Microsoft.Extensions.Configuration.AzureAppConfiguration - 8.0.0

While changing the value in Azure reflects in CDN response but Feature flag SDK still returns old value only.

bool remoteFeatureValue = await this.featuremanager.IsEnabledAsync(featureId, DeviceContext).ConfigureAwait(false);
return (T)Convert.ChangeType(remoteFeatureValue, typeof(T));

@jimmyca15 @rossgrambo @zhiyuanliang-ms Could you please let me know what's wrong here?

[sathishkumarkaliavaradhan]

It's used to work fine in WPF Application and now we have moved this logic into Windows service and it's failing now.

[sathishkumarkaliavaradhan]

@jimmyca15 @rossgrambo @zhiyuanliang-ms do you have any update on this issue?

[jimmyca15]

Hello @sathishkumarkaliavaradhan, It's difficult to pin down where the issue is occurring. Are you able to reproduce by calling App Configuration directly to take CDN configuration out of the equation?

[sathishkumarkaliavaradhan]

@jimmyca15 Thanks for your response.
Calling App configuration directly works fine but via CDN configuration is not working.
We can see the actual value of feature flag in CDN response JSON but SDK is not returning the correct value.

[jimmyca15]

@sathishkumarkaliavaradhan This would make me think that the CDN is causing the issue. However your mention that this worked before some migration in your application seems counter to that.

App Configuration has released a supported integration with Azure Front Door to support CDN scenarios. Using this may alleviate this and any future issues.

Alternatively, if you have a way to reproduce this issue, you can enable logging in the configuration provider to help pin down if the CDN is serving some stale responses unexpectedly. Here is a snippet to enable verbose logging at the Azure SDK level to capture calls (requests + response) to Azure App Configuration. In this case, it will be to your CDN though.

using Azure.Core.Diagnostics;
using var listener = AzureEventSourceListener.CreateConsoleLogger(System.Diagnostics.Tracing.EventLevel.Verbose);

IConfiguration configuration = new ConfigurationBuilder()
    .AddAzureAppConfiguration(o =>
    {
        o.Connect(new Uri("https://contoso.azconfig.io"), new DefaultAzureCredential());

        o.Select("_");

        o.ConfigureClientOptions(o =>
        {
            o.Diagnostics.IsLoggingContentEnabled = true;
            o.Diagnostics.LoggedQueryParameters.Add("*");
        });

        o.UseFeatureFlags();
    })
    .Build();

Of course, don't paste the results back here as responses may contain sensitive details. This should give you a way to see if the CDN is caching things for longer than expected.