DownloadToStream hangs indefinitely on network error

[martinknafvework]

Which service(blob, file, queue, table) does this issue concern?

Blob

Which version of the SDK was used?

9.3.2

Which platform are you using? (ex: .NET Core 2.1)

.NET Framework 4.5.2

What problem was encountered?

If I disconnect the network cable while calling DownloadToStream, the client hangs indefinitely.

How can we reproduce the problem in the simplest way?

1. Upload a fairly large blob to blob storage.
2. Run the following code (blockBlob is just a reference to the CloudBlockBlob uploaded in step 1).
3. While the call to DownloadToStream is executing, disconnect the network cable.

This causes the client to hang every time I try it (I've waited 30 minutes). I have not set any default timeouts on the blob client object (using DefaultRequestOptions).

           var blobRequestOptions = new BlobRequestOptions()
              {
                 RetryPolicy = new NoRetry(),
              };
           
           using (var stream = new MemoryStream())
           {
              blockBlob.DownloadToStream(stream, null, blobRequestOptions);
           }

Have you found a mitigation/solution?

Specifying MaximumExecutionTime in the BlobRequestOptions works, but it seems incorrect that the default behavior of the client would be to hang indefinitely.

I was expecting either a default built-in per-request timeout, or being forced to specify a timeout myself. Defaulting to hanging indefinitely seems like a bug.

Maybe I'm missing something?

[mbialecka]

I also see this behavior for other methods, like PutBlock or UploadFromStream on .Net Core 2.1 on a Linux VM. I don't see this behavior on a linux container on Docker for Windows.

[ConnorDickson]

Any update on this? Here is some more info I've found;

• When I unplug my network cable my computer switches to WiFi but the request never resumes
• If I start the download on WiFi and then plug in my network cable the same error occurs
• The “ServerTimeout” property never fails the request or acts as expected in accordance to the Documentation
• The “MaximumExecutionTime” property does fail the request but we don’t want to limit ourselves to a certain time period, especially because we’re dealing with large files

The following code fails 100% of the time if the network is changed during the call.

static void Main(string[] args)
{
    try
    {
        CloudStorageAccount.TryParse("<Connection String>", out var storageAccount);
        var cloudBlobClient = storageAccount.CreateCloudBlobClient();
        var container = cloudBlobClient.GetContainerReference("<Container Reference>");
        var blobRef = container.GetBlockBlobReference("Large Text.txt");
        Stream memoryStream = new MemoryStream();
        BlobRequestOptions optionsWithRetryPolicy = new BlobRequestOptions() { ServerTimeout = TimeSpan.FromSeconds(5), RetryPolicy = new LinearRetry(TimeSpan.FromSeconds(20), 4) };
        blobRef.DownloadToStreamAsync(memoryStream, null, optionsWithRetryPolicy, null).GetAwaiter().GetResult();
        Console.WriteLine("Completed");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Exception: {ex.Message}");
    }
    finally
    {
        Console.WriteLine("Finished");
    }
}

[ransagy]

This makes some of our scenarios extremely difficult right now, Especially cross region downloads. Some of our machines are on WUS2 (GPU) and some of our storage is on WUS. Apparently there are known networking issues between the two, leading to a lot of storage stalls because of this issue.

[ConnorDickson]

Thanks for getting back so quickly! The storage account I was using for testing this issue at first was East US paired with West US. I've just tried it again using UK South paired with UK West and had the exact same problem. I also created a Locally Redundant Storage Account in North Europe and it also hangs indefinitely so I'm not sure it's to do with cross Data Center networking

[ransagy]

Oh, I agree; I just meant that in our case it just made the errors a lot more likely to hang due to networking issues. (without forcing networking issues like pulling a cable, etc)

[dtotzke]

I am seeing the same behaviour with CloudFile.DownloadToFileAsync and CloudFile.UploadFromFileAsync. Disconnecting/reconnecting the network fails to resume the operation. The operation hangs forever and ServerTimeout is ignored. The operations do respect the cancellation token.

FileRequestOptions options = new FileRequestOptions(); options.ServerTimeout = TimeSpan.FromSeconds(30); await cloudFile.DownloadToFileAsync(localFilePath, System.IO.FileMode.Create, null, options, null, progress, cancellationToken).ConfigureAwait(false);
I agree that MaximumExecutionTime is not a good alternative.

[ConnorDickson]

Hey dtotzke, that's the exact same as what I found, so I wrote some code in order to monitor the stream for activity in my stackoverflow question and cancel the request if the download stalls. The code hasn't been battle tested yet but thanks to Mohit for the idea.

[dtotzke]

Thanks @ConnorDickson . I'm using the overload that accepts a IProgress<StorageProgress> to update the UI. I'll likely add monitoring in the progress handler and if there's no progress for 30 or 60 seconds then I'll just cancel the operation and the user can try again.

Progress<StorageProgress> progress = new Progress<StorageProgress>();
progress.ProgressChanged += ProgressChanged;
await cloudFile.UploadFromFileAsync(localFileNameAndPath, null, null, null, progress, cancellationToken).ConfigureAwait(false);

private void ProgressChanged(object sender, StorageProgress e)
{
    // set a flag that a timer event can check and reset
    // if the flag isn't set when the timer fires, there's been no progress so cancel
}

[kasobol-msft]

Fixed in 11.1.4 version
PR: #985