Duration of *PublishVolume requests

[jsafrane]

The spec does not describe any duration requirements of ControllerPublishVolume and NodePublishVolume. I my experience these operations can take quite long time, especially when things go wrong and start timing out. I've seen EBS volume in "Attaching" state for several days on AWS.

How long should CO wait for response from these calls? It should be defined in CSI and limited to minutes (or to timeout set by CO).

Alternatively, CO could monitor the workloads it needs to run and cancel *PublishVolume call when corresponding workload was either rescheduled to a different node or withdrawn completely. *UnpublishVolume call could be the cancellation signal - it says CO does not want the volume attached / mounted to a node any longer, regardless what's current state of attaching / mounting of the volume. (Note that this breaks assumption that there is no more than one call “in-flight” per volume at a given time)

[akutz]

Hi @jsafrane,

So the general idea is that for v0.1 of the spec, a call to any RPC should be allowed to timeout since all responses involving volumes should be idempotent. Ideally the SP does not place requests on the same thread/goroutine as the incoming gRPC request as this has potential for a timeout to completely disrupt the actual work. However, even if that should occur, my advice is that SPs should assume nothing with regards to the state of the node when an RPC request is received. When an SP handles and RPC, the SP should at that time attempt to determine if the state implied or contained by the RPC's payload matches the actual state on the node or storage platform.

Again, that's just my two cents...

[jdef]

The gRPC protocol has a built-in notion of timeouts. Perhaps a CO and plugin could leverage that to communicate expectations? https://grpc.io/docs/guides/wire.html Also related, I see this, but it looks like it's per-connection: https://godoc.org/google.golang.org/grpc#WithTimeout Browsing various threads, some comments suggest that deadlines from a Context might be propagated from gRPC client to server. But I haven't seen any code that proves that yet (and I haven't tested it).

…

On Tue, Oct 24, 2017 at 1:57 PM, Schley Andrew Kutz < ***@***.***> wrote: Hi @jsafrane <https://github.com/jsafrane>, So the general idea is that for v0.1 of the spec, a call to any RPC should be allowed to timeout since all responses involving volumes should be idempotent. Ideally the SP does not place requests on the same thread/goroutine as the incoming gRPC request as this has potential for a timeout to completely disrupt the actual work. However, even if that should occur, my advice is that SPs should assume nothing with regards to the state of the node when an RPC request is received. When an SP handles and RPC, the SP should at that time attempt to determine if the state implied or contained by the RPC's payload matches the actual state on the node or storage platform. Again, that's just my two cents... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#128 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACPVLK4_N3n2f3KKlBuwlpy_ESfphZTpks5sviUagaJpZM4QEYTB> .

[jieyu]

@jsafrane The spec does say that multiple concurrent calls are possible (and unavoidable in case of CO failures):

However, in some circumstances, the CO MAY lose state (for example when the CO crashes and restarts), and MAY issue multiple calls simultaneously for the same volume. The Plugin, SHOULD handle this as gracefully as possible

For instance, a NodePublishVolume is in progress, and CO crashes. CO checkpointed the fact that it has a pending NodePublishVolume that it does not the result yet. The CO might choose to call either NodePublishVolume or NodeUnpublishVolume (this is actually mentioned in the spec):

NodePublishVolume: This operation MUST be idempotent. If this RPC failed, or the CO does not know if it failed or not, it MAY choose to call NodePublishVolume again, or choose to call NodeUnpublishVolume.

If CO decides to call NodeUnpublishVolume, I see two ways that a plugin can do:

1. Queue NodeUnpublishVolume operation (not return a result) until the pending publish finishes.
2. Treat that as a cancellation signal, and returns an Error for the pending NodePublishVolume, and return a success for the NodeUnpublishVolume.

Does that address your concern?

[jsafrane]

Thanks, that addresses my concern, however there should be "the right way" how to time out, regardless if it is internal gRPC mechanism or some outer one. It should be clearly specified so volume plugins are not surprised and all COs use the same one.

[saad-ali]

We should clarify this in spec:

CO ultimately decides what the max time is that they are willing to wait for a call (it is not a negotiation between plugin and CO). A call can timeout and be retried and due to idepotency requirements, should continue where it left off. The only way to cancel a call is to do the "unpublish" or "delete" operation.