Handle network errors/stalls

[hannahhoward]

Goals

Using ipfs/go-graphsync#102, provide better information about what's going on with a request and when it stalls.

Implementation

• Change the FSM behvaior of Disconnected - remove the PeerDisconnected state, and instead just have it set an informational message that we believe the other peer to be stalled. The reason we want to do this is the actual idea behavior here should be that we go into a PeerDisconnected state, and when we come out of it, we reset to the state we were in before the disconnection. Our FSM module really doesn't support this well, so for the time being, I'm just removing the state change and setting a message -- we weren't really doing anything based on being in PeerDisconnected and it was ultimately a black-hole state. (I think PeerDisconnected was meant to handle the fact that graphsync requests were going into an error state when the peers were unlinked -- due to a message send failure in OnChannelCompleted)
• Change the FSM behavior of Restart -- it actually should do nothing to the state. The only reason we were changing it to Ongoing was to resolve the PeerDisconnected state, and there are several issues that could arise from forcing into the Ongoing state -- we should instead preserve the state of the request overall!
• Convert a channel removed due to inactivity to an Error -- this is not a request cancelled by a user but rather something bad that happened
• Add a new event to the Events interface -- OnRequestDisconnected, which occurs when the transport protocol believes there's been a network disconnection (different from OnRequestTimedOut which occurs when request is cancelled due to a restart)
• Convert ChannelRemoveTimeout to a configurable param on the data transfer manager
• Hook into the new NetworkError hook in go-graphsync to detect disconnections
• Convert some message sent failure to request disconnections rather than channel errors -- cause data transfer is an ongoing process, we shouldn't fail the channel entirely just cause a message doesn't go through -- we may want to restart the channel
• Buff out a few tests for the graphsync transport protocol

[aarshkshah1992]

@hannahhoward For my own curiosity:

What's the advantage of defining:

type errorType string

func (e errorType) Error() string {
	return string(e)
}

rather than using errors.New()

[hannahhoward]

it allows you to use const instead of var -- there is a danger of var being changed -- technically anyone using the module can do so.

[aarshkshah1992]

@hannahhoward Great stuff ! Really learn a lot from you simplify things.

Just one review comment.

[aarshkshah1992]

What happens if a channel times/out disconnects and then the FSM/data-trasfer crashes ? I think we should have a handler for this state that starts this wait even when the FSM restarts. Does that make sense ?

[hannahhoward]

yea we can address this when we have a better way to actually track disconencts other than the Message -- see comment above about challenges of essentially pushing state