Integrate m3 dsmgpty with hypervisor and various fixes. #166

evanlinjin · 2020-02-21T15:40:45Z

Did you run make format && make check?

Yes

Changes (closes #155):

Fixed RPCClientDialer logic to not return on hypervisor disconnect.
Updated hypervisor config format.
Added hypervisor /pty/{pk} endpoint.
Added logic to prepare unix file.
Improved logging.

How to test this:

Checkout repo skywire-services@feature/m3-dmsgpty-hypervisor
Uncomment replace statements in go.mod for skywire-services and skywire-mainnet repos.
Run the generic integration ENV.
Visor A via Hypervisor: http://localhost:8080/pty/031b80cd5773143a39d940dc0710b93dcccc262a85108018a7a95ab9af734f8055
Visor C via Hypervisor: http://localhost:8080/pty/031b80cd5773143a39d940dc0710b93dcccc262a85108018a7a95ab9af734f8055

Changes (fixes #160, fixes #161, fixes #171):

managedTransport.close() now optionally calls mt.wg.Wait() to avoid hangs if called in mt.Serve().
transport.Manager deletes mTp entry on SaveTransport if mTp has already stopped serving.
Improved various logs.
Added some comments.

How to test this:

Checkout repo skywire-services@feature/m3-dmsgpty-hypervisor
Uncomment replace statements in go.mod for skywire-services and skywire-mainnet repos.
Run the generic integration ENV.
Read further instructions in the respective tickets.

Changes (closes #68):

mTp.redial exponential backoff.
mTp.updateStatuses looks at discovery response codes to see if tp is still registered.

…t into feature/m3-dmsgpty-hypervisor

jdknives · 2020-02-21T15:50:40Z

Please add some documentation for @Senyoret1 before merging so that he can integrate these changes with the frontend.

The following has been commented out: * WIP: Visor endpoints exposed for hypervisor switched to RESTful. * WIP: Various new helper functions for having http over dmsg.

* Updated hypervisor config format. * Added hypervisor /pty/{pk} endpoint. * Added logic to prepare unix file. * Improved logging.

nkryuchkov

Good job!

Regarding the services repo, we also need to adjust integration/hypervisor.json for hypervisor config changes. The script for testing the /exec endpoint is to be modified.

I'm getting the following error on attempt to test the PR:

[2020-02-27T21:15:59+01:00] INFO [snet.dmsgC]: ClientSession.acceptStream() temporary error, continuing... error="dmsg error 306 - request has no associated listener" session=035915c609f71d0c7df27df85ec698ceca0cb262590a54f732e3bbd0cc68d89282

Did I set up anything wrong?

pkg/hypervisor/config.go

pkg/hypervisor/hypervisor.go

pkg/visor/rpc.go

pkg/visor/visor.go

evanlinjin · 2020-02-28T07:03:01Z

Good job!

Regarding the services repo, we also need to adjust integration/hypervisor.json for hypervisor config changes. The script for testing the /exec endpoint is to be modified.

I'm getting the following error on attempt to test the PR:
[2020-02-27T21:15:59+01:00] INFO [snet.dmsgC]: ClientSession.acceptStream() temporary error, continuing... error="dmsg error 306 - request has no associated listener" session=035915c609f71d0c7df27df85ec698ceca0cb262590a54f732e3bbd0cc68d89282
Did I set up anything wrong?

@nkryuchkov Did you test this with skywire-services@feature/m3-dmsgpty-hypervisor? I've slightly updated the integration env to accommodate these changes.

* managedTransport.close() now optionally calls mt.wg.Wait() to avoid hangs if called in mt.Serve(). * transport.Manager deletes mTp entry on SaveTransport if mTp has already stopped serving. * Improved various logs. * Added some comments.

nkryuchkov · 2020-02-28T17:48:00Z

@evanlinjin Yes. I'll have a look again, thanks.

nkryuchkov · 2020-03-02T09:17:02Z

@evanlinjin

I've tested again. The hypervisor.json file in skywire-services@feature/m3-dmsgpty-hypervisor still needs fixing. Instead of

"interfaces": {
  "http_address": ":8080",
  "rpc_addr": ":7080"
}

we need

"http_addr": ":8080"

An error related to accessing the removed /exec endpoint is still present.

Testing against #160, got the following:

Make the calls of step 3 again. You will see the transport is removed from the 0348c... visor, due to the call made on step 4. However, the transport will still be present in the 024ec... visor, which appears to be an error.

Reproduced. Is it intended behavior now?

Repeat the call from step 2, to create the transport again. This time the call will take much more time. In my case the first call is completed in just some ms, but the second one takes about 20 secs.

Could not reproduce. It seems to be fixed.

Make the calls of step 3 again. You will see the transport on the 0348c... visor, but the request for the 024ec... will never be completed.

Could not reproduce.

Call GET /api/visors to get the info about all the visors. The call will never return a response.

Could not reproduce.

evanlinjin · 2020-03-02T11:41:29Z

Note that this PR should pass manual integration tests specified in #152 as well.

* transport.Manager now deletes from 'tps' when the associated mTp stops serving. * The initiating settlement handshake no longer updates transport discovery status. This logic is now moved to ManagedTransport. * ManagedTransport now has a better mechanism to update statuses. * Only the least-significant edge of a transport can redial the underlying connection.

evanlinjin · 2020-03-02T18:21:59Z

@evanlinjin

I've tested again. The hypervisor.json file in skywire-services@feature/m3-dmsgpty-hypervisor still needs fixing. Instead of
"interfaces": {
  "http_address": ":8080",
  "rpc_addr": ":7080"
}
we need
"http_addr": ":8080"
An error related to accessing the removed /exec endpoint is still present.

Testing against #160, got the following:

Make the calls of step 3 again. You will see the transport is removed from the 0348c... visor, due to the call made on step 4. However, the transport will still be present in the 024ec... visor, which appears to be an error.

Reproduced. Is it intended behavior now?

Repeat the call from step 2, to create the transport again. This time the call will take much more time. In my case the first call is completed in just some ms, but the second one takes about 20 secs.

Could not reproduce. It seems to be fixed.

Make the calls of step 3 again. You will see the transport on the 0348c... visor, but the request for the 024ec... will never be completed.

Could not reproduce.

Call GET /api/visors to get the info about all the visors. The call will never return a response.

Could not reproduce.

I think I fixed these. Please retest!

jdknives

Are these logs supposed to remain on [Info] level:

[2020-03-02T22:53:37+01:00] INFO [router]: Removed 0 rules
They pop up every 2 seconds it seems.

Fixes [M2] DMSG transports fail to deliver packets after visor restart #171 in integration env.
Does not fix Routes are not being created again #161 for me. But I saw that there is another PR from @Darkren trying to fix that.

Darkren · 2020-03-03T04:56:50Z

Confirm fix of #171 . Both STCP and DMSG transports work with all of my test scenarios

* mTp.redial exponential backoff. * mTp.updateStatuses looks at discovery response codes to see if tp is still registered. * Various logging improvements. * tpDisc.Client: Made errors analyzable.

nkryuchkov

Great job overall!

pkg/hypervisor/hypervisor.go

pkg/transport/manager.go

pkg/visor/rpc.go

Makefile

cmd/hypervisor/commands/root.go

pkg/transport/managed_transport.go

Darkren

Great job!

Makefile

pkg/router/router.go

evanlinjin · 2020-03-03T17:11:35Z

This is meant to be proof that this PR in fact does fix #160, #161 and #171.
However, YouTube is not showing it in 4k yet.
https://www.youtube.com/watch?v=IUt6ZMzkaBo

@nkryuchkov

As suggested by @nkryuchkov

nkryuchkov

Looks good to me, really great job!

nkryuchkov · 2020-03-04T08:58:01Z

cmd/hypervisor/commands/root.go

-			httpAddr = config.Interfaces.HTTPAddr
-			rpcAddr  = config.Interfaces.RPCAddr
-		)
+		conf := prepareConfig(args)


The function looks nice! 🙂

jdknives

Overall a really great job! All of the issues seem fixed in integration env. Will test in deployment after merge. Only small issues that we may fix:

The terminal art or previous bash history is disappearing bit by bit when using arrow keys to scroll through history. (Note where the cursor is)

There does not seem to be an attempt to actually redial transport upon remote shutdown. When starting remote visor though, the transport is initialized properly by the remote, which means there is no problem with overall functionality right now. Also, we still get this error log upon redial from remote after restart.

[2020-03-04T10:52:57+01:00] WARN [tp:031b80]: Failed to read packet. error="EOF" src="read_loop"
[2020-03-04T10:53:09+01:00] INFO [tp_manager]: recv transport connection request: type(dmsg) remote(031b80cd5773143a39d940dc0710b93dcccc262a85108018a7a95ab9af734f8055)
[2020-03-04T10:53:09+01:00] DEBUG [tp_manager]: TP found, accepting...
[2020-03-04T10:53:09+01:00] DEBUG [tp:031b80]: Performing settlement handshake...
[2020-03-04T10:53:09+01:00] ERROR [transport]: Failed to register transports error="(500)Internal Server Error: {"error":"ID already registered"}

Overall however, great work.

evanlinjin · 2020-03-04T11:00:49Z

Overall a really great job! All of the issues seem fixed in integration env. Will test in deployment after merge. Only small issues that we may fix:

1. The terminal art or previous bash history is disappearing bit by bit when using arrow keys to scroll through history. (Note where the cursor is)

1. There does not seem to be an attempt to actually redial transport upon remote shutdown. When starting remote visor though, the transport is initialized properly by the remote, which means there is no problem with overall functionality right now. Also, we still get this error log upon redial from remote after restart.

[2020-03-04T10:52:57+01:00] WARN [tp:031b80]: Failed to read packet. error="EOF" src="read_loop"
[2020-03-04T10:53:09+01:00] INFO [tp_manager]: recv transport connection request: type(dmsg) remote(031b80cd5773143a39d940dc0710b93dcccc262a85108018a7a95ab9af734f8055)
[2020-03-04T10:53:09+01:00] DEBUG [tp_manager]: TP found, accepting...
[2020-03-04T10:53:09+01:00] DEBUG [tp:031b80]: Performing settlement handshake...
[2020-03-04T10:53:09+01:00] ERROR [transport]: Failed to register transports error="(500)Internal Server Error: {"error":"ID already registered"}

Overall however, great work.

I'm not sure what can be done.
Only the least-significant edge is responsible for redialing now.

志宇 added 5 commits January 30, 2020 23:46

Refactored dmsgpty for easier readability.

8cda7a1

Merge branch 'milestone2' of github.com:SkycoinProject/skywire-mainne…

874bb73

…t into feature/m3-dmsgpty-hypervisor

Changed to use new dmsgpty.

f896ef1

Vendor

a03f7af

Tidy vendor.

1699469

志宇 added 8 commits February 24, 2020 22:27

Paused progress on hypervisor endpoints refactor.

04ade12

The following has been commented out: * WIP: Visor endpoints exposed for hypervisor switched to RESTful. * WIP: Various new helper functions for having http over dmsg.

Re-implemented RPCClientDialer.

68c8de2

Integrated dmsgpty into hypervisor.

69dab69

Format.

ec482bc

Remove legacy integration folder and update vendor.

e44d75d

Changes to integrate dmsgpty

c63edc1

First working implementation of dmsgpty from hypervisor.

487296b

* Updated hypervisor config format. * Added hypervisor /pty/{pk} endpoint. * Added logic to prepare unix file. * Improved logging.

Tidy vendor.

18f482d

evanlinjin marked this pull request as ready for review February 27, 2020 08:57

evanlinjin changed the title ~~[WIP] Integrate m3 dsmgpty with hypervisor pty web ui.~~ Integrate m3 dsmgpty with hypervisor pty web ui. Feb 27, 2020

evanlinjin changed the title ~~Integrate m3 dsmgpty with hypervisor pty web ui.~~ Integrate m3 dsmgpty with hypervisor. Feb 27, 2020

nkryuchkov suggested changes Feb 27, 2020

View reviewed changes

志宇 added 2 commits February 28, 2020 14:19

Make hypervisor getVisors concurrent.

09c5eb7

Tidy vendor.

299f80f

evanlinjin mentioned this pull request Feb 28, 2020

Api/nodes call hangs on hypervisor #143

Closed

志宇 added 2 commits February 28, 2020 15:08

Made changes as requested by @nkryuchkov

bc6f38e

evanlinjin changed the title ~~Integrate m3 dsmgpty with hypervisor.~~ Integrate m3 dsmgpty with hypervisor and various fixes. Feb 28, 2020

evanlinjin mentioned this pull request Mar 2, 2020

Split visor.Start method. #174

Closed

Fix conflicts.

36fa94c

志宇 added 4 commits March 2, 2020 20:18

Made hypervisor ping endpoint a separate function.

bdc72de

Fix merge conflicts.

c769c60

Update vendor.

226086d

志宇 added 2 commits March 3, 2020 02:25

Fix logging.

f92427b

Improved logging.

e847cae

jdknives reviewed Mar 2, 2020

View reviewed changes

Improve logging.

1830d40

jdknives mentioned this pull request Mar 3, 2020

Implemented backoff for transport reconnection #148

Closed

This comment has been minimized.

Sign in to view

Added exponential backoff to mTp.redial and various fixes.

447862e

* mTp.redial exponential backoff. * mTp.updateStatuses looks at discovery response codes to see if tp is still registered. * Various logging improvements. * tpDisc.Client: Made errors analyzable.

evanlinjin mentioned this pull request Mar 3, 2020

Rename Loop to RouteGroup for hypervisor endpoints and RPC. #180

Closed

1 task

志宇 added 2 commits March 3, 2020 19:00

Format.

0e696f0

Make travis happy.

b16defe

nkryuchkov suggested changes Mar 3, 2020

View reviewed changes

Darkren reviewed Mar 3, 2020

View reviewed changes

Makefile Outdated Show resolved Hide resolved

pkg/router/router.go Outdated Show resolved Hide resolved

Changes as suggested by @nkryuchkov and @Darkren

bf93616

志宇 added 4 commits March 4, 2020 16:01

Fix merge conflicts.

33ec06d

Separate /cmd/hypervisor/commands/root.go:rootCmd() into smaller funcs.

608c380

As suggested by @nkryuchkov

Update vendor.

047271b

Update vendor.

8341370

nkryuchkov approved these changes Mar 4, 2020

View reviewed changes

jdknives approved these changes Mar 4, 2020

View reviewed changes

jdknives merged commit 52fe903 into skycoin:milestone2 Mar 4, 2020

jdknives mentioned this pull request Mar 4, 2020

Allow to use hypervisor with dmsgpty #132

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate m3 dsmgpty with hypervisor and various fixes. #166

Integrate m3 dsmgpty with hypervisor and various fixes. #166

evanlinjin commented Feb 21, 2020 •

edited

Loading

jdknives commented Feb 21, 2020

nkryuchkov left a comment

evanlinjin commented Feb 28, 2020 •

edited

Loading

nkryuchkov commented Feb 28, 2020

nkryuchkov commented Mar 2, 2020

evanlinjin commented Mar 2, 2020

evanlinjin commented Mar 2, 2020 •

edited

Loading

jdknives left a comment

Darkren commented Mar 3, 2020

This comment has been minimized.

nkryuchkov left a comment

Darkren left a comment

evanlinjin commented Mar 3, 2020

nkryuchkov left a comment

nkryuchkov Mar 4, 2020

jdknives left a comment

evanlinjin commented Mar 4, 2020

Integrate m3 dsmgpty with hypervisor and various fixes. #166

Integrate m3 dsmgpty with hypervisor and various fixes. #166

Conversation

evanlinjin commented Feb 21, 2020 • edited Loading

jdknives commented Feb 21, 2020

nkryuchkov left a comment

Choose a reason for hiding this comment

evanlinjin commented Feb 28, 2020 • edited Loading

nkryuchkov commented Feb 28, 2020

nkryuchkov commented Mar 2, 2020

evanlinjin commented Mar 2, 2020

evanlinjin commented Mar 2, 2020 • edited Loading

jdknives left a comment

Choose a reason for hiding this comment

Darkren commented Mar 3, 2020

This comment has been minimized.

nkryuchkov left a comment

Choose a reason for hiding this comment

Darkren left a comment

Choose a reason for hiding this comment

evanlinjin commented Mar 3, 2020

nkryuchkov left a comment

Choose a reason for hiding this comment

nkryuchkov Mar 4, 2020

Choose a reason for hiding this comment

jdknives left a comment

Choose a reason for hiding this comment

evanlinjin commented Mar 4, 2020

evanlinjin commented Feb 21, 2020 •

edited

Loading

evanlinjin commented Feb 28, 2020 •

edited

Loading

evanlinjin commented Mar 2, 2020 •

edited

Loading