Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CT: net::ERR_FAILED in phet-io-wrappers-tests #185

Open
zepumph opened this issue May 18, 2023 · 21 comments
Open

CT: net::ERR_FAILED in phet-io-wrappers-tests #185

zepumph opened this issue May 18, 2023 · 21 comments
Assignees
@mattpen
Labels
status:on-hold

Comments

@zepumph
Copy link
Member

zepumph commented May 18, 2023

About half of the errors on CT are this right now

balloons-and-static-electricity : phet-io-wrappers-tests : assert
Tried to run 3 times, never completed, failure: Error: Error: Did not get next-test message in 400000ms: ["balloons-and-static-electricity","phet-io-wrappers-tests","assert"]
[URL] https://sparky.colorado.edu/continuous-testing/aqua/html/qunit-test.html?url=..%2F..%2Fct-snapshots%2F1684429403510%2Fphet-io-wrappers%2Fphet-io-wrappers-tests.html%3Fsim%3Dballoons-and-static-electricity%26phetioDebug%3Dtrue%26phetioWrapperDebug%3Dtrue&duration=600000&testInfo=%7B%22test%22%3A%5B%22balloons-and-static-electricity%22%2C%22phet-io-wrappers-tests%22%2C%22assert%22%5D%2C%22snapshotName%22%3A%22snapshot-1684429403510%22%2C%22timestamp%22%3A1684430949895%7D
[NAVIGATED] https://sparky.colorado.edu/continuous-testing/aqua/html/qunit-test.html?url=..%2F..%2Fct-snapshots%2F1684429403510%2Fphet-io-wrappers%2Fphet-io-wrappers-tests.html%3Fsim%3Dballoons-and-static-electricity%26phetioDebug%3Dtrue%26phetioWrapperDebug%3Dtrue&duration=600000&testInfo=%7B%22test%22%3A%5B%22balloons-and-static-electricity%22%2C%22phet-io-wrappers-tests%22%2C%22assert%22%5D%2C%22snapshotName%22%3A%22snapshot-1684429403510%22%2C%22timestamp%22%3A1684430949895%7D
[NAVIGATED] about:blank
[NAVIGATED] https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/phet-io-wrappers/phet-io-wrappers-tests.html?sim=balloons-and-static-electricity&phetioDebug=true&phetioWrapperDebug=true
[CONSOLE] Failed to load resource: the server responded with a status of 404 (): https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/phet-io-wrappers/phet-io-wrappers-strings_en.json
[CONSOLE] enabling assert
[NAVIGATED] about:blank
[NAVIGATED] https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/balloons-and-static-electricity/balloons-and-static-electricity_en.html?brand=phet-io&ea&debugger&postMessageOnError
[CONSOLE] enabling assert
[NAVIGATED] about:blank
[NAVIGATED] https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/balloons-and-static-electricity/balloons-and-static-electricity_en.html?brand=phet-io&ea&debugger&postMessageOnError
[CONSOLE] enabling assert
[NAVIGATED] about:blank
[NAVIGATED] https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/balloons-and-static-electricity/balloons-and-static-electricity_en.html?brand=phet-io&ea&debugger&postMessageOnError
[CONSOLE] enabling assert
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/phet-core/js/Poolable.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/axon/js/DynamicProperty.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/joist/js/i18n/localeProperty.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/chipper/js/chipper.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/scenery/js/accessibility/pdom/PDOMUtils.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/scenery/js/util/colorProfileProperty.js
[CONSOLE] Failed to load resource: net::ERR_FAILED: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684429403510/chipper/dist/js/scenery/js/util/ProfileColorProperty.js

id: "Bayes Node Puppeteer"
Snapshot from 5/18/2023, 11:03:23 AM
@zepumph zepumph self-assigned this May 18, 2023
@zepumph
Copy link
Member Author

zepumph commented May 18, 2023

https://stackoverflow.com/questions/22665232/what-can-cause-chrome-to-give-an-neterr-failed-on-cached-content-against-a-ser helped me reproduce this while recording chrome network traffic. It was important to reproduce with a link like this (from CT) and this chrome recording link

chrome://net-export
View log with: https://netlog-viewer.appspot.com/#import

Reproduce with https://sparky.colorado.edu/continuous-testing/aqua/html/qunit-test.html?url=..%2F..%2Fct-snapshots%2F1684429403510%2Fphet-io-wrappers%2Fphet-io-wrappers-tests.html%3Fsim%3Dballoons-and-static-electricity%26phetioDebug%3Dtrue%26phetioWrapperDebug%3Dtrue&duration=600000&testInfo=%7B%22test%22%3A%5B%22balloons-and-static-electricity%22%2C%22phet-io-wrappers-tests%22%2C%22assert%22%5D%2C%22snapshotName%22%3A%22snapshot-1684429403510%22%2C%22timestamp%22%3A1684430949895%7D

@zepumph
Copy link
Member Author

zepumph commented May 19, 2023 •
edited
Loading

Here is the log viewer for the failed request:
'

1099607: URL_REQUEST
https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684350529851/chipper/dist/js/scenery/js/overlays/PointerOverlay.js
Start Time: 2023-05-17 14:18:51.718

t=105060 [st=  0] +CORS_REQUEST  [dt=535]
                   --> cors_preflight_policy = "consider_preflight"
                   --> headers = "sec-ch-ua: \"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"\r\nOrigin: https://sparky.colorado.edu\r\nsec-ch-ua-mobile: ?0\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36\r\nsec-ch-ua-platform: \"Windows\"\r\nAccept: */*\r\n\r\n"
                   --> is_revalidating = false
                   --> method = "GET"
                   --> url = "https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684350529851/chipper/dist/js/scenery/js/overlays/PointerOverlay.js"
t=105060 [st=  0]    CHECK_CORS_PREFLIGHT_REQUIRED
                     --> preflight_required = false
t=105060 [st=  0]   +REQUEST_ALIVE  [dt=535]
                     --> priority = "MEDIUM"
                     --> traffic_annotation = 101845102
                     --> url = "https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684350529851/chipper/dist/js/scenery/js/overlays/PointerOverlay.js"
t=105060 [st=  0]      NETWORK_DELEGATE_BEFORE_URL_REQUEST  [dt=0]
t=105060 [st=  0]     +URL_REQUEST_START_JOB  [dt=535]
                       --> initiator = "https://sparky.colorado.edu"
                       --> load_flags = 16386 (BYPASS_CACHE | SUPPORT_ASYNC_REVALIDATION)
                       --> method = "GET"
                       --> network_isolation_key = "https://colorado.edu https://colorado.edu"
                       --> request_type = "other"
                       --> site_for_cookies = "SiteForCookies: {site=https://colorado.edu; schemefully_same=true}"
                       --> url = "https://sparky.colorado.edu/continuous-testing/ct-snapshots/1684350529851/chipper/dist/js/scenery/js/overlays/PointerOverlay.js"
t=105061 [st=  1]        COMPUTED_PRIVACY_MODE
                         --> privacy_mode = "disabled"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, EXCLUDE_NOT_ON_PATH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        COOKIE_INCLUSION_STATUS
                         --> operation = "send"
                         --> status = "INCLUDE, DO_NOT_WARN"
t=105061 [st=  1]        NETWORK_DELEGATE_BEFORE_START_TRANSACTION  [dt=475]
t=105536 [st=476]        HTTP_CACHE_GET_BACKEND  [dt=0]
t=105536 [st=476]        HTTP_CACHE_DOOM_ENTRY  [dt=3]
t=105539 [st=479]        HTTP_CACHE_CREATE_ENTRY  [dt=3]
t=105542 [st=482]        HTTP_CACHE_ADD_TO_ENTRY  [dt=8]
t=105550 [st=490]       +HTTP_STREAM_REQUEST  [dt=3]
t=105550 [st=490]          HTTP_STREAM_JOB_CONTROLLER_BOUND
                          [ --> source_dependency = 1100774 (HTTP_STREAM_JOB_CONTROLLER)](https://netlog-viewer.appspot.com/#events&s=1100774)
t=105553 [st=493]          HTTP_STREAM_REQUEST_BOUND_TO_JOB
                          [ --> source_dependency = 1100775 (HTTP_STREAM_JOB)](https://netlog-viewer.appspot.com/#events&s=1100775)
t=105553 [st=493]       -HTTP_STREAM_REQUEST
t=105553 [st=493]       +URL_REQUEST_DELEGATE_CONNECTED  [dt=0]
t=105553 [st=493]          LOCAL_NETWORK_ACCESS_CHECK
                           --> client_address_space = "public"
                           --> resource_address_space = "public"
                           --> result = "allowed-no-less-public"
t=105553 [st=493]       -URL_REQUEST_DELEGATE_CONNECTED
t=105595 [st=535]     -URL_REQUEST_START_JOB
                       --> net_error = -2 (ERR_FAILED)
t=105595 [st=535]      URL_REQUEST_DELEGATE_RESPONSE_STARTED  [dt=0]
t=105595 [st=535] -CORS_REQUEST
t=105595 [st=535] -URL_REQUEST_DELEGATE_RESPONSE_STARTED
t=105595 [st=535] -REQUEST_ALIVE
                   --> net_error = -2 (ERR_FAILED)

That is not very helpful to me, but I guess it is good to note.

@mattpen, could we please work to brainstorm what may be happening here? I can't figure out if this is even getting the nginx.

@mattpen
Copy link
Contributor

mattpen commented May 19, 2023 •
edited
Loading

It looks like the 404 is correct to me. Did a snapshot get deleted or renamed somehow?

[mape5853@sparky ct-snapshots]$ cd 1684429403510
-bash: cd: 1684429403510: No such file or directory
[mape5853@sparky ct-snapshots]$ pwd
/data/share/phet/continuous-testing/ct-main/ct-snapshots
[mape5853@sparky ct-snapshots]$

@zepumph
Copy link
Member Author

zepumph commented May 23, 2023

@mattpen, @chrisklus, @zepumph had a meeting. In it, we looked into nginx error logs and found a pretty consistent one like:

SSL_read() failed (SSL: error:0A000126:SSL routines::unexpected eof while reading) while processing HTTP/2 connection

Then after googling we updated to nginx 1.22, and it went away, but that didn't solve our problem.

We didn't make any more progress in the chrome net-export tool, as there really isn't much else to see there.

The best we did was found that this is only a problem with Chrome, and not firefox or safari, so we would like to switch bayes to use firefox, as a workaround, and see if that silences these errors.

Thanks @mattpen and @chrisklus for all your time!

@zepumph
Copy link
Member Author

zepumph commented May 24, 2023

After trying to run firefox on bayes (RHEL 7), I ran into an error that pointed me to https://access.redhat.com/solutions/2853631 and https://bugzilla.redhat.com/show_bug.cgi?id=1369859.

So until we figure out the RHEL 8 upgrade, I'm just going to turn off the bayes clients.

@zepumph
Copy link
Member Author

zepumph commented May 24, 2023

All chrome clients have been turned off on bayes. Unassigning until https://github.com/phetsims/special-ops/issues/242 is done.

@zepumph zepumph removed their assignment May 24, 2023
zepumph added a commit that referenced this issue May 30, 2023
@zepumph
comment out puppeteer clients for bayes, #185
f7087ac
@zepumph
Copy link
Member Author

zepumph commented Jun 22, 2023

Interestingly enough. It looks like this has to do with using https://sparky.colorado.edu as a URL, because on sparky for its local clients, I swapped from 127.0.0.1 and ran into this again. So perhaps this has something to do with things outside of our node server. I'm not sure how to proceed, but would like to touch base with @mattpen one more time.

@mattpen
Copy link
Contributor

mattpen commented Jun 23, 2023

Does it happen if you use 128.138.93.172? If not, it may be a problem with the DNS server or queries. If it does happen with this IP ... I'm really confused as it looks like after the DNS lookup it would essentially follow the same path as localhost or 127.0.0.1.

[mape5853@sparky ~]$ nslookup sparky.colorado.edu
Server:		128.138.240.1
Address:	128.138.240.1#53

Name:	sparky.colorado.edu
Address: 128.138.93.172

[mape5853@sparky ~]$ tracepath 128.138.93.172
 1:  sparky.colorado.edu                                   0.082ms reached
     Resume: pmtu 65535 hops 1 back 1 

[mape5853@sparky ~]$ tracepath 127.0.0.1
 1:  localhost                                             0.091ms reached
     Resume: pmtu 65535 hops 1 back 1

zepumph added a commit that referenced this issue Jun 23, 2023
@zepumph
try sparky's ip address instead of localhost or sparky.colorado.edu, …
7003eab 
…#185
@zepumph
Copy link
Member Author

zepumph commented Jun 23, 2023

I'll try it out!

zepumph added a commit that referenced this issue Jun 26, 2023
@zepumph
update doc and turn back on bayes with static IP address, #185
d15fa6a
@zepumph
Copy link
Member Author

zepumph commented Jun 26, 2023

When trying to use https://128.138.93.172/ as the server name for clients on bayes (testing sparky), I get this error:

Fatal error: Hostname/IP does not match certificate's altnames: IP: 128.138.93.172 is not in the cert's list

This is not an issue when using clients directly from sparky.

I am also able to ping the IP address. Thoughts?

@mattpen
Copy link
Contributor

mattpen commented Jun 26, 2023 •
edited
Loading

Might need to add 128.138.93.172 as a hostname in the nginx config?

Hostname/IP does not match certificate's altnames:

I think we might need to generate a new SSL cert that has the IP as an alternate name. Maybe we should just use HTTP instead of SSL. I wonder if there is a way we can allow HTTP from a specific set of IPs instead of the internet in general. Or maybe it just doesn't even matter, the phet sims are accessible over http and we have no plan to change it, e.g. http://phet.colorado.edu/sims/html/friction/latest/friction_all.html

@zepumph
Copy link
Member Author

zepumph commented Jun 26, 2023

I am blocked by http because for testing, I need to be able to postMessage to https://phet-io.colorado.edu over in https://github.com/phetsims/phet-io/issues/1944.

Looks like http://phet-io.colorado.edu/sims/phet-io-test-sim/2.12.0/ redirects to https, so perhaps I'm still blocked about this?

@zepumph
Copy link
Member Author

zepumph commented Jun 27, 2023

CT has not been testing since yesterday becuase of the altnames issue:

166|ct-nod | 2023-06-27T14:36:46: (node:4105856) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
166|ct-nod | (Use `node --trace-warnings ...` to show where the warning was created)
166|ct-nod | You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
166|ct-nod | Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames: IP: 128.138.93.172 is not in the cert's list:
166|ct-nod |     at new NodeError (node:internal/errors:399:5)
166|ct-nod |     at Object.checkServerIdentity (node:tls:337:12)
166|ct-nod |     at TLSSocket.onConnectSecure (node:_tls_wrap:1550:27)
166|ct-nod |     at TLSSocket.emit (node:events:513:28)
166|ct-nod |     at TLSSocket._finishInit (node:_tls_wrap:959:8)
166|ct-nod |     at ssl.onhandshakedone (node:_tls_wrap:743:12)

How do we regenerate the SLL cert? Want to pair on this?

@zepumph
Copy link
Member Author

zepumph commented Jun 27, 2023

The alternative I think would be to reach out to OIT about the general DNS issue, which I believe we both barely understand, and see if we get traction on that. Do you have a preference?

@mattpen
Copy link
Contributor

mattpen commented Jun 27, 2023 •
edited
Loading

I changed the default :80 block for the nginx.conf file so it doesn't do SSL escalation:

    server {
        listen 80;
        listen [::]:80;
        server_name sparky.colorado.edu;

        include /etc/nginx/default.d/*.conf;
        #return 301 https://sparky.colorado.edu$request_uri;
    }

I confirmed that this is no longer enforcing redirects from http://128.138.93.172 to https://sparky.colorado.edu.

// BEFORE
$ curl -I 'http://128.138.93.172/continuous-testing/ct-snapshots/1687902130737/build-a-fraction/build-a-fraction_en.html?continuousTest=%7B%22test%22%3A%5B%22build-a-fraction%22%2C%22xss-fuzz%22%5D%2C%22snapshotName%22%3A%22snapshot-1687902130737%22%2C%22timestamp%22%3A1687902383452%7D&brand=phet&ea&fuzz&stringTest=xss'
HTTP/1.1 301 Moved Permanently
Server: nginx/1.22.1
Date: Tue, 27 Jun 2023 22:05:11 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://sparky.colorado.edu/continuous-testing/ct-snapshots/1687902130737/build-a-fraction/build-a-fraction_en.html?continuousTest=%7B%22test%22%3A%5B%22build-a-fraction%22%2C%22xss-fuzz%22%5D%2C%22snapshotName%22%3A%22snapshot-1687902130737%22%2C%22timestamp%22%3A1687902383452%7D&brand=phet&ea&fuzz&stringTest=xss

// AFTER
$ curl -I 'http://128.138.93.172/continuous-testing/ct-snapshots/1687902130737/build-a-fraction/build-a-fraction_en.html?continuousTest=%7B%22test%22%3A%5B%22build-a-fraction%22%2C%22xss-fuzz%22%5D%2C%22snapshotName%22%3A%22snapshot-1687902130737%22%2C%22timestamp%22%3A1687902383452%7D&brand=phet&ea&fuzz&stringTest=xss'
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Tue, 27 Jun 2023 22:07:30 GMT
Content-Type: text/html
Content-Length: 4896
Last-Modified: Tue, 27 Jun 2023 21:42:28 GMT
Connection: keep-alive
ETag: "649b57c4-1320"
Accept-Ranges: bytes

@zepumph -- I hope this might be helpful. If it's not and you'd like to revert, the previous version of the conf is saved in /etc/nginx/nginx.conf.bk_06272023.

I don't understand the comment about http->https redirects for phet-io.colorado.edu. Are you getting the net err failed problems for that domain too? If so, this is likely a browser issue and not a webserver or DNS server problem.

If we need to add the ip as an alt name to the SSL cert for sparky, we can request a new cert from OIT then install it. My notes that briefly describe how to do this are here: https://github.com/phetsims/website#updating-an-ssl-cert . I'm not sure if these will be useful to people who are not Matt Pennington, OIT's instructions for requesting certs are here: https://oit.colorado.edu/services/web-content-applications/ssl-certificates and Nginx instructions for installing certs are here: http://nginx.org/en/docs/http/configuring_https_servers.html. But I'd be happy to update the cert.

@zepumph zepumph self-assigned this Jun 28, 2023
zepumph added a commit that referenced this issue Jun 28, 2023
@zepumph
use http to work around alt ssl certs for the static ip, #185
5e10a48
@zepumph
Copy link
Member Author

zepumph commented Jun 28, 2023

After discussing with @mattpen:

  1. We would like to use the http:static_ip address on bayes to see if that works around the original problem. (@zepumph)
  2. We would like to update the certs to allow the static IP as an alternate: (from CT: net::ERR_FAILED in phet-io-wrappers-tests #185 (comment)). @mattpen will take the lead.

Thanks @mattpen for working with me so much on this.

zepumph added a commit that referenced this issue Jun 28, 2023
@zepumph
use http to work around alt ssl certs for the static ip, #185
6ec3492
@mattpen
Copy link
Contributor

mattpen commented Jun 28, 2023

New SSL cert with the IP included as an SAN (Subject Alternative Name) has been requested with OIT.

@zepumph
Copy link
Member Author

zepumph commented Jun 28, 2023

#185 (comment)
I am blocked by http because for testing, I need to be able to postMessage to https://phet-io.colorado.edu/

In https://github.com/phetsims/phet-io/issues/1944#issuecomment-1612160549 this was solved by
by f838cc4. Sorry to relate that piece of the puzzle over here. I believe that I am fully unblocked by using http, and I also still think we should continue investigating where and how the err_faileds are occurring so that this doesn't bite us worse at some point in the future.

@zepumph
Copy link
Member Author

zepumph commented Jun 29, 2023

Let me know when the static IP is ready for testing on https.

@zepumph zepumph removed their assignment Jun 29, 2023
@mattpen
Copy link
Contributor

mattpen commented Jun 30, 2023

Looks like IP Addresses are not eligible to add as SANs in the Sectigo tool provided by CU. So we won't be able to test https://{{ip address}}, unless we self-sign a cert which is likely to raise additional errors.

@zepumph
Copy link
Member Author

zepumph commented Jul 18, 2024

I believe that once we have upgraded bayes over in https://github.com/phetsims/special-ops/issues/242 this issue is worth another look. It would be nice to have bayes clients for CT back in here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattpen mattpen

Labels
status:on-hold
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zepumph @mattpen