Add StellarTomlResolver timeout option #166

robertDurst · 2018-05-24T18:32:30Z

Closes: #158

Motivation

Say you are a service that wants to query all asset issuers and are using StellarSdk.StellarTomlResolver.resolve(uri.host()) to get the TOML information. Unfortunately this sometimes gets caught up and can lag for long periods of time.

An example url that causes this lag: xggtoken.com

Solution

Add an optional timeout field to the opts param in StellarSdk.StellarTomlResolver.resolve.

Using the timeout config in the axios fetch, we can let sdk users specify a timeout length (in ms) to avoid these long TOML resolve lags.

In case users don't want to use the timeout config, it is set to 0 by default.

Example usage:

await StellarTomlResolver.resolve("xggtoken.com", { timeout: 1000 });

Note

I thought about writing a test for this, however to future proof the test suite, I did not want to test against the example I have above in case this domain changes or is taken down. Is this reasoning correct?

bartekn · 2018-05-25T12:09:18Z

It looks good however it won't work if stellar.toml file is requested in federation context (StellarTomlResover is called in FederationServer). To solve this I think we could add timeout to Config class and use it like allowHttp.

I thought about writing a test for this, however to future proof the test suite, I did not want to test against the example I have above in case this domain changes or is taken down. Is this reasoning correct?

You are correct. You can start a simple HTTP server like here with a setTimeout that sends a response after a longer time period.

coveralls · 2018-05-25T21:35:01Z

Coverage increased (+0.2%) to 93.277% when pulling 236c473 on robertDurst:toml-timeout into 3a70aed on stellar:master.

robertDurst · 2018-05-27T12:45:52Z

Made the following changes:

Added timeout to Config
Added tow unit tests, one where I specify timeout in Config, other where I specific in StellarTomlResolver opts param

robertDurst · 2018-05-27T12:50:11Z

I also see that this allowHttp opts param is set in:

server
federation server

However, unless I am mistaken, it looks like the opts is function specific and since you can now set timeout on the Config I don't think it is necessary to allow for opts.timeout.

…TomlResolver tests

bartekn · 2018-05-28T15:35:41Z

src/config.js

+   * Returns the value of `timeout` flag.
+   * @static
+   */
+  static hasTimeout() {


Should be getTimeout.

bartekn · 2018-05-28T15:36:58Z

test/unit/stellar_toml_resolver_test.js

+    it("rejects after given timeout when global Config.timeout flag is set", function (done) {
+      StellarSdk.Config.setTimeout(1000);
+
+      StellarSdk.StellarTomlResolver.resolve('acme.com')


Tests shouldn't make network connections (other to local test server).

bartekn · 2018-05-28T15:55:28Z

However, unless I am mistaken, it looks like the opts is function specific and since you can now set timeout on the Config I don't think it is necessary to allow for opts.timeout.

I think it's good to have a global and local settings. For example you may want to change the timeout value per request.

We should also add timeout to Server and FederationServer.

robertDurst · 2018-05-28T22:57:50Z

Fixed the above:

hasTimeout --> getTimeout
Reworked StellarTomlResolver test so that it used a local server.

Also looked at where the opts.allowHttp was used.

Besides stellar_toml_resolver.js it is used in:

server.js:
- in the constructor to ensure no connection to insecure horizon server.
federation_server.js:
- in the constructor to ensure no connection to insecure horizon server.
- for querying StellarTomlResolver via createForDomain

So I added it to federation_server.js for createForDomain (and resolve which calls createForDomain) and also added a test to federation_server.js.

…d up a few things

bartekn · 2018-05-29T18:30:14Z

I all looks good, but I'd change one last thing: we should probably add timeout option everywhere else because the name doesn't suggest it's for StellarTomlResolver only but also I suspect this will be needed in federation or horizon calls as well. So this should be added in:

FederationServer.resolveAddress: both opts param and Config checks.
Server.constructor: both opts param and Config checks and we should also add it as a instance variable on CallBuilders to apply this in _sendNormalRequest when making calls.

We shouldn't change Server.submitTransaction timeout.

robertDurst · 2018-05-30T21:41:41Z

Following my last commit that addressed your feedback, I have a few questions:

Should I include options (such as allowHttp) even in places where it won't be used (such as in some of the federation_server methods)?
For resolveAddress, resolveAccountId, resolveTransactionId in the federation_server is it best to pass in opts like I do here?
For call_builder should I pass in timeout as an opts param like I do or should I just pass in timeout?

… add timeout to call_builder

bartekn

Should I include options (such as allowHttp) even in places where it won't be used (such as in some of the federation_server methods)?

I added some inline comments. allowHttp is class setting checked in constructor (for attempt to use http server if it's not allowed the object won't be created). So I think we don't need other options than timeout for class methods.

For resolveAddress, resolveAccountId, resolveTransactionId in the federation_server is it best to pass in opts like I do here?

I think we should only allow setting timeout for static methods. For normal methods the value should be taken from this.timeout set in constructor.

For call_builder should I pass in timeout as an opts param like I do or should I just pass in timeout?

Let's leave opts, will be helpful for other options in a future.

Can we also add some tests for a) static methods, b) normal methods on FederationServer, c) methods using CallBuilders on 'Server'.

bartekn · 2018-06-04T10:58:29Z

src/call_builder.js

 * @class CallBuilder
 */
 export class CallBuilder {
-  constructor(serverUrl) {
+  constructor(serverUrl, opts = {}) {
+    this.timeout = 0;


Default here should be a global setting so Config.getTimeout().

bartekn · 2018-06-04T11:00:31Z

src/config.js

@@ -32,6 +35,16 @@ class Config {
    config.allowHttp = value;
  }

+  /**
+   * Sets `timeout` flag globally. When set to anything besides 0, StellarTOMLResolver will timeout after specified time (ms).


Update the comment. It's not StellarTOMLResolver only anymore.

bartekn · 2018-06-04T11:02:04Z

src/federation_server.js

@@ -20,6 +20,7 @@ export const FEDERATION_RESPONSE_MAX_SIZE = 100 * 1024;
 * @param {string} domain Domain this server represents
 * @param {object} [opts]
 * @param {boolean} [opts.allowHttp] - Allow connecting to http servers, default: `false`. This must be set to false in production deployments! You can also use {@link Config} class to set this globally.
+ * @param {number} [opts.timeout] - Allow a timeout, default: 0. In ms.
 */
 export class FederationServer {
  constructor(serverURL, domain, opts = {}) {


Static method instantiate this class but opts.timeout is not persisted anywhere.

bartekn · 2018-06-04T11:04:13Z

src/federation_server.js

+
+    if (typeof opts.allowHttp !== 'undefined') {
+        allowHttp = opts.allowHttp;
+    }


No need for allowHttp processing here, it's been checked in constructor.

bartekn · 2018-06-04T11:11:08Z

src/federation_server.js

  }

  /**
   * Returns a Promise that resolves to federation record if the sender of the transaction was found for a given transaction ID.
   * @see <a href="https://www.stellar.org/developers/learn/concepts/federation.html" target="_blank">Federation doc</a>
   * @param {string} transactionId Transaction ID (ex. `3389e9f0f1a65f19736cacf544c2e825313e8447f569233bb8db39aa607c8889`)
+   * @param {object} [opts]
+   * @param {boolean} [opts.allowHttp] - Allow connecting to http servers, default: `false`. This must be set to false in production deployments!


allowHttp is not needed here as it's checked in constructor per server.

bartekn · 2018-06-04T11:16:45Z

src/server.js

@@ -32,6 +32,7 @@ export const SUBMIT_TRANSACTION_TIMEOUT = 60*1000;
 * @param {string} serverURL Horizon Server URL (ex. `https://horizon-testnet.stellar.org`).
 * @param {object} [opts]
 * @param {boolean} [opts.allowHttp] - Allow connecting to http servers, default: `false`. This must be set to false in production deployments! You can also use {@link Config} class to set this globally.
+ * @param {number} [opts.timeout] - Allow a timeout, default: 0. In ms.


Not passed to CallBuilder anywhere.

robertDurst · 2018-06-04T23:53:03Z

@bartekn started adding timeout to callbuilder and realized callbuilder was the parent of all the different callbuilders like TradesCallbuilder and that furthermore TradesCallbuilder was only called from server...

Since that code started to get a bit long and since this PR is focused on solving #158 which only deals with Federation Server and Stellar Toml Resolver that are seperate from server and callbuilder, I decided to cut out the extra code for callbuilder.

So what I now have is just the code for Federation Server and StellarTomlResolver that directly addresses #158.

Is this reasoning sound? I think adding timeout to callbuilder may be worth a seperate PR.

bartekn

OK, I agree to limit this PR to StellarTomlResolver and FederationServer. There are still a few issues to fix.

bartekn · 2018-06-05T09:45:19Z

src/federation_server.js

@@ -161,7 +163,9 @@ export class FederationServer {
  }

  _sendRequest(url) {
-    return axios.get(url.toString(), {maxContentLength: FEDERATION_RESPONSE_MAX_SIZE})
+    let timeout = this.timeout | Config.getTimeout();


this.timeout is not set anywhere.

In general it should work like this:

In constructor set this.timeout to opts.timeout or Config.getTimeout() (if former not set).

static method create FederationServer instance internally so we should just pass opts to class constructor.

Finally _sendRequest should just use this.timeout. No need to check Config again.

Also, should be ||. | is bitwise OR.

bartekn · 2018-06-05T09:46:41Z

test/unit/federation_server_test.js

+
+  describe('FederationServer times out when response lags and timeout set', function () {
+    it("resolveAddress times out", function (done) {
+      StellarSdk.Config.setTimeout(100);


We should reset it to default value so it doesn't influence other tests. Maybe in afterEach? ~~Would be great to have Config.setDefaults() or something like this, in case default values change in a future.~~ We already have Config.setDefault().

bartekn · 2018-06-05T09:49:15Z

test/unit/federation_server_test.js

+        setTimeout(() => {}, 10000);
+      }).listen(4444, () => {
+        StellarSdk.FederationServer
+          .resolve('bob*localhost:4444', {allowHttp: true, timeout: 100})


This test shouldn't work because opts is not passed to FederationServer.constructor (see other comment). It only works because StellarSdk.Config.setTimeout(100); is not reset after previous tests.

bartekn · 2018-06-05T09:52:33Z

test/unit/federation_server_test.js

+      let tempServer = http.createServer((req, res) => {
+        setTimeout(() => {}, 10000);
+      }).listen(4444, () => {
+        new StellarSdk.FederationServer('http://localhost:4444/federation', 'stellar.org', {allowHttp: true})


The last missing test case is creating a new FederationServer with opts.timeout set (without calling Config.setTimeout).

I think you can create a loop that will test all the methods (resolveTransactionId, resolveAccountId, ...) with both cases: Config in the first iteration and opts in the second. Without it we will have a lot of code duplication here.

bartekn

Other than my last 2 comments looks great!

bartekn · 2018-06-05T13:45:16Z

test/unit/federation_server_test.js

+      // Unable to create temp server in a browser
+      if (typeof window != 'undefined') {
+        return done();
+      }


This has to be checked inside its functions. done is not defined here. You can run gulp test:browser to test this locally. CI tests pass because we do not perform SauceLabs tests in PRs at it could leak our keys.

https://docs.travis-ci.com/user/environment-variables/#Defining-encrypted-variables-in-.travis.yml

Encrypted environment variables are not available to pull requests from forks due to the security risk of exposing such information to unknown code.

ah, ok had to install firefox for this one

bartekn · 2018-06-05T13:46:36Z

test/unit/stellar_toml_resolver_test.js

@@ -106,5 +106,42 @@ FEDERATION_SERVER="https://api.stellar.org/federation"
          .then(() => tempServer.close());
      });
    });
+
+    it("rejects after given timeout when global Config.timeout flag is set", function (done) {
+      StellarSdk.Config.setTimeout(1000);


Please add Config.setDefault(); to afterEach here too.

Add federation timeout

717772a

robertDurst requested a review from bartekn May 24, 2018 18:32

Added timeout to global config and added two timeout tests to Stellar…

96acbe2

…TomlResolver tests

bartekn reviewed May 28, 2018

View reviewed changes

Added timeout opts to federation, added test to federation and cleane…

6aa24d6

…d up a few things

Add timeout opt to Server, add opts to federation_server methods, and…

bb82eed

… add timeout to call_builder

bartekn reviewed Jun 4, 2018

View reviewed changes

bartekn mentioned this pull request Jun 4, 2018

server.trades(...).forAccount is not a function #172

Closed

Add more federation server tests and remove non-stellartomlresolver code

d8529c8

Cleanup extra code in tests

acf091c

bartekn suggested changes Jun 5, 2018

View reviewed changes

Correct Federation server error and federation server tests

2e3c769

bartekn reviewed Jun 5, 2018

View reviewed changes

Final tests fix

236c473

bartekn approved these changes Jun 5, 2018

View reviewed changes

bartekn merged commit d43fbd8 into stellar:master Jun 5, 2018

tomerweller mentioned this pull request Feb 19, 2019

provide utility functions to dynamically retrieve fees #233

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add StellarTomlResolver timeout option #166

Add StellarTomlResolver timeout option #166

robertDurst commented May 24, 2018

bartekn commented May 25, 2018

coveralls commented May 25, 2018 •

edited

Loading

robertDurst commented May 27, 2018

robertDurst commented May 27, 2018

bartekn May 28, 2018

bartekn May 28, 2018

bartekn commented May 28, 2018

robertDurst commented May 28, 2018 •

edited

Loading

bartekn commented May 29, 2018

robertDurst commented May 30, 2018

bartekn left a comment

bartekn Jun 4, 2018

bartekn Jun 4, 2018

bartekn Jun 4, 2018

bartekn Jun 4, 2018

bartekn Jun 4, 2018

bartekn Jun 4, 2018

robertDurst commented Jun 4, 2018

bartekn left a comment

bartekn Jun 5, 2018

bartekn Jun 5, 2018 •

edited

Loading

bartekn Jun 5, 2018

bartekn Jun 5, 2018

bartekn left a comment

bartekn Jun 5, 2018 •

edited

Loading

robertDurst Jun 5, 2018

bartekn Jun 5, 2018

Add StellarTomlResolver timeout option #166

Add StellarTomlResolver timeout option #166

Conversation

robertDurst commented May 24, 2018

Motivation

Solution

Example usage:

Note

bartekn commented May 25, 2018

coveralls commented May 25, 2018 • edited Loading

robertDurst commented May 27, 2018

robertDurst commented May 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bartekn commented May 28, 2018

robertDurst commented May 28, 2018 • edited Loading

bartekn commented May 29, 2018

robertDurst commented May 30, 2018

bartekn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertDurst commented Jun 4, 2018

bartekn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bartekn Jun 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bartekn left a comment

Choose a reason for hiding this comment

bartekn Jun 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 25, 2018 •

edited

Loading

robertDurst commented May 28, 2018 •

edited

Loading

bartekn Jun 5, 2018 •

edited

Loading

bartekn Jun 5, 2018 •

edited

Loading