Automated tests with Saucelabs?

[nolanlawson]

@brodybits you mentioned you were looking into adding better automated tests for this repo. What's the current plan? I see a .travis.yml config file; is there a Travis page somewhere as well?

A bit of background: with PouchDB, I've been trying to get Android/iOS Appium tests running in Travis for a long time, with basically zero luck (see pouchdb/pouchdb#3481; there are also many branches I abandoned, the closest being pouchdb/pouchdb#3751).

However, up to now I haven't tried to use Saucelabs, which incidentally was a very helpful service when we wanted to test browsers like Safari and IE. It seems it may be more stable than Travis, since Travis ran out of memory pretty fast when I tried to run a headless Android emulator.

Are you interested in setting up a Saucelabs account? Do you think that would be a good approach? The limit of the current approach seems to be that you're bound to Travis, and I believe you can only have one .travis.yml with one "language" style (e.g. either android or objective-c, but not both).

Thoughts? Also @daleharvey you may be interested in this conversation, since you were so adept at setting up the browser tests with Saucelabs. If we can figure out how to use Saucelabs with Appium, that also opens us up to actually start testing Android, and to start running the PouchDB test suite against the SQLite Plugin as well.

[nolanlawson]

Oh, I'm dumb; I should have just scrolled down the page. 😆 CircleCI looks neat; maybe we can use it for PouchDB!

[brodycj]

Are you interested in setting up a Saucelabs account? Do you think that
would be a good approach?

Absolutely! I am now planning to look into it when I return from vacation in Poland next week.

If you or @daleharvey want to integrating this project with Saucelabs that's great-just issue a pull request and I can integrate it when I return.

My idea, as a very strong suggestion, is to start with a very simple test such as the "ASCII string test" that you can find in simple-test.js and legacy.js.

[nolanlawson]

Hm, I also recall there were some tests I added last summer, which don't seem to be there anymore. :( They tested pretty important stuff, such as the format of binary/UTF-8 data. Would you be adverse if I tried to revive them? (Assuming they passed of course.)

[brodycj]

Yes but look very carefully since some tests have been moved and/or
massively modified. Notes: tests must pass on all platforms, including both
windows (universal) and wp8. (There are some tests with conditions to skip
platforms where they are not working).

Also I am not supporting Blob (for now). Planning to support blobs again
once I get some more immediate issues resolved.
On Apr 28, 2015 3:57 PM, "Nolan Lawson" [email protected] wrote:

Hm, I also recall there were some tests I added last summer, which don't
seem to be there anymore. :( They tested pretty important stuff, such as
the format of binary/UTF-8 data. Would you be adverse if I tried to revive
them? (Assuming they passed of course.)

—
Reply to this email directly or view it on GitHub
#251 (comment)
.

[nolanlawson]

Blob support would be neat, but yeah, it is a huge mess due to inconsistent browser implementations. When you do try to support it, might I recommend blob-util? 😃

[brodycj]

Blob support would be neat, but yeah, it is a huge mess due to inconsistent browser implementations.

Right-keeping blob support at (very) low priority, unless people start screaming for it.

When you do try to support it, might I recommend blob-util? 😃

Thanks-I took a quick look (from the airport) and will try to interface it, assuming I do add/fix the support for blob types.

What I think we want, which I suspect is perhaps covered by blob-util, is to give a user essentially one way to read and write data types such as array/binary/image/base64/etc. etc. to a resource such as a Blob, file, sqlite database, even PouchDB.

But this is digressing. Going "dark" in 10-20 minutes for sabbath.

[brodycj]

And digressing even more. Also, I am not sure if you looked at the Blob enhancements in PR #170, especially b62bdff which added a "SQLBlob" type to make the plugin work better with blogs. I had been thinking the "SQLBlob" type could/should become its own project, that could be useful both with this plugin and with normal Web SQL API. But if your blob-util could cover or supersede the functionality introduced by the "SQLBlob" it would be even better. I can take a look, (someday).

[brodycj]

From #247:

Hm, it gets to 49% and then just stops (Android 4.4 emulator + SQLite plugin).

Yes it is a very serious issue that (yours truly) managed to break the PouchDB integration. And we know this is by far not the only major/critical issue! In addition, there is also some other project work (not related to this plugin) that I really want to take care of in the next few weeks. While I cannot make any promises, an idea I thought of is to make a special version, just for the PouchDB integration, for which we will not add new features and all changes to be reviewed between us. For discussion.

[nolanlawson]

I think it would just add to the confusion to maintain two different versions of this repo.

PouchDB is not using the WebSQL API in any unusual way; we just happen to have a very thorough set of tests (over 1,000 now). This repo doesn't have anywhere near the same number of tests (I count 2 :)), which is why it's easy to accidentally introduce bugs.

Adding new features is fine, so long as existing functionality isn't broken. And the only way to be sure of that is to have automated tests.

We learned this the hard way in PouchDB, as we constantly introduced "fixes" for Safari that would break IE, or fixes for PhantomJS that would break Chrome, etc. In the hostile world of browser libraries, it's laughably easy to introduce bugs. The only thing that tamed the bugs was writing a rigorous test suite and running them against every browser we could think of in continuous integration.

So with that in mind, my next goal for this plugin is to figure out which PouchDB test(s) broke, and then rewrite it as a test in this repo. Now that we have continuous integration, it should just be a matter of finding the bug, writing a test to reproduce it, and fixing it. As long as the tests don't get removed, we can be certain the same bug won't rear its ugly head again. (I note with some dismay that the tests I added last summer have been removed. I will happily add them back in again and fix them. :))

Anyway, I'm now a co-owner of this repo (thanks @brodybits!), so I think I have some responsibility to do my part and get stuff back into a working state. :)

[nolanlawson]

And that being said, I did fork it for the time being: https://github.com/nolanlawson/sqlite-plugin-fork. I hope to remove the fork soon.

SQLBlob looks neat. :) I'll check it out, but it doesn't impact PouchDB since we just use binary strings.

[brodycj]

One thing: I do have continuous integration working as follows:

• for Android, running full test suite on Circle CI (takes over 90 minutes)
• for iOS, running a minor subset of the existing test suite on Travis CI (cannot run full test suite since Travis CI limits each run to 50 minutes)

For working with Saucelabs: I do fully support this level of CI testing, unfortunately cannot say when I will have a chance to look at this. I would like to treat this with reduced priority for now due to the existing backlog.