Update SpiderMonkey version

[janl]

Follow up for: #1334

Very WIP branch here: https://github.com/apache/couchdb/tree/sm52

The plan roughly is to bring our SM dependencies to a new release, which means rewriting a bunch of our C/C++ goop that binds SM to our CLI tool.

The newer version will not accept CouchDB-style function declaration as-is, but there are two options to deal with this:

1. the Chakra work included an auto-AST transform thinger to adjust things to we maintain backwards compatibility.
2. we bite the BC bullet, because the impact is mostly theoretical.

Benefits: faster js, modern JS language features, fewer packaging woes.

[denib]

@jiangphcn - do you have an update here? Tnx!

[jiangphcn]

thanks for checking @denib

I am working on this task using #2170.

Below is current status:

• codes of couch_js are updated to use SpiderMonkey 60
• couch_js can be compiled locally in mac environment
• the javascript for main.js is updated and is able to be compiled during runtime
• new couchjs can started by couchdb and communicated with couchdb to accept reset, addFun, mapDoc etc

There are some to-dos I am working on

• there is code tweak needed to complete the whole mrview action
• compile and test in linux and windows
• adding codes to workaround anonymous javascript function name
• test where multiple sandbox are working together.

Hope I can get more progress in coming weeks.

[jiangphcn]

I marked two of them "completed" because mrview action works now.

• there is code tweak needed to complete the whole mrview action
• compile and test in linux
• compile and test in windows
• adding codes to workaround anonymous javascript function name
• test where multiple sandbox are working together

[jiangphcn]

There are more progress on this task

• compiled and tested in linux environment and prove that function is working
• most of make javascript test suites passed
• test related to sandbox passed
• the latest commit is 349a30b

To-do:

• fix collation_test_ eunit test
• fix the left javascript test case
• test in windows
• decide how to deploy with new spidermonkey runtime.

cc @denib @janl @davisp @kocolosk

[jiangphcn]

The recent update:

• fixed collation_test_ eunit test failure
• addressed utf8 related javascript test failures
• modify .travis.yml to accelerate CI elapsed time by installing spidermonkey dev instead of building on-the-fly
• other changes to make javascript test happy

To-do or ongoing:

• change for windows platform
• in parallel, send review request tomorrow
• decide how to deploy with new spidermonkey runtime.
  cc @denib @janl @davisp @kocolosk

[jiangphcn]

For windows-related stuff, finally made progress to build spidermonkey 60 using visual studio 2017. Next thing is to build changed CouchDB codes and test them.

It took me about 1 week to setup environment and build spidermonkey 60 for windows.

• First, I follow the guide in https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Build_Documentation

cd js/src
autoconf2.13

# This name should end with "_OPT.OBJ" to make the version control system ignore it.
mkdir build_OPT.OBJ
cd build_OPT.OBJ
../configure
mozmake

But I stuck during executing configure.

$ ../configure
Reexecuting in the virtualenv
checking for vcs source checkout... no
checking for a shell... C:/mozilla-build-3.2/msys/bin/sh.exe
checking for host system type... i686-pc-mingw32
checking for target system type... i686-pc-mingw32
checking for a shell... C:/mozilla-build-3.2/msys/bin/sh.exe
checking for host system type... i686-pc-mingw32
checking for target system type... i686-pc-mingw32
checking for android platform directory... no
checking for android sysroot directory... no
checking for android system directory... no
checking for the Android toolchain directory... not found
checking for android platform directory... no
checking for android system directory... no
checking for android sysroot directory... no
checking for the Android toolchain directory... not found
checking for vcs source checkout... no
checking whether cross compiling... no
traceback (most recent call last):
  File "../../../configure.py", line 130, in <module>
    sys.exit(main(sys.argv))
  File "../../../configure.py", line 29, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 409, in run
    self._value_for(option)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 477, in _value_for
    return self._value_for_option(obj)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 494, in _value_for_option
    not self._value_for(implied_option.when)):
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 199, in result
    return self._func(resolved_args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 198, in <genexpr>
    for d in self.dependencies)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 122, in result
    for d in self.dependencies]
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 122, in result
    for d in self.dependencies]
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 122, in result
    for d in self.dependencies]
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 122, in result
    for d in self.dependencies]
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 474, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 483, in _value_for_depends
    return obj.result(need_help_dependency)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\util.py", line 944, in method_call
    cache[args] = self.func(instance, *args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 123, in result
    return self._func(*resolved_args)
  File "c:\mozjs\mozjs-60.1.1pre3\python\mozbuild\mozbuild\configure\__init__.py", line 1003, in wrapped
    return new_func(*args, **kwargs)
  File "c:/mozjs/mozjs-60.1.1pre3/build/moz.configure/toolchain.configure", line 638, in vc_compiler_path
    all_versions = sorted(get_vc_paths(env.topsrcdir), key=itemgetter(0))
  File "c:/mozjs/mozjs-60.1.1pre3/build/moz.configure/toolchain.configure", line 585, in get_vc_paths
    for install in vswhere(['-legacy', '-version', '[14.0,15.0)']):
  File "c:/mozjs/mozjs-60.1.1pre3/build/moz.configure/toolchain.configure", line 581, in vswhere
    ] + args).decode(encoding, 'replace'))
  File "c:\mozilla-build-3.2\python\Lib\subprocess.py", line 216, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "c:\mozilla-build-3.2\python\Lib\subprocess.py", line 394, in __init__
    errread, errwrite)
  File "c:\mozilla-build-3.2\python\Lib\subprocess.py", line 644, in _execute_child
    startupinfo)
WindowsError: [Error 5] Access is denied

Knowing that it is related to file access, but error message doesn't tell details about problem location. Took days to investigate build system of spidermonkey and use half-interval search and finally identify that it is related to vswhere.exe in spidermonkey source code package. And it failed to execute.

Administrator@ph-win10 /c/mozjs/mozjs-60.1.1pre3/build/win32
$ vswhere.exe
bash: ./vswhere.exe: Bad file number

After using vswhere.exe provided from visual studio 2017, the configure can continue.

And then mozmake can build spidermonkey source.

For now, the jsapi-tests.exe passed without error.

$ ./js/src/build_OPT.OBJ/dist/bin/jsapi-tests.exe
testXDR_sourceMap
TEST-PASS | testXDR_sourceMap | ok
testXDR_source
TEST-PASS | testXDR_source | ok
...
testAssemblerBuffer_BufferOffset
TEST-PASS | testAssemblerBuffer_BufferOffset | ok

Passed: ran 281 tests.

cc @denib @janl @davisp @kocolosk

[jiangphcn]

I want to give some update before thanksgiving day about this ticket. Recently, I used a lot of time to work for window platform. Every time I am almost about to send to email to community to get support, I got some progress. I plan to still take two days to try the last approach, and then decide how to proceed for windows platform.

• a. using guide in https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Build_Documentation, spidermonkey 60 can be built, but the default is 32 bit. 32 bit library is not for 64bit couchjs. I will get link issue.
• b. same as approach a, I used ../configure --host=x86_64-pc-mingw32 --target=x86_64-pc-mingw32 --enable-debug to setup 64bit environment. There is error when comping js/src/Unified_cpp_js_src19.cpp.

Unified_cpp_js_src19.cpp
C:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared\stralign.h(120): warning C4090: 'argument': different '__unaligned' qualifiers
c:/mozjs/mozjs-60.1.1pre3/js/src/jit/ProcessExecutableMemory.cpp(65): fatal error C1189: #error:  "Unsupported architecture"
mozmake[3]: *** [c:/mozjs/mozjs-60.1.1pre3/config/rules.mk:1049: Unified_cpp_js_src19.obj] Error 2

• c. later on, I found one repository https://github.com/mozilla/gecko-dev. I made it work and compile mozjs-72a1.lib. When compiling couchjs using this library, we can pass compile, and there is 64bit-32bit issue when linking them together. Because there are interface change, it is not surprise that I got some undefined symbols error.
• d. this week, I found another repository https://github.com/servo/mozjs. I made it almost work and my thought is that it can build spidermonkey version6x with 64bit. The error I got in case b was resolved. Most of files can be compiled today. I am working on Cannot open include file corecrt.h error.

I found that configure --disable-jemalloc --disable-js-shell --disable-tests --disable-shared-js --build-backends=RecursiveMake --enable-posix-nspr-emulati on --with-visual-studio-version=2017 --enable-debug --disable-optimize --enable-gczeal --target=x86_64-pc-mingw32 --host=x86_64-pc-mingw32 was used in approach d, this can be used for approach b.

So let me check whether I can get around using approach d and enhanced approach b in coming days.
cc @denib @janl @davisp @kocolosk @wohali

[jiangphcn]

I built out spidermonkey 60 64bit for windows now. It is using enhanced approach b. The problem mentioned in above approach d was resolved. However, it will not build mozjs-60.lib.

I wrote up one guide in PR apache/couchdb-glazier#11

@denib @janl @davisp @kocolosk @wohali

[jiangphcn]

PR #2345 was merged.