Make Ray work on 32 bit Linux

[atumanov]

We should make sure Ray reliably builds and works on 32-bit Linux, specifically 32-bit Ubuntu LTS. We should probably push 32-bit binary releases as well. Currently, it's possible to segfault the local scheduler by doing the following on 32-bit Linux:

import ray
ray.init()
@ray.remote
def f():
  return 1

oids = [f.remote() for _ in range(100)]

This came up in the context of user-reported error in #1008 and is addressed in #1126.
Additional problems currently observed on 32bit Ubuntu LTS include unit test failure in testRegisterClass :

======================================================================
ERROR: testRegisterClass (__main__.APITest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/runtest.py", line 379, in testRegisterClass
    new_regex = ray.get(f.remote(regex))
  File "/home/atumanov/ray/python/ray/worker.py", line 2342, in func_call
    objectids = _submit_task(function_id, args)
  File "/home/atumanov/ray/python/ray/worker.py", line 2194, in _submit_task
    return worker.submit_task(function_id, args)
  File "/home/atumanov/ray/python/ray/worker.py", line 482, in submit_task
    args_for_local_scheduler.append(put(arg))
  File "/home/atumanov/ray/python/ray/worker.py", line 2108, in put
    worker.put_object(object_id, value)
  File "/home/atumanov/ray/python/ray/worker.py", line 339, in put_object
    self.store_and_register(object_id, value)
  File "/home/atumanov/ray/python/ray/worker.py", line 296, in store_and_register
    _register_class(type(e.example_object))
  File "/home/atumanov/ray/python/ray/worker.py", line 1953, in _register_class
    class_id = hashlib.sha1(pickle.dumps(cls)).digest()
  File "/home/atumanov/miniconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 900, in dumps
    cp.dump(obj)
  File "/home/atumanov/miniconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 234, in dump
    return Pickler.dump(self, obj)
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/atumanov/miniconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 621, in save_global
    self.save_dynamic_class(obj)
  File "/home/atumanov/miniconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 459, in save_dynamic_class
    save(clsdict)
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 687, in _batch_setitems
    save(v)
  File "/home/atumanov/miniconda2/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
TypeError: can't pickle member_descriptor objects

Serialization tests are also failing, possibly due to the difference between 32 and 64 bit platforms:

======================================================================
FAIL: testPassingArgumentsByValue (__main__.SerializationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/runtest.py", line 249, in testPassingArgumentsByValue
    assert_equal(obj, ray.get(f.remote(obj)))
  File "test/runtest.py", line 71, in assert_equal
    obj1, obj2)
AssertionError: Objects 4611686018427387904 and 0 are different.

[xhochy]

Note that Arrow currently neither tests nor supports 64bit, so you probably should test first if Arrow works on 32bit systems.

[atumanov]

that's good to know, thanks @xhochy . So far, though, the plasma component worked for some simple puts and gets. I went ahead and ran Arrow cpp unit tests. Unit tests are passing:

$ make unittest
Test project /home/atumanov/arrow/cpp/debug
      Start  1: allocator-test
 1/23 Test  #1: allocator-test ...................   Passed    0.05 sec
      Start  2: array-test
 2/23 Test  #2: array-test .......................   Passed    1.01 sec
      Start  3: buffer-test
 3/23 Test  #3: buffer-test ......................   Passed    0.05 sec
      Start  4: memory_pool-test
 4/23 Test  #4: memory_pool-test .................   Passed    0.04 sec
      Start  5: pretty_print-test
 5/23 Test  #5: pretty_print-test ................   Passed    0.04 sec
      Start  6: status-test
 6/23 Test  #6: status-test ......................   Passed    0.04 sec
      Start  7: type-test
 7/23 Test  #7: type-test ........................   Passed    0.04 sec
      Start  8: table-test
 8/23 Test  #8: table-test .......................   Passed    0.04 sec
      Start  9: tensor-test
 9/23 Test  #9: tensor-test ......................   Passed    0.04 sec
      Start 12: compute-test
10/23 Test #12: compute-test .....................   Passed    0.04 sec
      Start 13: feather-test
11/23 Test #13: feather-test .....................   Passed    0.04 sec
      Start 14: ipc-read-write-test
12/23 Test #14: ipc-read-write-test ..............   Passed    1.41 sec
      Start 15: ipc-json-test
13/23 Test #15: ipc-json-test ....................   Passed    0.08 sec
      Start 16: json-integration-test
14/23 Test #16: json-integration-test ............   Passed    0.04 sec
      Start 18: io-file-test
15/23 Test #18: io-file-test .....................   Passed    0.08 sec
      Start 19: io-hdfs-test
16/23 Test #19: io-hdfs-test .....................   Passed    0.04 sec
      Start 20: io-memory-test
17/23 Test #20: io-memory-test ...................   Passed    0.57 sec
      Start 22: bit-util-test
18/23 Test #22: bit-util-test ....................   Passed    0.04 sec
      Start 23: compression-test
19/23 Test #23: compression-test .................   Passed    0.07 sec
      Start 24: decimal-test
20/23 Test #24: decimal-test .....................   Passed    0.04 sec
      Start 25: key-value-metadata-test
21/23 Test #25: key-value-metadata-test ..........   Passed    0.04 sec
      Start 26: rle-encoding-test
22/23 Test #26: rle-encoding-test ................   Passed    0.14 sec
      Start 27: stl-util-test
23/23 Test #27: stl-util-test ....................   Passed    0.04 sec

100% tests passed, 0 tests failed out of 23

$ uname -a
Linux 32bitubu1604 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:54:58 UTC 2017 i686 i686 i686 GNU/Linux

[atumanov]

Benchmarks are also passing on i686:

~/arrow/cpp/debug$ make ; ctest
Test project /home/atumanov/arrow/cpp/debug
      Start  1: allocator-test
 1/27 Test  #1: allocator-test ...................   Passed    0.04 sec
      Start  2: array-test
 2/27 Test  #2: array-test .......................   Passed    0.32 sec
      Start  3: buffer-test
 3/27 Test  #3: buffer-test ......................   Passed    0.04 sec
      Start  4: memory_pool-test
 4/27 Test  #4: memory_pool-test .................   Passed    0.04 sec
      Start  5: pretty_print-test
 5/27 Test  #5: pretty_print-test ................   Passed    0.04 sec
      Start  6: status-test
 6/27 Test  #6: status-test ......................   Passed    0.04 sec
      Start  7: type-test
 7/27 Test  #7: type-test ........................   Passed    0.04 sec
      Start  8: table-test
 8/27 Test  #8: table-test .......................   Passed    0.04 sec
      Start  9: tensor-test
 9/27 Test  #9: tensor-test ......................   Passed    0.04 sec
      Start 10: builder-benchmark
10/27 Test #10: builder-benchmark ................   Passed   19.81 sec
      Start 11: column-benchmark
11/27 Test #11: column-benchmark .................   Passed    7.45 sec
      Start 12: compute-test
12/27 Test #12: compute-test .....................   Passed    0.04 sec
      Start 13: feather-test
13/27 Test #13: feather-test .....................   Passed    0.04 sec
      Start 14: ipc-read-write-test
14/27 Test #14: ipc-read-write-test ..............   Passed    1.01 sec
      Start 15: ipc-json-test
15/27 Test #15: ipc-json-test ....................   Passed    0.12 sec
      Start 16: json-integration-test
16/27 Test #16: json-integration-test ............   Passed    0.04 sec
      Start 17: ipc-read-write-benchmark
17/27 Test #17: ipc-read-write-benchmark .........   Passed   30.35 sec
      Start 18: io-file-test
18/27 Test #18: io-file-test .....................   Passed    0.08 sec
      Start 19: io-hdfs-test
19/27 Test #19: io-hdfs-test .....................   Passed    0.04 sec
      Start 20: io-memory-test
20/27 Test #20: io-memory-test ...................   Passed    0.58 sec
      Start 21: io-memory-benchmark
21/27 Test #21: io-memory-benchmark ..............   Passed  263.50 sec
      Start 22: bit-util-test
22/27 Test #22: bit-util-test ....................   Passed    0.04 sec
      Start 23: compression-test
23/27 Test #23: compression-test .................   Passed    0.08 sec
      Start 24: decimal-test
24/27 Test #24: decimal-test .....................   Passed    0.04 sec
      Start 25: key-value-metadata-test
25/27 Test #25: key-value-metadata-test ..........   Passed    0.04 sec
      Start 26: rle-encoding-test
26/27 Test #26: rle-encoding-test ................   Passed    0.15 sec
      Start 27: stl-util-test
27/27 Test #27: stl-util-test ....................   Passed    0.04 sec

100% tests passed, 0 tests failed out of 27

Label Time Summary:
benchmark    = 321.11 sec (4 tests)
unittest     =   2.95 sec (23 tests)