build boost with zlib support

[smanders]

we've been relying on a separate install of the zlib developement package (on linux, perhaps also on Solaris? - I don't remember, but most likely) to build boost iostreams

now that we build zlib as part of externpro, we should make an externpro project dependency between the two (boost on zlib) and have boost build against our zlib build

this would also enhance our windows build of boost, which doesn't have zlib support yet

[smanders]

NSS branch may have a need for our Windows build of boost to support zlib/gzip:

From: Keith Blonquist
Sent: Tuesday, August 18, 2015 4:43 PM
To: Scott M Anderson
Cc: Kenny Reese
Subject: gzip support in Boost in ExternPro

Scott,

Do you know if extern pro includes gzip support in the Boost library? When trying to use gzip functions in Boost I'm getting a LINK error (LNK1104: cannot open file libboost_zlib-vc120-mt-s-1_55.lib). This library file does not exist in the extern pro libraries.

Thanks,
Keith

[smanders]

Boost.Iostreams Installation
http://www.boost.org/doc/libs/1_57_0/libs/iostreams/doc/installation.html

How to build boost iostreams with gzip and bzip2 support on Windows
http://stackoverflow.com/questions/7282645/how-to-build-boost-iostreams-with-gzip-and-bzip2-support-on-windows

Building Zlib and Iostreams
http://boost.2283326.n4.nabble.com/Building-Zlib-and-Iostreams-td2695701.html
one of the last posts in the thread mentions setting BOOST_ZLIB_BINARY=xxx when building your project that uses boost iostreams

[smanders]

feature branch started

[smanders]

bzip2 source in git repo
https://github.com/LuaDist/bzip2
https://github.com/osrf/bzip2_cmake

bzip2 website
http://www.bzip.org/
current version is 1.0.6 released 2010.09.20

bzip2 downloads
http://www.bzip.org/downloads.html

[smanders]

http://www.boost.org/doc/libs/1_56_0/libs/iostreams/doc/classes/bzip2.html#examples
The following code decompresses data from a file and writes it to standard output.

#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/bzip2.hpp>

int main() 
{
    using namespace std;
    using namespace boost::iostreams;

    ifstream file("hello.bz2", ios_base::in | ios_base::binary);
    filtering_streambuf<input> in;
    in.push(bzip2_decompressor());
    in.push(file);
    boost::iostreams::copy(in, cout);
}

http://www.boost.org/doc/libs/1_50_0/libs/iostreams/doc/classes/zlib.html#examples
The following code decompresses data from a file and writes it to standard output.

#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/zlib.hpp>

int main() 
{
    using namespace std;

    ifstream file("hello.z", ios_base::in | ios_base::binary);
    filtering_streambuf<input> in;
    in.push(zlib_decompressor());
    in.push(file);
    boost::iostreams::copy(in, cout);
}

[smanders]

http://www.boost.org/doc/libs/1_42_0/libs/iostreams/doc/classes/gzip.html#examples
The following code decompresses data from a file and writes it to standard output.

#include <fstream>
#include <iostream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>

int main() 
{
    using namespace std;

    ifstream file("hello.gz", ios_base::in | ios_base::binary);
    filtering_streambuf<input> in;
    in.push(gzip_decompressor());
    in.push(file);
    boost::iostreams::copy(in, cout);
}

[smanders]

Combined example code that decompresses data based on file extension of input argument.

#include <fstream>
#include <iostream>
#include <boost/filesystem.hpp>
#include <boost/iostreams/filtering_streambuf.hpp>
#pragma warning(push)
// conversion from std::streamsize to int, possible loss of data
#pragma warning(disable : 4244)
#include <boost/iostreams/copy.hpp>
#pragma warning(pop)
#include <boost/iostreams/filter/bzip2.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/filter/zlib.hpp>

int main(int argc, char** argv)
{
  namespace bfs = boost::filesystem;
  namespace bio = boost::iostreams;
  bfs::path exepath = bfs::path(std::string(argv[0]));
  if (argc != 2)
  {
    std::cerr << "usage: " << exepath.filename().string() << " file.[bz2|gz|Z]" << std::endl;
    return 1;
  }
  try
  {
    bfs::path filepath = bfs::path(std::string(argv[1]));
    if (!bfs::exists(filepath))
    {
      std::cerr << filepath.string() << ": doesn't exist" << std::endl;
      return 1;
    }
    bio::filtering_streambuf<bio::input> in;
    std::string ext = filepath.extension().string();
    if (ext.compare(".bz2") == 0)
      in.push(bio::bzip2_decompressor());
    else if (ext.compare(".gz") == 0)
      in.push(bio::gzip_decompressor());
    else if (ext.compare(".Z") == 0)
      in.push(bio::zlib_decompressor());
    else
    {
      std::cerr << filepath.filename().string() << ": unsupported extension (must be .[bz2|gz|Z])" << std::endl;
      return 1;
    }
    std::ifstream file(argv[1], std::ios_base::in | std::ios_base::binary);
    in.push(file);
    bio::copy(in, std::cout);
  }
  //catch (const bio::bzip2_error& e)
  catch (const bio::zlib_error& e)
  {
    int err = e.error();
    if (err == bio::zlib::buf_error)
      std::cerr << "zlib buffer error" << std::endl;
    else if (err == bio::zlib::data_error)
      std::cerr << "zlib data error" << std::endl;
    else if (err == bio::zlib::mem_error)
      std::cerr << "zlib memory error" << std::endl;
    else if (err == bio::zlib::stream_error)
      std::cerr << "zlib stream error" << std::endl;
    else if (err == bio::zlib::version_error)
      std::cerr << "zlib version error" << std::endl;
    else
      std::cerr << "zlib unknown error" << std::endl;
    return 1;
  }
  catch (const std::exception& e)
  {
    std::cerr << e.what() << std::endl;
    return 1;
  }
  return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.3)
project(decompressor)
set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/cmake)
set(externpro_REV 16.01.1)
find_package(externpro REQUIRED)
include(${externpro_DIR}/share/cmake/flags.cmake)
set(Boost_LIBS filesystem system iostreams)
if(MSVC)
  xpGetExtern(externIncs externLibs PUBLIC boost)
  xpGetExtern(dontcare zlibBinary PRIVATE zlib)
  add_definitions(
    -DBOOST_ZLIB_BINARY=$<TARGET_FILE:${zlibBinary}>
    #-DBOOST_BZIP2_BINARY=
    )
else()
  xpGetExtern(externIncs externLibs PUBLIC boost zlib)
  find_package(BZip2 REQUIRED)
  list(APPEND externLibs ${BZIP2_LIBRARIES})
endif()
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
add_executable(${PROJECT_NAME} decompress.cpp)
target_link_libraries(${PROJECT_NAME} ${externLibs})
target_include_directories(${PROJECT_NAME} ${externIncs})

[smanders]

completed with pull to dev branch

[smanders]

with work merged to dev branch (the zlib and bzip2 dependencies are part of the boost use script now), the CMakeLists.txt for the sample app is now simply:

cmake_minimum_required(VERSION 3.7.2)
project(decompressor)
set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/cmake)
set(externpro_REV 17.05.1)
find_package(externpro REQUIRED)
include(${externpro_DIR}/share/cmake/flags.cmake)
set(Boost_LIBS filesystem system iostreams)
xpGetExtern(externIncs externLibs PUBLIC boost)
set_property(GLOBAL PROPERTY USE_FOLDERS ON)
add_executable(${PROJECT_NAME} decompress.cpp)
target_link_libraries(${PROJECT_NAME} ${externLibs})
target_include_directories(${PROJECT_NAME} ${externIncs})

[smanders]

Testing
I created a sample .bz2 and .gz file with bzip2 and gzip by writing a simple text message in a file named hello and hi, then run bzip2 hello to create hello.bz2, and gzip hi to create hi.gz.

Then I verified that bzcat hello.bz2 produced the same result as ./decompressor hello.bz2 (the message that was contained in hello), and zcat hi.gz produced the same result as ./decompressor hi.gz.

I also created a yo.Z file with compress -f yo. The message is decompressed with zcat yo.Z, but when I attempt to run ./decompressor yo.Z an exception is thrown:

smanders@bluepill:~/src/boostio/_bld$ ./decompressor yo.Z
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::iostreams::zlib_error> >'
  what():  zlib error
Aborted (core dumped)

When I put in the try/catch, specifically catching the boost::iostreams::zlib_error, the error is reported:

zlib data error

[smanders]

A similar exception is reported "Boost Iostreams zlib_error with Custom Source"
http://stackoverflow.com/questions/19551213/boost-iostreams-zlib-error-with-custom-source
http://gaabbe.win/issue/6764740/boost-iostreams-zlib-error-with-custom-source

A similar exception (but gzip_error, not zlib_error) was also reported in "Possible bug in gzip_decompressor"
http://lists.boost.org/boost-users/2012/05/74550.php

• I verified that change #1 was already in the boost 1.57.0 source (libs/iostreams/src/gzip.cpp:65)
• I attempted to patch iostreams with change #2 (libs/iostreams/src/zlib.cpp:153) changing crc_imp_ = 0; to crc_ = crc_imp_ = 0;, but it had no impact on the problem I was experiencing

A ticket for change #2 was created awhile ago:

• http://lists.boost.org/boost-users/2012/06/74871.php
• https://svn.boost.org/trac/boost/ticket/6994

The proposed fix (change #2) may fix an issue, just not the one I'm seeing.

[smanders]

Searching for zlib_decompressor on the boost trac site:
https://svn.boost.org/trac/boost/search?q=zlib_decompressor

I found #7810: Bugs: zlib_error::check throws on Z_BUF_ERROR
https://svn.boost.org/trac/boost/ticket/7810
was still open as of Feb 2016

There is an open pull to boostorg/iostreams
boostorg/iostreams#17

I forked boost iostreams, cherry-picked the commits to an xp1.57.0 and xphpp1.57.0 branch -- NOTE: had to separate cpp mods from hpp mods so the patching of the downloaded boost .tar.bz2 file works correctly

And created an externpro patch of iostreams -- but this also did not fix the issue I'm seeing

[smanders]

The sample decompress.cpp code above determines the decompressor to push by looking at the file extension:

std::string ext = filepath.extension().string();
if (ext.compare(".bz2") == 0)
  in.push(bio::bzip2_decompressor());
else if (ext.compare(".gz") == 0)
  in.push(bio::gzip_decompressor());
else if (ext.compare(".Z") == 0)
  in.push(bio::zlib_decompressor());
else
{
  std::cerr << filepath.filename().string() << ": unsupported extension (must be .[bz2|gz|Z])" << std::endl;
  return 1;
}

As I was searching for answers to issues I was dealing with, I came across an interesting idea:
"[iostreams] experience with automatically decompressing with gzip or bzip2"
http://boost-users.boost.narkive.com/fgd3ubSC/iostreams-experience-with-automatically-decompressing-with-gzip-or-bzip2

• read any file compressed with either gzip or bzip2, by analyzing the input stream and deducing which decompressor to use