OPJ fails to decode image that KDU manages correctly

[gcode-importer]

Originally reported on Google Code with ID 419

trunk r2908 :
./bin/opj_decompress -i ../../ex/4.jp2 -o 4.bmp
[INFO] Start to read j2k main header (123).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Header of tile 1 / 1 has been read.
[ERROR] Stream too short, expected SOT
[ERROR] Failed to decode tile 1/1
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!


kdu_expand -i ../../ex/4.jp2 -o 4.bmp
Consumed 6 tile-part(s) from a total of 1 tile(s).
Consumed 1,834 codestream bytes (excluding any file format) = 1.084886
bits/pel.
Processed using the multi-threaded environment, with
    2 parallel threads of execution

Reported by mayeut on 2014-10-22 05:55:21

---

- _Attachment: [4.jp2](https://storage.googleapis.com/google-code-attachments/openjpeg/issue-419/comment-0/4.jp2)_

[gcode-importer]

Reported by mayeut on 2014-11-04 21:02:32

[gcode-importer]

Reported by malaterre on 2015-01-07 17:35:52

[emmenlau]

I experience a very similar (possibly identical?) issue with OpenJPEG 2.3.1. This comes as a regression over the 2.1 release for me. The image bVZ016-1M_wA20_s2_z0_t1_cDAPI_u001.jp2 has been created with Matlab R2014b, which AFAIK uses the Kakadu writer. It can be successfully decoded with OpenJPEG 2.1 and Kakadu, but not with OpenJPEG 2.3.1:

• OpenJPEG 2.3.1 output:

[JP2 codec error] Stream too short
[JP2 codec error] Failed to decode tile 1/1
[JP2 codec error] Failed to decode the codestream in the JP2 file

• Kakadu KDU7A2_Demo_Apps_for_Ubuntu-x86-64_170827 output:

> ./kdu_expand \
    -i /tmp/bVZ016-1M_wA20_s2_z0_t1_cDAPI_u001.jp2 \
    -o /tmp/bVZ016-1M_wA20_s2_z0_t1_cDAPI_u001.tif

Consumed 1 tile-part(s) from a total of 1 tile(s).
Consumed 1,027,757 codestream bytes (excluding any file format) = 5.679471 bits/pel.
Processed using the multi-threaded environment, with
    8 parallel threads of execution

We have a large image archive in this format. Help would be greatly appreciated.

[rouault]

@emmenlau No problem for me. Demo:

$ wget https://github.com/uclouvain/openjpeg/releases/download/v2.3.1/openjpeg-v2.3.1-linux-x86_64.tar.gz
$ tar xvzf openjpeg-v2.3.1-linux-x86_64.tar.gz
$ LD_LIBRARY_PATH=openjpeg-v2.3.1-linux-x86_64/lib openjpeg-v2.3.1-linux-x86_64/bin/opj_decompress -i  bVZ016-1M_wA20_s2_z0_t1_cDAPI_u001.jp2 -o out.tif

[INFO] Start to read j2k main header (85).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Header of tile 1 / 1 has been read.
[INFO] Stream reached its end !
[INFO] Generated Outfile out.tif
decode time: 187 ms

[emmenlau]

Dear @rouault , thanks a lot for the quick reply! I can reproduce your result, so the downloaded opj_decompress works for me too. This is very curious because I can actually trace the problem to a specific location in j2k.c around line 8987:

    if (p_j2k->m_specific_param.m_decoder.m_state != J2K_STATE_EOC) {
        // This comes to the end of the stream and prints "Stream reached its end !":
        if (opj_stream_read_data(p_stream, l_data, 2, p_manager) != 2) {
            opj_event_msg(p_manager, EVT_ERROR, "Stream too short\n");
            // So that subsequently this fails for me:
            return OPJ_FALSE;
        }

It seems opj_decompress does also print "Stream reached its end !" but then does not fail. Does that make sense to you? Could you help me understand if I'm looking in the right location of j2k.c?

[rouault]

Might be related to how you use the openjpeg API. Posting a self contained C code reproducing your issue would help.

[emmenlau]

I've made some progress but the issue is still not completely clear to me. The error comes from the fact that we use a c++ iostream for reading, with the following callbacks:

static OPJ_UINT64 _LengthProc(std::fstream *file) {
    file->seekg(0, std::fstream::end);
    OPJ_UINT64 fsize = file->tellg();
    file->seekg(0, std::fstream::beg);
    return fsize;
}

static OPJ_SIZE_T _ReadProc(void *p_buffer, OPJ_SIZE_T p_nb_bytes, void *p_user_data) {
    std::fstream *file = (std::fstream*) p_user_data;
    file->read((char*)p_buffer, p_nb_bytes);
    OPJ_SIZE_T l_nb_read = file->gcount();
    return l_nb_read ? l_nb_read : (OPJ_SIZE_T) - 1;
}

static OPJ_SIZE_T _WriteProc(void *p_buffer, OPJ_SIZE_T p_nb_bytes, void *p_user_data) {
    std::fstream *file = (std::fstream*) p_user_data;
    file->write((char*)p_buffer, p_nb_bytes);
    return (!file->fail()) ? p_nb_bytes : -1;
}

static OPJ_OFF_T _SkipProc(OPJ_OFF_T p_nb_bytes, void *p_user_data) {
    std::fstream *file = (std::fstream*) p_user_data;
    file->seekg(p_nb_bytes, std::fstream::cur);
    return (!file->bad()) ? p_nb_bytes : -1;
}

static OPJ_BOOL _SeekProc(OPJ_OFF_T p_nb_bytes, void * p_user_data) {
    std::fstream *file = (std::fstream*) p_user_data;
    file->seekg(p_nb_bytes, std::fstream::beg);
    return (!file->bad()) ? OPJ_TRUE : OPJ_FALSE;
}

When I switch to the C-style FILE* interface as used in openjpeg the callbacks work, so the error must be related to a difference in the behavior of std::fstream vs FILE*. I understand that this is out of scope now, but you'd greatly help me if there is something that jumps to your eye?