Low precision types support in Convert operation

[jdanieck]

Details:

• Add u1 type support + tests
• Add u4 type support + tests
• Add i4 type support + tests
• Extended Serialization SLT with LP types
• Added Convert to summarize.py report

Tickets:

• 46096

[jdanieck]

@ilyachur @jane-intel @pelszkow guys I think we should introduce general support of LP types to ngraph. We should consider introduction of ngraph::u1 type & vector<ngraph::u1> (an the same for u4 & i4) where we could encapsulate all this "binary magic". Also we will be able to dispatch templates against this type and avoid "if(LP_TYPE)" tricks like in this PR. Please notice that I needed to repeat LP types handling code from Constant here, and who knows how many times we'll need it in the future for other operations. What do you think ?

[ilyachur]

@ilyachur @jane-intel @pelszkow guys I think we should introduce general support of LP types to ngraph. We should consider introduction of ngraph::u1 type & vector<ngraph::u1> (an the same for u4 & i4) where we could encapsulate all this "binary magic". Also we will be able to dispatch templates against this type and avoid "if(LP_TYPE)" tricks like in this PR. Please notice that I needed to repeat LP types handling code from Constant here, and who knows how many times we'll need it in the future for other operations. What do you think ?

The LP types are introduced in order to minimize the size of weights. If we want to work with vector of u1 or other LP types it means what vector will contain holes (for u1 the size of each element in vector is 1 byte not 1 bit), it means that we will increase the memory consumption.

[jdanieck]

@ilyachur @jane-intel @pelszkow guys I think we should introduce general support of LP types to ngraph. We should consider introduction of ngraph::u1 type & vector<ngraph::u1> (an the same for u4 & i4) where we could encapsulate all this "binary magic". Also we will be able to dispatch templates against this type and avoid "if(LP_TYPE)" tricks like in this PR. Please notice that I needed to repeat LP types handling code from Constant here, and who knows how many times we'll need it in the future for other operations. What do you think ?

The LP types are introduced in order to minimize the size of weights. If we want to work with vector of u1 or other LP types it means what vector will contain holes (for u1 the size of each element in vector is 1 byte not 1 bit), it means that we will increase the memory consumption.

I was thinking about specialization like with vector of bool which would allow us to keep the same memory usage.

My main point is that we should have some common place for code handling LP types in ngraph, implementation details we can discuss when we agree that it's needed. If you want to think about it on concreate use case you can look at TestCase class and think how to properly introduce LP types handling there (at this point they are not handled).

[ilyachur]

I was thinking about specialization like with vector of bool which would allow us to keep the same memory usage.

My main point is that we should have some common place for code handling LP types in ngraph, implementation details we can discuss when we agree that it's needed. If you want to think about it on concreate use case you can look at TestCase class and think how to properly introduce LP types handling there (at this point they are not handled).

Yeah, I know about the vector of bool, but unfortunately we also have u4, i4 lp types. And for such types we don't have a good solution

[jdanieck]

I was thinking about specialization like with vector of bool which would allow us to keep the same memory usage.
My main point is that we should have some common place for code handling LP types in ngraph, implementation details we can discuss when we agree that it's needed. If you want to think about it on concreate use case you can look at TestCase class and think how to properly introduce LP types handling there (at this point they are not handled).

Yeah, I know about the vector of bool, but unfortunately we also have u4, i4 lp types. And for such types we don't have a good solution

I am really bad in communicating today :) I think it is technically possible to implement our own specialization of vector, e.g. vector<ngraph::u4> & vector<ngraph::i4> and follow the same implementation concept as in vector<bool> to decrease memory usage. @pelszkow could you confirm it is technically possible? I think we should at least consider it if is technically possible, and if we conclude it is too much work then we can reject.

[pelszkow]

It looks like we should provide some tool which will be some mix of std::vector<bool> and std::span. This class should allow to iterate over the buffer/memory and access to LP like std::vector<bool> does with bits/bools.
e.g.:

template<int BitsNo>
struct Span{
    class Iterator;
    BitWrapper <BitsNo> operator[](size_t i);
    Iterator begin();
    Iterator end();
    //....
};

template<int BitsNo>
struct SpanIterator{
    BitWrapper <BitsNo> operator*();
    operator++();
    //....
}

template<int BitsNo>
struct BitWrapper {
    BitWrapper (uint8_t* mem, int index);
    operator uint8_t () const {/*do bit magic */}
    BitWrapper operator=(uint8_t value) {/*do bit magic */}
    //....
}

[ilyachur]

I was thinking about specialization like with vector of bool which would allow us to keep the same memory usage.
My main point is that we should have some common place for code handling LP types in ngraph, implementation details we can discuss when we agree that it's needed. If you want to think about it on concreate use case you can look at TestCase class and think how to properly introduce LP types handling there (at this point they are not handled).

Yeah, I know about the vector of bool, but unfortunately we also have u4, i4 lp types. And for such types we don't have a good solution

I am really bad in communicating today :) I think it is technically possible to implement our own specialization of vector, e.g. vector<ngraph::u4> & vector<ngraph::i4> and follow the same implementation concept as in vector<bool> to decrease memory usage. @pelszkow could you confirm it is technically possible? I think we should at least consider it if is technically possible, and if we conclude it is too much work then we can reject.

Oh, ok, now I got your point. So in this case, I think we need to try to implement it.

[jdanieck]

I was thinking about specialization like with vector of bool which would allow us to keep the same memory usage.
My main point is that we should have some common place for code handling LP types in ngraph, implementation details we can discuss when we agree that it's needed. If you want to think about it on concreate use case you can look at TestCase class and think how to properly introduce LP types handling there (at this point they are not handled).

Yeah, I know about the vector of bool, but unfortunately we also have u4, i4 lp types. And for such types we don't have a good solution

I am really bad in communicating today :) I think it is technically possible to implement our own specialization of vector, e.g. vector<ngraph::u4> & vector<ngraph::i4> and follow the same implementation concept as in vector<bool> to decrease memory usage. @pelszkow could you confirm it is technically possible? I think we should at least consider it if is technically possible, and if we conclude it is too much work then we can reject.

Oh, ok, now I got your point. So in this case, I think we need to try to implement it.

Ok, so just to be on the same page, I'll create separate ticket for this. Let's merge this PR as is, including your comments ofz.

[jdanieck]

@ilyachur any chance you will review today? I'd love to merge it before FF :)

[ilyachur]

@jdanieck I merged this PR, can we add more tests in the next PR?