Error: BLPAPI_ERROR_INVALID_CONVERSION for field PX_VOLUME

[volemont]

Downloading PX_VOLUME for the ticker SPX Index fails with Error: BLPAPI_ERROR_INVALID_CONVERSION. The field VOLUME works.

Reproducible example:

library(Rblpapi)
blpConnect()
bdh("SPX Index", "PX_VOLUME", as.Date("1970-01-01"))

Output:

> library(Rblpapi)
Rblpapi version 0.3.3.2 using Blpapi headers 3.8.1.1 and run-time 3.8.1.1.
Please respect the Bloomberg licensing agreement and terms of service.
> blpConnect()
> bdh("SPX Index", "PX_VOLUME", as.Date("1970-01-01"))
Error: BLPAPI_ERROR_INVALID_CONVERSION

Output of traceback():

4: stop(list(message = "BLPAPI_ERROR_INVALID_CONVERSION", call = NULL, 
       cppstack = NULL))
3: .Call("Rblpapi_bdh_Impl", PACKAGE = "Rblpapi", con_, securities, 
       fields, start_date_, end_date_, options_, overrides_, verbose, 
       identity_)
2: bdh_Impl(con, securities, fields, start.date, end.date, options, 
       overrides, verbose, identity)
1: bdh("SPX Index", "PX_VOLUME", as.Date("1970-01-01"))

Output of sessionInfo():

R version 3.2.3 (2015-12-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.4 (El Capitan)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Rblpapi_0.3.3.2

loaded via a namespace (and not attached):
[1] tools_3.2.3 Rcpp_0.12.4

[eddelbuettel]

I'd say "their fault":

R> library(Rblpapi)
Rblpapi version 0.3.3.2 using Blpapi headers 3.8.8.1 and run-time 3.8.8.1.
Please respect the Bloomberg licensing agreement and terms of service.
R> fieldInfo("PX_VOLUME")
             id  mnemonic datatype  ftype
PX_VOLUME PR013 PX_VOLUME    Int32 String
R> fieldInfo("VOLUME")
          id mnemonic datatype ftype
VOLUME RQ013   VOLUME    Int64  Real
R> 

It is declared 'String' and 'Int32' which does not make a lot of sense. Maybe we can add another workaround. Thoughts, @armstrtw ?

[joel23888]

I realize this likely does not help the OP, but I found it peculiar and wanted to note it here in case it helps with deciding how to work around it. The bdh request returns correct data on my system if the end date is set before 12/15/2000, otherwise I get the same error as OP.

> d1 <- bdh("SPX Index", c("PX_VOLUME"), as.Date("1970-01-01"), as.Date("2000-12-14"))
> str(d1)
'data.frame':   2784 obs. of  2 variables:
 $ date     : Date, format: "1990-01-01" "1990-01-02" ...
 $ PX_VOLUME: int  0 0 0 0 0 0 0 0 0 0 ...
> tail(d1)
        date        PX_VOLUME
2779    2000-12-07  1407995648
2780    2000-12-08  1663385600
2781    2000-12-11  1712259712
2782    2000-12-12  1445625856
2783    2000-12-13  1582478592
2784    2000-12-14  1403351296
> d2 <- bdh("SPX Index", c("PX_VOLUME"), as.Date("1970-01-01"), as.Date("2000-12-15"))
Error in eval(expr, envir, enclos): Attempt to access value of element 'PX_VOLUME'(type: 'Float64') as 'Int32' type.

I would volunteer to help but at present I cannot push code written on my BBG terminal to Github.

[johnlaing]

Fine-tuning Joel's idea:

> bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-14"), verbose=TRUE)
HistoricalDataResponse = {
    securityData = {
        security = "SPX Index"
        eidData[] = {
        }
        sequenceNumber = 0
        fieldExceptions[] = {
        }
        fieldData[] = {
            fieldData = {
                date = 2000-12-14
                PX_VOLUME = 1403351296.000000
            }
        }
    }
}
        date  PX_VOLUME
1 2000-12-14 1403351296
> bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-15"), verbose=TRUE)
HistoricalDataResponse = {
    securityData = {
        security = "SPX Index"
        eidData[] = {
        }
        sequenceNumber = 0
        fieldExceptions[] = {
        }
        fieldData[] = {
            fieldData = {
                date = 2000-12-14
                PX_VOLUME = 1403351296.000000
            }
            fieldData = {
                date = 2000-12-15
                PX_VOLUME = 2449779712.000000
            }
        }
    }
}
Error: Attempt to access value of element 'PX_VOLUME'(type: 'Float64') as 'Int32' type.

The volume on 12/15 exceeds the max for int32 of ~2.1b. So that explains why we can't make it an int32. What's unexplained is why they declare it as such. I guess someone should ask Bloomberg...

[joel23888]

Int32 does not make sense for this field. I am happy to raise it with BBG.

[joel23888]

Raised with BBG, they are having their programmers look at it. Will update here when I hear from them.

BBG helpdesk reference: H#680960625

[joel23888]

@wmorgan85 I have pasted in the helpdesk reference for this issue.

[wmorgan85]

Thanks, I am following up internally

On Tue, 19 Apr 2016 at 00:21 joel23888 [email protected] wrote:

@wmorgan85 https://github.com/wmorgan85 I have pasted in the helpdesk
reference for this issue.

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#163 (comment)

[eiwin]

@joel23888 have your got BBG's reply?
Thanks, Then how to deal with the error:
Error: Attempt to access value of element 'PX_VOLUME'(type: 'Float64') as 'Int32' type.

The same error occurs while I try to retrieve 'PX_VOLUME' of UKX Index 2012-06-16.

[joel23888]

I have not received a response from them. @wmorgan85 any updates on this?

@eiwin would it work for you if you use VOLUME instead of PX_VOLUME? See #159. Also please update to the v0.3.4 if you have not already. Then you should get:

> bdh("UKX Index", "VOLUME", as.Date("2012-06-13"), as.Date("2012-06-18"))
        date     VOLUME
1 2012-06-13  831658496
2 2012-06-14 1064836032
3 2012-06-15 2531612416
4 2012-06-18  934686144

Incidentally, 2012-06-16 was a Saturday so are you actually expecting a data point on that date?

[armstrtw]

The overall issue seems to be that many fields show up in fieldInfo as int32, they are automatically upscaled to int64 when the magnitude of the observation exceeds int32, which seems to happen frequently upon frequency reduction / aggregation.

There are two fixes I can see.

1. we maintain a list of exceptions map<stringName, storageType> and add to this list as we encounter exceptions

2. we simply import all int32 as doubles.

As much as I don't like the idea of forcing ints as doubles it would be far less work to maintain if we go this route.

I don't see much hope for this being fixed by bbg in any reasonable time horizon. So, I think we should deal with it in the package.

Totally open to other suggestions.

[eddelbuettel]

Just thinking out loud: can we not 'pretest' by casting to double and compare to INT_MAX (maybe via abs() too) and if it exceeds, store as double. A bit of work, but 'generic' without the need for a map.

A little bird whispered something to me recently too that there are plans about more reflective answer data sets as the issue has been recognised in some quarters. No timeline, of course.

[armstrtw]

but we are not doing lazyFrame any more. We need to know the types before we make the request. Or did I misunderstand your suggestion.

[eddelbuettel]

I was thinking about that too. Well that sucks.

New idea then: always store in double vector, if values "contained" then downcast to int vector.

[eiwin]

@joel23888
I made a mistake. I mean I want the data point on Jun 15-2012. And it is OK with the field "VOLUME" rather "PX_VOLUME". Great, Thanks.

[joel23888]

I asked the BBG helpdesk for an update and their advice boils down to handling this issue on the Rblpapi side. They also said that their programmers are reviewing the data types list (no timelines).

[joel23888]

A further update. BBG helpdesk explained how this is handled in VBA: there is a DataType property returned along with data, which can be checked and handled appropriately. I then went and looked in the C++ API Reference v3.8 to check this out and will share what I found, which is hopefully on the right path. (Also checked v3.7 and it is the same in this respect.)

If you look in http://bloomberg.github.io/blpapi-docs/cpp/3.8/classblpapi_1_1Element.html there is a function int datatype () const which should return the actual type for data returned in the response. The possible return values are enumerated in blpapi_datatype, which can be found here: http://bloomberg.github.io/blpapi-docs/cpp/3.8/structblpapi_1_1DataType.html.

Also in http://bloomberg.github.io/blpapi-docs/cpp/3.8/classblpapi_1_1Element.html if you search "API will convert data types as long as there is no loss of precision involved" you can see that they explicitly mention "An INT32 can be returned as Int32, Int64, Float64, std::string".

[armstrtw]

Yes, thanks @joel23888 , that is the field we used to use for lazyFrame. The problem is that as you walk through the rows of the response having the datatype change. It's not so much a problem in Excel as the cells don't care about what 'type' the cell above it is. However, in a dataframe or time series, you've already allocated a vector for the whole response. So, if it changes in mid-read, then you've got to go back and reallocate the whole thing to the new type, and then recopy your previous captured rows into it, and repoint the list element in your response to the new vector.

Not such a big deal, but in my opinion, just to allocate all int32's as doubles (at least for BDH) would be a simple way to avoid the messiness of the mid-read reallocation.

[wmorgan85]

So I've done a little further digging and gathered thoughts on this. The field search service (//blp/apiflds) gives you per field the data type and 'ftype' (unfortunately not the beautiful Jaguar, but instead some mysterious internal Bloomberg enumeration). This data type represents the expected type for the current value of the field, i.e. that which comes from the ReferenceDataRequest.

HOWEVER, looking at each of the requests in C# (ReferenceDataRequest and HistoricalDataRequest), PX_LAST and PX_VOLUME are both coming through as 64-bit floats.

As mentioned in a previous post, these are accessible by accessing the type property of the specific field. @armstrtw mentioned that there is a concern that the type of the column will change mid-read. This should never be the case for historical fields. Indeed, the response from the development team suggests that all historical fields will always return 64-bit floats.

Hope this helps.

In the interim, here's the example of the suggested Bloomberg approach of how to do the type inference (albeit truncated)

for(int k = 0; k < (int)d_fields.size(); k++)
{
  const char *temp_field_str = d_fields[k].c_str();
  if(element.hasElement(temp_field_str))
  {
    Element temp_field = element.getElement(temp_field_str);
    const Name TEMP_FIELD_STR(temp_field_str);

    datatype = temp_field.datatype();

    switch(datatype)
    {
      case BLPAPI_DATATYPE_FLOAT64://Float64
      {
         blpapi_Float64_t field1;
         field1 = element.getElementAsFloat64(TEMP_FIELD_STR);
         std::cout << field1 << *delimiter;
         break;
      }

      // etc...

      default:
      {
                 const char *field1;
                 field1 = element.getElementAsString(TEMP_FIELD_STR);
                 std::cout << field1 << *delimiter;
                 break;
      }
    }
  }
}

[randomee]

Any progress on this? I'm hitting this converting Rbbg code.

Thanks

[armstrtw]

I thought we had dealt with this issue. I think easiest to pull any ints as doubles in bdh.

[johnlaing]

So this works now (thanks to Whit) with a new option that we introduced in bdh. My previous example was:

> bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-15"))
Error in bdh_Impl(con, securities, fields, start.date, end.date, options,  : 
  Attempt to access value of element 'PX_VOLUME'(type: 'Float64') as 'Int32' type.

But now:

> bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-15"), int.as.double=TRUE)
        date  PX_VOLUME
1 2000-12-14 1403351296
2 2000-12-15 2449779712

[clivelamkk]

Hi,

I have followed johnlaing's instruction to add int.as.double=TRUE in bdh() function call as following:

bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-15"), int.as.double=TRUE)

However, I have received the following error message:

Error in bdh("SPX Index", "PX_VOLUME", as.Date("2000-12-14"), as.Date("2000-12-15"),  : 
  unused argument (int.as.double = TRUE)

May I know if I have missed some other packages?

• I am running under Windows 10
• RStudio version 1.0.136
• R version 3.3.2, 64-bit
• Rblpapi version 0.3.5

Regards,
Clive

[eddelbuettel]

@clivelamkk Why are you a) highjacking another thread and b) still refusing to format your post?

Show some courtesy, please.

[clivelamkk]

@eddelbuettel @ I have found out the solution from your previous post #207 and it was working. Thank you and sorry for making my mistake of not formatting my question properly in my previous posts.