0047 XLS-47d Price Oracles

[gregtatcam]

Latest version of spec: XLS-47 Price Oracles

Implementation: PR #4789

 
   Title:        Oracles on XRP Ledger
   Revision:     1 (2023-08-18)
 
  Author: Gregory Tsipenyuk
   Affiliation:  Ripple
 
 

Oracles on XRP Ledger

Abstract

This proposal adds on-chain PriceOracle object to XRPL ledger. A blockchain oracle is a system or service that acts as a bridge between a blockchain network and the external world, providing off-chain data or information to decentralized applications (dApps) on the blockchain. Oracles are used to bring real-world data, for instance market prices, exchange rates, interest rates, or weather conditions onto the blockchain, enabling dApps to access and utilize information that resides outside the blockchain. This document outlines the protocols involved in PriceOracle on XRPL ledger and provides guidelines for developers and system architects to implement and utilize this solution effectively. It introduces a new on-ledger PriceOracle object and the transactions to create, delete, and update the PriceOracle and adds get_aggregate_price API to retrieve an aggregate mean, trimmed mean, and median for the provided price oracles. This feature requires an amendment.

Terminology

• Oracle Provider is a service or technology that enables the integration of external data and real-world events into a blockchain network.
• dApp, short for decentralized application, refers to an application that is built on a blockchain network and operates using smart contracts or rely on other mechanisms or protocols for their functionality.

Creating PriceOracle instance on XRPL

On-Ledger Data Structure

PriceOracle object

The PriceOracle ledger entry represents the PriceOracle object on XRPL ledger and contains the following fields:

FieldName	Required?	JSON Type	Internal Type
OracleID	✔️	string	HASH256
Owner	✔️	string	ACCOUNTID
Symbol	✔️	string	CURRENCY
PriceUnit	✔️	string	CURRENCY
SymbolClass	✔️	string	BLOB
SymbolPrice	✔️	number	UINT64
Scale	✔️	number	UINT8
LastUpdateTime	✔️	number	UINT32
Provider	✔️	string	BLOB
PreviousTxnID	✔️	string	UINT256
PreviousTxnLgrSeq	✔️	string	UINT32

• OracleID. The ID of a PriceOracle object is the SHA-512Half of the following values, concatenated in order:
  • The Oracle space key ORACLE (0x52)
  • Owner field
  • Symbol field
  • PriceUnit field
• Owner is the account that has the update and delete privileges. It is recommended that this account has an associated signer list.
• Symbol is the symbol to be priced. Any arbitrary value should be allowed and interpreted exactly like other asset code fields in the ledger. A new enum value STI_CURRENCY and class STCurrency are introduced to support the CURRENCY field.
• PriceUnit is the denomination in which the prices are expressed. Any arbitrary value should be allowed and interpreted exactly like other asset code fields in the ledger.
• SymbolClass describes a type of the asset, for instance "currency", "commodity", "index". It is a string of up to ten ASCII hex encoded characters (0x20-0x7E).
• SymbolPrice is the scaled asset price, which is the price value after applying the scaling factor.
• Scale is the price's scaling factor. It represents the price's precision level. For instance, if Scale is 6 and the original price is 0.155 then the scaled price is 155000. Formally, $scaledPrice = originalPrice*{10}^{scale}$. Valid Scale range is {0-10}.
• LastUpdateTime is the specific point in time when the data was last updated. The LastUpdateTime is the ripple epoch time.
• Provider identifies an Oracle Provider, for instance chainlink. It is a string of up to 16 ASCII hex encoded characters (0x20-0x7E).
• PreviousTxnID is the hash of the previous transaction to modify this entry. (Same as on other objects with this field.).
• PreviousTxnLgrSeq is the ledger index of the ledger when this object was most recently updated/created. (Same as other objects with this field.)

Example of PriceOracle JSON

{
    "LedgerEntryType": "PriceOracle",
    "OracleID": "00070C4495F14B0E44F78A264E41713C64B5F89242540EE255534400000000000000",
    "Owner": "rsA2LpzuawewSBQXkiju3YQTMzW13pAAdW",
    "Symbol": "XRP",
    "PriceUnit": "USD",
    "SymbolClass": "63757272656E6379",
    "SymbolPrice": 74,
    "Scale": 2,
    "LastUpdateTime": 743609414,
    "Provider": "70726F7669646572",
    "PreviousTxnID": "C53ECF838647FA5A4C780377025FEC7999AB4182590510CA461444B207AB74A9",
    "PreviousTxnLgrSeq": 56865244
}

Transactions

This proposal introduces several new transactions to allow for the creation, update, and deletion of the PriceOracle object.

Transaction for creating PriceOracle instance

We define a new transaction CreateOracle for creating a new PriceOracle instance. Before the transaction can be submitted, the Oracle Provider has to do the following:

• Create or own the Account XRPL account with sufficient XRP balance to meet the reserve and the transaction fee requirements.
• The Oracle Provider has to publish the Account account public key so that it can be used for verification by dApp’s.

Transaction fields for CreateOracle transaction

FieldName	Required?	JSON Type	Internal Type
TransactionType	✔️	string	UINT16

Indicates a new transaction type CreateOracle. The integer value is 41.

FieldName	Required?	JSON Type	Internal Type
Account	✔️	string	ACCOUNTID

Account is the account that has the Oracle update and delete privileges.

FieldName	Required?	JSON Type	Internal Type
Symbol	✔️	string	CURRENCY

Symbol is the symbol to be priced.

FieldName	Required?	JSON Type	Internal Type
PriceUnit	✔️	string	CURRENCY

PriceUnit is the denomination in which the prices are expressed.

FieldName	Required?	JSON Type	Internal Type
SymbolClass	✔️	string	BLOB

SymbolClass describes an asset type.

FieldName	Required?	JSON Type	Internal Type
Provider	✔️	string	BLOB

Provider identifies an Oracle Provider.

The transaction fails if:

• An object with the OracleID already exists.
• A required field is missing.
• The Account account doesn't exist.
• The transaction is not signed by the Account account or the account's multi signers.

On success the transaction creates a new PriceOracle object.

Example of CreateOracle transaction JSON

{
    "TransactionType": "CreateOracle",
    "Account": "rsA2LpzuawewSBQXkiju3YQTMzW13pAAdW",
    "Symbol": "XRP",
    "SymbolClass": "63757272656E6379",
    "PriceUnit": "USD",
    "Provider": "70726F7669646572"
}

Transaction for deleting Oracle instance

We define a new transaction DeleteOracle for deleting an Oracle instance.

Transaction fields for DeleteOracle transaction

FieldName	Required?	JSON Type
TransactionType	✔️	string

Indicates a new transaction type DeleteOracle. The integer value is 42.

FieldName	Required?	JSON Type
OracleID	✔️	string

The ID of an Oracle object.

FieldName	Required?	JSON Type	Internal Type
Account	✔️	string	ACCOUNTID

Account is the account that has the Oracle update and delete privileges.

DeleteOracle transaction deletes the Oracle object from the ledger.

The transaction fails if:

• Object with the OracleID doesn't exist.
• The transaction is not signed by the Account account or the account's multi signers.

On success the transaction deletes the Oracle object.

Example of DeleteOracle transaction JSON

{
    "TransactionType": "DeleteOracle",
    "OracleID": "00070C4495F14B0E44F78A264E41713C64B5F89242540EE255534400000000000000",
    "Account": "rsA2LpzuawewSBQXkiju3YQTMzW13pAAdW"
}

Transaction for updating PriceOracle instance

We define a new transaction UpdateOracle for updating an Oracle instance.

Transaction fields for UpdateOracle transaction

FieldName	Required?	JSON Type	Internal Type
TransactionType	✔️	string	UINT16

Indicates a new transaction type UpdateOracle. The integer value is 43.

FieldName	Required?	JSON Type	Internal Type
OracleID	✔️	string	HASH256

The ID of an Oracle object.

FieldName	Required?	JSON Type	Internal Type
Account	✔️	string	ACCOUNTID

Account is the account that has the Oracle update and delete privileges.

FieldName	Required?	JSON Type	Internal Type
SymbolPrice	✔️	number	UINT64

SymbolPrice is the scaled asset price, which is the price value after applying the scaling factor.

FieldName	Required?	JSON Type	Internal Type
Scale	✔️	number	UINT8

Scale is the price's scaling factor.

FieldName	Required?	JSON Type	Internal Type
LastUpdateTime	✔️	number	UINT32

LastUpdateTime is the specific point in time when the data was last updated.

The transaction fails if:

• Object with the OracleID doesn't exist.
• The transaction is not signed by the Account account or the account's multi signers.

On success the transaction updates the respective Oracle data.

Example of Price UpdateOracle transaction JSON

{
    "TransactionType": "UpdateOracle",
    "OracleID": "00070C4495F14B0E44F78A264E41713C64B5F89242540EE255534400000000000000",
    "Account": "rsA2LpzuawewSBQXkiju3YQTMzW13pAAdW",
    "SymbolPrice": 740,
    "Scale": 3,
    "LastUpdateTime": 743609014
}

API's

Retrieving The Oracle

An Oracle object can be retrieved with the ledger_entry API call by specifying the OracleID ID.

Example of ledger_entry API JSON

Request JSON

{
     "method ":  "ledger_entry ",
     "params" : [
         "oracle ":  "D08D87CE50F215520D7AFEABADFD0669F7B89F9E40036FCA1906056E229FA800",
         "ledger_index ":  "validated "
     ]
}

Response JSON

{
   "index" : "CF2C20122022DE908C4F521A96DC2C1E5EFFD1EFD47AA244E9EE9A442451162E",
   "ledger_current_index" : 23,
   "node" : {
      "Flags" : 0,
      "LastUpdateTime" : 743609014,
      "LedgerEntryType" : "Oracle",
      "Owner" : "rp847ow9WcPmnNpVHMQV5A4BF6vaL9Abm6",
      "PreviousTxnID" : "6F120537D0D212FEA6E11A0DCC5410AFCA95BD98D451D046832E6C4C4398164D",
      "PreviousTxnLgrSeq" : 22,
      "PriceUnit" : {
         "currency" : "USD"
      },
      "Scale" : 1,
      "Symbol" : {
         "currency" : "XRP"
      },
      "SymbolClass" : "63757272656E6379",
      "SymbolPrice" : "2ed",
      "index" : "CF2C20122022DE908C4F521A96DC2C1E5EFFD1EFD47AA244E9EE9A442451162E"
   },
   "status" : "success",
   "validated" : true
}

Oracle Aggregation

get_aggregate_price API retrieves the aggregate price of the PriceOracle.

API fields are:

FieldName	Required?	JSON Type
ledger_index		string or number (positive integer)

The ledger index of the max ledger to use, or a shortcut string to choose a ledger automatically.

FieldName	Required?	JSON Type
ledger_hash		string

A 20-byte hex string for the max ledger version to use.

FieldName	Required?	JSON Type
symbol	✔️	string

symbol is the symbol to be priced.

FieldName	Required?	JSON Type
price_unit	✔️	string

price_unit is the denomination in which the prices are expressed.

FieldName	Required?	JSON Type
oracles	✔️	array

oracles is an array of oracle_id objects to aggregate over.

FieldName	Required?	JSON Type
trim		number

trim is the percentage of outliers to trim. Valid trim range is 1-25.

FieldName	Required?	JSON Type
flags	✔️	number

flags specifies aggregation type. At least one flag must be specified. The flags can be bitwise added.

Flag Name	Flag Value	Description
fSimpleAverage	0x01	Calculate the average price by summing up the prices of all the individual transactions or assets and dividing it by the total number of transactions/assets. This method treats each transaction or asset equally.
fMedian	0x02	Arrange the prices of all the transactions or assets in ascending order and select the middle value as the median price. This method can be used when there are extreme outliers that may skew the average.
fTrimmedMean	0x04	Calculate the average price after removing a specified percentage of extreme values from both ends of the data distribution. trim parameter should specify the percentage to trim. For instance, if trim is set to 10% then the lowest 5% and highest 5% of price points are removed.

Example of get_aggregate_price API JSON

Request JSON

{
"method": "get_aggregate_price",
"params": [
    {
        "ledger_index": "current",
        "symbol": "XRP",
        "price_unit": "USD",
        "flags": 7,
        "trim": 20,
        "oracles": [
          {
            "oracle_id": "00070C4495F14B0E44F78A264E41713C64B5F89242540EE255534400000000000000"
          },
          {
            "oracle_id": "1B4F0E9851971998E732078544C96B36C3D01CEDF7CAA332359D6F1D835670140000"
          },
          {
            "oracle_id": "A4E624D686E03ED2767C0ABD85C14426B0B1157D2CE81D27BB4FE4F6F01D688A0000"
          },
          {
            "oracle_id": "FD61A03AF4F77D870FC21E05E7E80678095C92D808CFB3B5C279EE04C74ACA130000"
          },
          {
            "oracle_id": "A140C0C1EDA2DEF2B830363BA362AA4D7D255C262960544821F556E16661B6FF0000"
          }
        ]
    }
]
}

Response JSON

{
   "ledger_current_index" : 23,
   "simple_average" : "74.45",
   "median" : "74.45",
   "trimmed_mean": "70",
   "status" : "success",
   "validated" : false
}

[Silkjaer]

Serveral existing oracles are using trustlines for the same purpose (e.g. https://github.com/XRPL-Labs/XRPL-Persist-Price-Oracle and https://dev.to/wietse/aggregated-xrp-usd-price-info-on-the-xrp-ledger-1087).

This XLS suggests a new LedgerObject and API methods for the same purpose. The main benefit is less abstraction and better discoverability? Or do I miss something 😀

IMO on-chain oracles are mainly beneficial to smart contracts (as you also write in the abstract), as off-chain applications might be better served by faster APIs than waiting for off-chain data to be updated to an on-chain oracle to be consumed off-chain again. What is the main purpose for on-chain oracles without smart contract support, or is this envisioned in a Hooks environment?

In that case, it might not even be necessary as Hooks, if I understand correctly, already offers a higher abstraction for storing arbitrary information.

[jazzi-cooper]

Thank you, @Silkjaer. I agree with you on the advantages: reduced abstraction, improved discoverability - and generally an enhanced developer experience, particularly beneficial for newcomers to XRPL.

Regarding applications, an oracle becomes essential for potential native DeFi protocols such as lending/borrowing - or possibly the AMM, if adopted. Additionally, there's a need for enhanced price feed infrastructure for explorers and information-focused apps. Hooks, as you noted, will further broaden DeFi opportunities on XRPL, potentially ushering in a range of new use cases.

[Silkjaer]

an oracle becomes essential for potential native DeFi protocols such as lending/borrowing - or possibly the AMM, if adopted.

Then I think it should be part of one of those native amendments?

Additionally, there's a need for enhanced price feed infrastructure for explorers and information-focused apps.

Isn't it faster to use off-chain information, and more reliable?

Hooks, as you noted, will further broaden DeFi opportunities on XRPL, potentially ushering in a range of new use cases.

Hooks won't need a native oracle object?

[jazzi-cooper]

A dedicated specification for oracles will increase composability, efficiency, and standardization on XRPL. If, say, a lending protocol incorporates its own oracle functionality, it would restrict other protocols, like AMM or other on-chain applications, from tapping into that data without a dependency on lending protocol.

By setting a standard for price feed data creation, storage, and management, and collaborating with trusted decentralized providers like Chainlink or Band protocol, all XRPL apps and protocols, whether native or hooks-based, can access a dependable shared data source.

[jazzi-cooper]

Wallets are another example: The request for in-app pricing feature seems to have been the impetus for this proposal: https://dev.to/wietse/aggregated-xrp-usd-price-info-on-the-xrp-ledger-1087

We would like to formalize this via this proposal for a number of reasons related to improving dev experience, both for data providers and especially data consuming dApps.

[mvadari]

Hooks won't need a native oracle object?

Hooks could still use a native oracle object, right? Since you can just emit a tx. And this way it can be used by other non-Hooks usecases as well, such as integrating with external APIs.
(I might be wrong but I believe that HookState objects can only be edited by hooks.)

[RichardAH]

Aside from the obvious question "how is this useful without smart contracts?" ... why are price and scale specified individually as uint64 and uint8? The XRPL already has a bespoke base-10 floating point system STAmount which gives you 54 bits of mantissa and 8 bits of exponent. All IOU balances are currently encoded this way, why not use it here too?

[gregtatcam]

It doesn't fit well into STAmount because there is no issuer account in this case. Also it's common for Price Oracles to provide the price as a scaled uint.

[dangell7]

I'm interested in the trust part? Anyone can create prices for any asset? I could see a price, spin up 30 accounts and submit prices?

EDIT: You choose the oracles that you want the price aggregated from. I misunderstood that.

[dangell7]

I guess to further, do you see dapps creating their own oracles or do you see dapps using other already created oracles?

[jazzi-cooper]

We expect dapps to use third-party oracles from established price feed data furnishers, such as Chainlink, Band, DIA, etc.

Regarding trust, you are correct, we expect dapps to vet the oracles they choose to use by reviewing the creator's site/docs.

[wojake]

I like this concept but let's talk use case.

If there were native smart contracts, this would definitely be useful to add to the ledger but what's the rationale behind this without smart contracts as users could AND would be requesting price data from third party sites, APIs and services?

[wojake]

Without native computation (smart contracts), there is no real (immediate) need for native off-chain data.

[jazzi-cooper]

Native computation can be done at the protocol-level without smart contracts - for example the DEX and proposed AMM. Oracles would also be required for a native borrow/lending protocol on XRPL.

For token-based applications, a native Oracle could be useful for decentralized stablecoins or other collateral-backed assets. Even apps like explorers or token informational sites could benefit from using an on-chain decentralized price data as opposed to centralized APIs, especially if traders are relying on that information to execute trading strategies.

[shortthefomo]

Having spent a number of hours actually building Oracles. There are a number of issues I would like to cover with this spec.

1. One ledger entry per Pair is extremely restrictive. If I have 100 pairs I would like data for that now means a 100 records I have to submit.
2. The data is all contained in the ledger there is no optionality. I could very well decide my data should be paid for as a oracle provider.
3. There is no accountability included in the data set that is stored here. How can an oracle provider attest to the data that was submitted? Typically a hash from their ledger or data base should be provided with the submission. So that I as a user can point to the providers data at some point in time and ask for attestation how the result was derived.
4. We already achieve this via an accountSet transaction as done by xumm. It faces the same issues as the data submitted is per pair (https://xrpscan.com/account/rXUMMaPpZqPutoRszR29jtC8amWq3APkx)

Here is an example of data I produce as a Oracle.

Notice the vast amount of data I produce with the records, this speaks to the attestation of the result set. The rawData, exchanges data was collected from, timestamp, age of data, etc... From this raw data im able as a provider to attest to the results produced.

I would suggest reworking this spec for the following.

1. Add a URI field to allow off data to be referenced. (optional)
2. Remove specific fields around a single pair and rather have a single field that allows a limited number of pairs. (if you have more pairs than that its only then an extra transaction to the ledger).
3. Remove this notion of aggregated price from all the oracles unless im able to spec which oracles I want to aggregate price from... this data is much like a trustline you have to trust the provider is not feeding you rubbish.

Here I can now optionally keep my oracle data private (offline to those i service) via the URI. And/Or publicly publish the data via the pair list field.

This makes this a lot more flexible in the way data can be provided from oracles and allows even other ledgers/chains to submit their records here (the URI simply references their hashes) or external centralised databases can be referenced in the same way.

[gregtatcam]

Hi @lathanbritz. Thank you for the suggestions.

1. How is off chain data can be used by an on-chain dApp, for instance landing protocol?
2. I have originally considered having an array of pairs in the oracle object but then decided against it because a) the on-ledger object size; b) update transaction complexity as would have to search for each pair in the array. The desired functionality in terms of multiple pair update can be accomplished by changing the proposed update transaction to support array of pairs. The update transaction would look then something like this:

{
   "TransactionType": "UpdateOracle",
   "Account": "rsA2LpzuawewSBQXkiju3YQTMzW13pAAdW",
   "prices": [ 
   {
       "OracleID": "00070C4495F14B0E44F78A264E41713C64B5F89242540EE255123400000000000000",
       "SymbolPrice": 540,
       "Scale": 2,
       "LastUpdateTime": 743609124
   },
  ...
  {
     "OracleID": "12370C4495F14B0E44F78A264E41713C64B5F89242540EE255534400000000000000",
     "SymbolPrice": 740,
     "Scale": 3,
     "LastUpdateTime": 743609014
 }
]
}

3. The aggregate price does require specifying the oracle by providing the OracleID, which is associated with specific Oracle Provider.

Regarding the attestation. The oracle object includes the Oracle's public key (Account), which the Oracle should publish on their website, just like they publish the smart contract address. For instance, this ETH/USD price feed from chainlink. The transaction must be signed by the Account or multi-signers if it is setup for multi-signing. The Oracle provider can publish the hashes of successful update transactions, similarly to the way they publish the smart contract transactions (see the same link above).

[shortthefomo]

Hi @gregtatcam

1. For instance my oracle can return realtime data, the XRPL only closes ledgers every 4 seconds. As open free data that might be perfect for many usecases. If they wanted faster data they could use a connection into the realtime oracle, by me.
2. This is the trade off I understand, but let's do a small example say we had x5 oracle providers each pushing an average of 50 pairs. That would be 250 transactions every single ledger on top of the normal traffic the XRPL has to handle. This is not sustainable... esp if the number of data providers grows.
3. Missed that super.

Fair enough with regards to using the submission hash as the identifier back to the attestation, agreed that can be achieved like that.

[gregtatcam]

Hi @lathanbritz

1. But this off-chain data can't be used by the ledger, right? First, XRPL doesn't support communicating to an off-chain entity. Even if it did support it then validators would not be able to achieve consensus as they might get a different data because of the latency, etc.
2. On-ledger object must be limited in size. We might have more leeway with the transaction size. As I said, we could batch updates for the same provider into one transaction. So 5 oracles with 50 pairs will result in 5 transactions, not 250.

[shortthefomo]

1. Correct, but as XRPL is today there is nothing on chain that would be able to react to this data (that will change i know). But interfaces could opt to use realtime data over ledger data if the provider is trusted. Additionally some L2's eg. Evernode can handle off chain and on chain data, so that consensus is handled there.
2. Right, but the nature of this data is up loaded every ledger so its effect on transaction throughput is much higher than any other type of submission the ledger has had to deal with to date. The balance needs to be both at the size of data on ledger and the traffic to the nodes. Hence again why offering the split between offering RAW price values and/or additional data via a URI is more flexible. Can you point me to how this batching is done I did not see it mentioned here?

[gregtatcam]

1. I assume the URI is for each pair, right? So it's probably like 256 bytes. With an array of pairs we are kind of pushing the boundaries. We could increase the reserve requirements depending on the array size. With 50 pairs it's probably going to be one of the largest objects on the ledger.
2. The traffic is going to be about the same, is it not? It might be actually more in case of one oracle with array of pairs since we have to include the asset pair to locate the right entry in the array. And locating the entry is going to have an impact on the transaction throughput since we have to search the array for each pair. The example is in my comment above: 0047 XLS-47d Price Oracles #129 (reply in thread)

[gregtatcam]

1. But URI has to be stored on the ledger, right?

[shortthefomo]

1. yes but it not per pair but just for the batch.

[gregtatcam]

So it's per oracle and then and it makes a convincing argument (if we include it) to have an array of pairs. Otherwise every oracle would have to have it. I'm still not convinced if we should include it or not based on the current way the ledger works. It could be of course optional, but still if it's used then it might substantially increase the object size. If we do go with the array, we have to reasonably limit the number of pairs. And there is still an issue of a) including in update transaction the asset pair for each price; b) locating the asset pair in the array (though it's not a big deal if it's reasonably small).

[shortthefomo]

The URI would need to be at the collection of data submitted. Not the oracle definition. But its not one URI per pair but one URI for all 50 records (like previous examples)

[gregtatcam]

If we move ahead with the array then, I think, we'd have to increase the reserve requirement for Oracle to two. We could have two tiers with one reserve and two reserves requirement. The first tier has the array size of 5, which results in Oracle size ~360. The second tier has the array size of 10, which results in Oracle size ~600.

[gregtatcam]

Thank you all for the feedback. I'm going to close this discussion by the end of the next week (October 6) and move the updated specs draft to the standards. The update is going to include the array of token pairs and URI per @lathanbritz suggestion. It'll also consolidate create and update into one transaction.

[gregtatcam]

Moved to standards.