API for passive network scanning

[prekucki]

Problem

Reviewing Available Offers

We want to review circulating offers on the network without specifying demand upfront. Currently, when inquiring about offers, we must declare what we will be ordering. This has optimization sense; the engine can implicitly present our request to the other party and preliminarily filter it based on the conditions given in the offer.

Inquiry for Offers

Offers on the network spread through slow broadcasting. In situations where we want to quickly learn about a specific node, we don't want to wait 4-8 minutes for information about the offer to reach us. In such a situation, it would be useful to have a calling feature that allows us to receive offers directly from a given node.

REST API proposal

Getting offer iterator

=> POST /market-api/v1/scan HTTP/1.1
=> Authorization: Bearer xxx
=> Content-Type: application/json
=> Accept: application/json
=> 
=> {"timeout": 300,  "type": "offer"}
<= HTTP/1.1 201 Created
<= Content-Type: application/json
<= 
<= "2e4939969a994127b621de742736d4"

• timeout - specifies how long an unused iterator will live in the server's memory. The value may be unspecified, defaulting to 30 seconds. This is intended to protect the server from memory leaks in applications that do not clean up their resources.

• type - scan type. offer for offers. demand for demands.

Iterating over offers

=> GET /market-api/v1/scan/2e4939969a994127b621de742736d4/events?timeout=5&maxEvents=100 HTTP/1.1
=> Authorization: Bearer xxx
=> Accept: application/json
=> 
<= HTTP/1.1 200 OK
<= content-type: application/json
<= 
<= [{
<=  "issuerId":"0xc7f9a611dab2734842e9833aa8cd5f38e2f75b65",
<=  "timestamp":"2024-03-01T08:40:45.887781403Z",
<=  "properties": {"golem.activity.caps.transfer.protocol": ["http","https","gftp"], ...}
<=  "constraints": "...",

<= }, ... ]

Deleting iterator

=> DELETE /market-api/v1/scan HTTP/1.1
=> 
<= HTTP/1.1 204 No Content
<=

Inquiry from given node

=> GET /market-api/v1/scan/2e4939969a994127b621de742736d4/events?peer=0xc7f9a611dab2734842e9833aa8cd5f38e2f75b65 HTTP/1.1
=> Authorization: Bearer xxx
=> Accept: application/json
=> 
<= HTTP/1.1 200 OK
<= content-type: application/json
<= 
<= [...]

[cryptobench]

Approved from my side. Only change was the events to something like offers

[cryptobench]

@prekucki Do you know anything more about when this could be added?

[grisha87]

I like the idea that we can ask for demands and offers from other nodes. Interestingly, this would allow Providers to inspect the demands of a particular requestor if they have their node ID?

You also stated:

the engine can implicitly present our request to the other party and preliminarily filter it based on the conditions given in the offer.

But in the provided example I don't see the part where we specify the conditions which the engine would use to filter the offers before returning them via the iterator. I assume that would be another property in the POST /market-api/v1/scan request and would be comprised of properties and constraints similarly to the Offer/Demand entities/types?

[cryptobench]

To me, not being able to specify the engine would be a benefit to the stats page. It would allow me to see all nodes on the network, all runtimes, even those i'm not aware of.

Basically it would give me a broader overview instead of only looking for what I know about.

[prekucki]

@cryptobench
Ready to test
https://github.com/golemfactory/yagna/releases/tag/pre-rel-v0.15.2-rc1

[cryptobench]

Thank you, much appreciated. I am not sure if I have capacity to test it next week, but I will try my best to squeeze it in.

[cryptobench]

Do you have some docs on how to use it? @prekucki

I don't know if its just expected to work like above or if you made changes.

[SewerynKras]

@prekucki @cryptobench
implementation in JS SDK is ready: golemfactory/golem-js#972 (with instructions how to run locally)
PR will remain in draft stage until this version of yagna is released