General chain stats website

[Hiyorimi]

Summary

Website with stats about the mainnet.

Context

TON now lacks a single website / project to report blockchain stats like Solana bench or Etherscan/charts.

Goals

• Provide a reliable source of statistics regarding the TON Blockchain

Deliverables

• Working and functioning service with reasonable SLAs.

Definition of Done

[ ] Website is available online
[ ] Information on validators, transactions speed, volume etc. is displayed.

Reward

• Standard TON Footstep NFT
• 7500 USD equivalent in TON

Total: $7500

[pachainti]

I completely agree. The current network monitor is really limited and not up to the quality of the TON blockchain.

[shayan-shojaei]

I'm willing to develop such a website. Just wondering, won't TON rewards be available for the task like the other ones?

[tsivarev]

I'm willing to develop such a website. Just wondering, won't TON rewards be available for the task like the other ones?

It's a good news! How long do you think it will take?

[alfredonodo]

I'm willing to develop such a website. Just wondering, won't TON rewards be available for the task like the other ones?

It's a good news! How long do you think it will take?

Is it possible to use these projects from TON contest as starting point?

[tsivarev]

I'm willing to develop such a website. Just wondering, won't TON rewards be available for the task like the other ones?

It's a good news! How long do you think it will take?

Is it possible to use these projects from TON contest as starting point?

Why not. The purpose of this footstep is to provide a useful tool for the community.

[shayan-shojaei]

I'm willing to develop such a website. Just wondering, won't TON rewards be available for the task like the other ones?

It's a good news! How long do you think it will take?

It depends on the details, but for basic statistics, it would take roughly 2-3 weeks. I suggest that we break it up into multiple releases so that we can iteratively upgrade it afterward.
To maximize efficiency, I'm considering a system that has its own backend. Please let me know if you intend to use an available API.

[Hiyorimi]

Updated reward on this issue

[tontechio]

We have some data points at https://tontech.io/stats/ and a lot more under the hood. If somebody wants to work on that task we can provide the input data in exchange for reference on the source.

[shayan-shojaei]

Updated reward on this issue

Great! I'll get started then.
I'll be investigating the data sources as well to pick the best solution.

[Hiyorimi]

@shayan-shojaei hey, how are you doing?

[shayan-shojaei]

@shayan-shojaei hey, how are you doing?

Hey. it's mostly been going smoothly on the front-end side of things. but as for the back-end, I think it's going to take a little more time than I anticipated in the beginning, though I will try to get back to you with a progress report in a few days.

[shayan-shojaei]

@Hiyorimi
Hey. Following up on the progress report, I've set up three microservices, two for the back end and one for the front end.
The front-end is a Next.js application that uses SSR for the initial data print and Socket.io to keep data flowing into the app. I haven't done much work on the design, but it will most likely change in the future.
In the backend, a producer and a consumer instance are running and intercepting data (eg. transactions). The producer receives events from the ton-http-api, emits them to Kafka, and consumes them before inserting them into a mongo database.
There is also a Nest.js application for the API that returns data from its REST endpoints and provides sockets that consume data from Kafka in real-time and transmit it to the client side.
I tried to deploy the services so I could provide you with more of a demo, but I can't use any cloud provider services as I'm located in a country that is under sanctions by all of them. I did try a combination of Vercel and Heroku but the Kafka instance wasn't going to run on Heroku. So I could really use access to some cloud instances so I can deploy them and propose a functioning demo. I could also give you access to the repositories if you wanted to check them out.

[liketurbo]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

[liketurbo]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

Cause as I understand toncenter/ton-http-api it’s a http server, from which you can only call GET, POST methods, but it’s about it. Or am I missing something?

Or you do something like ‘GET getTransactions’ every second?

[liketurbo]

@Hiyorimi
Hey. Following up on the progress report, I've set up three microservices, two for the back end and one for the front end.
The front-end is a Next.js application that uses SSR for the initial data print and Socket.io to keep data flowing into the app. I haven't done much work on the design, but it will most likely change in the future.
In the backend, a producer and a consumer instance are running and intercepting data (eg. transactions). The producer receives events from the ton-http-api, emits them to Kafka, and consumes them before inserting them into a mongo database.
There is also a Nest.js application for the API that returns data from its REST endpoints and provides sockets that consume data from Kafka in real-time and transmit it to the client side.
I tried to deploy the services so I could provide you with more of a demo, but I can't use any cloud provider services as I'm located in a country that is under sanctions by all of them. I did try a combination of Vercel and Heroku but the Kafka instance wasn't going to run on Heroku. So I could really use access to some cloud instances so I can deploy them and propose a functioning demo. I could also give you access to the repositories if you wanted to check them out.

I hear you about sanctions, recently the country I’m living in have joined that club 😄. Only option for me now is to use a local service provider, what you gonna do 🤷‍♂️

[shayan-shojaei]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

Cause as I understand toncenter/ton-http-api it’s a http server, from which you can only call GET, POST methods, but it’s about it. Or am I missing something?

Or you do something like ‘GET getTransactions’ every second?

So yes, pretty much. I first started doing the same thing using the lite-client, but there was the hassle of cleaning up the output which was a raw string, but now I run an instance of the ton-http-api and ask for new transactions every second and there's no "rate limit" because I'm running the instance myself.

[liketurbo]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

Cause as I understand toncenter/ton-http-api it’s a http server, from which you can only call GET, POST methods, but it’s about it. Or am I missing something?
Or you do something like ‘GET getTransactions’ every second?

So yes, pretty much. I first started doing the same thing using the lite-client, but there was the hassle of cleaning up the output which was a raw string, but now I run an instance of the ton-http-api and ask for new transactions every second and there's no "rate limit" because I'm running the instance myself.

It's too bad that we have to rely on the tricks with GET getTransactions to get the latest blockchain events 😩

I myself went through this using the same principle of "mimicking subscribe functionality" in my recent project.

@Hiyorimi can we create a footstep for the "subscribe to the stream of blocks" project?

[shayan-shojaei]

2022-08-15.20-02-19.mp4

Here's a demo as well. I'm pretty new to TON and would love to get your input on what kind of data (i.e charts/tables) I can extract from the network. and also I'm not a designer so the design is not set in stone and can change with any good ideas :)

[liketurbo]

@shayan-shojaei can you share your code?

[shayan-shojaei]

@shayan-shojaei can you share your code?

I'm planning to make it open-source once it's reached the first usable release but I don't think it's in such a state right now.

[Hiyorimi]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

Cause as I understand toncenter/ton-http-api it’s a http server, from which you can only call GET, POST methods, but it’s about it. Or am I missing something?
Or you do something like ‘GET getTransactions’ every second?

So yes, pretty much. I first started doing the same thing using the lite-client, but there was the hassle of cleaning up the output which was a raw string, but now I run an instance of the ton-http-api and ask for new transactions every second and there's no "rate limit" because I'm running the instance myself.

It's too bad that we have to rely on the tricks with GET getTransactions to get the latest blockchain events 😩

I myself went through this using the same principle of "mimicking subscribe functionality" in my recent project.

@Hiyorimi can we create a footstep for the "subscribe to the stream of blocks" project?

yeah sure, why not? this is exactly what TON Footsteps was designed for!

[shayan-shojaei]

@Hiyorimi Hey! I've been waiting for a response from you for a couple of days now. Would you please answer my question so we can proceed forward in footstep's development? =)

[Hiyorimi]

@Hiyorimi Hey! I've been waiting for a response from you for a couple of days now. Would you please answer my question so we can proceed forward in footstep's development? =)

Hey, I'm really sorry. I carefully re-read the whole issue and haven't noticed any questions left unanswered. Would you be so kind as to point me the questions I missed?

[shayan-shojaei]

@Hiyorimi Hey! I've been waiting for a response from you for a couple of days now. Would you please answer my question so we can proceed forward in footstep's development? =)

Hey, I'm really sorry. I carefully re-read the whole issue and haven't noticed any questions left unanswered. Would you be so kind as to point me the questions I missed?

No worries :)
well, basically I was wondering whether I could be provided with access to an EC2 instance for deployment purposes.
and also if you have any ideas about what sort of data could be retrieved from the network that would be necessary or beneficial for the application to show/visualize? (as I said, I'm fairly new to the whole ecosystem.)

[liketurbo]

@shayan-shojaei Can I ask how you receiving events from toncenter/ton-http-api?

Cause as I understand toncenter/ton-http-api it’s a http server, from which you can only call GET, POST methods, but it’s about it. Or am I missing something?
Or you do something like ‘GET getTransactions’ every second?

So yes, pretty much. I first started doing the same thing using the lite-client, but there was the hassle of cleaning up the output which was a raw string, but now I run an instance of the ton-http-api and ask for new transactions every second and there's no "rate limit" because I'm running the instance myself.

It's too bad that we have to rely on the tricks with GET getTransactions to get the latest blockchain events 😩
I myself went through this using the same principle of "mimicking subscribe functionality" in my recent project.
@Hiyorimi can we create a footstep for the "subscribe to the stream of blocks" project?

yeah sure, why not? this is exactly what TON Footsteps was designed for!

Created a new footstep: Library for handling blockchain data.
General idea is to create a library that returns a stream object, which can be used to get the latest blockchain transactions.

[Hiyorimi]

@shayan-shojaei what do you think?

[shayan-shojaei]

@shayan-shojaei what do you think?

It's ok.
The work is almost done, my issue is just the first part of the deliverables as I asked you earlier and got no response about it.
The project includes multiple parts and it can't be deployed as an static website, so I will need an access to some cloud VPS instance. I don't have an access currently and buying from online providers isn't available as I'm in Iran. If you provide an environment to deploy, I would setup and make the site online. Can you provide any VPS instances?

[Hiyorimi]

@shayan-shojaei what do you think?

It's ok. The work is almost done, my issue is just the first part of the deliverables as I asked you earlier and got no response about it. The project includes multiple parts and it can't be deployed as an static website, so I will need an access to some cloud VPS instance. I don't have an access currently and buying from online providers isn't available as I'm in Iran. If you provide an environment to deploy, I would setup and make the site online. Can you provide any VPS instances?

TBD later

[AminRezaei0x443]

Is the lack of a VPS currently what keeps this footstep from getting forward? If so, I'd like to sponsor the VPS instance for about 2-3 months so this footstep could move forward. Also, We can move the project into @sky-ring and I'd be glad to assist in making the project better in many aspects like UI/UX.
@shayan-shojaei Please confirm whether you will be able to resume the progress once VPS is ready.
@Hiyorimi I think footsteps like this should offer a clear workflow about domain and infrastructure upon completion, since maintaining just the infrastructure comes with recurring costs.

[Hiyorimi]

Is the lack of a VPS currently what keeps this footstep from getting forward? If so, I'd like to sponsor the VPS instance for about 2-3 months so this footstep could move forward. Also, We can move the project into @sky-ring and I'd be glad to assist in making the project better in many aspects like UI/UX.
@shayan-shojaei Please confirm whether you will be able to resume the progress once VPS is ready.
@Hiyorimi I think footsteps like this should offer a clear workflow about domain and infrastructure upon completion, since maintaining just the infrastructure comes with recurring costs.

I see no problem in discussing moving the infrastructure costs of the project benefiting the TON community to the budget of TON Foundation.

[shayan-shojaei]

Is the lack of a VPS currently what keeps this footstep from getting forward? If so, I'd like to sponsor the VPS instance for about 2-3 months so this footstep could move forward. Also, We can move the project into @sky-ring and I'd be glad to assist in making the project better in many aspects like UI/UX. @shayan-shojaei Please confirm whether you will be able to resume the progress once VPS is ready. @Hiyorimi I think footsteps like this should offer a clear workflow about domain and infrastructure upon completion, since maintaining just the infrastructure comes with recurring costs.

Yes, as @Hiyorimi said that TF would cover the infrastructure costs, A VPS for two or even a month, would be enough to deliver this footstep and I'd be grateful.

[AminRezaei0x443]

@shayan-shojaei
Great then! I'll contact you with the credentials for a VPS. After the footstep completion and verification, we can handle the infrastructure as TF suggests.
@Hiyorimi Is the suggested flow Ok?

[Hiyorimi]

@shayan-shojaei
Great then! I'll contact you with the credentials for a VPS. After the footstep completion and verification, we can handle the infrastructure as TF suggests.
@Hiyorimi Is the suggested flow Ok?

of course, it makes sense!

[AminRezaei0x443]

Great, Just provided Shayan an instance's credentials!

[Gusarich]

@shayan-shojaei @AminRezaei0x443 any progress on this footstep?

[shayan-shojaei]

@shayan-shojaei @AminRezaei0x443 any progress on this footstep?

I've been communicating with @AminRezaei0x443 and we've come up with a new design both for the UI/UX and the project structure (data collection, ingestion, etc.) There's still some work to be done and it's not going as fast considering I'm working and studying full-time, but I'm trying to get things up and running soon.

[Gusarich]

@shayan-shojaei nice! Take your time

[AminRezaei0x443]

We're happy to announce that the implementation of the website is done, Thanks to @shayan-shojaei's efforts. We chose the name TONHouse, Lighthouse of TON, for it.

Website: https://tonhouse.skyring.io/
Frontend Repository: https://github.com/sky-ring/ton-house
Backend Repository: https://github.com/sky-ring/ton-house-api

As previously stated in the thread, the project has been merged to @sky-ring, and the UI/UX has been upgraded.

The project satisfies the footstep's DoD; Nevertheless, we expect to continue developing the website and adding additional data and analytics later, as outlined in TONHouse's Roadmap, as separate footsteps or grants if relevant. We believe that a complete analytics and statistics website would demonstrate how active TON and its ecosystem are, resulting in increased user growth.

@shayan-shojaei will proceed to create the PR, But feel free to add any comments on the project, so we can resolve it.

[alfredonodo]

Good work. Will TONHouse become the official network monitor of ton.org?

[tsivarev]

@AminRezaei0x443 Can you revert dates in these charts? And format the total weight counter?

[shayan-shojaei]

@AminRezaei0x443 Can you revert dates in these charts?

Yes, I was planning to do that today. Thanks for the reminder!

[tsivarev]

Maybe we can write a full form of LT (Logical Time) here? Looks like we have enough space.

[tsivarev]

What does "main" mean?

[shayan-shojaei]

@tsivarev Could you please create an issue in the project's repository so I can have all these there?

[tsivarev]

@tsivarev Could you please create an issue in the project's repository so I can have all these there?

Yes, sure. Thanks!

[alfredonodo]

The project site has been offiline for some time and the roadmap has not been completed.
Is there any news on this?
Thank you

[AminRezaei0x443]

Hi @alfredonodo,

We hit a bit of a bump on our end! We're currently untangling a tech glitch and we'll get the site up and running ASAP. As for the roadmap and future features, we're planning to cover that in a separate footstep soon. Sorry for any inconvenience, and thanks for your patience!