[Bug] The behavior of starting Dragonfly project is different with the document

[fengzixu]

Ⅰ. Issue Description

The actual download image behavior is inconsistent with the document description

Ⅱ. Describe what happened

I try to use Dragonfly according to document QuickStart

When I performed Step 5, I didn't see any output. After I checked the log file, I found that there is no words downloading piece to illustrate the Dragonfly run normally. It seems like that we modify the output of logging in dfclient. But we don't revise the doc at the same time.

Ⅲ. Describe what you expected to happen

The actual behavior should be the same as the document.

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Just perform every step in that doc.

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

• dragonfly version: 0.4.3
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[yeya24]

Yes, since #783, the log file will be rotated regularly. So we cannot find the output in dfclient.log. I think we should add some docs about this behavior. @inoc603

[fengzixu]

@yeya24 I think we need to find an approach to show "The file has been downloaded successfully by Dragonfly", not only to describe the behavior of log roration in doc. WDYT?

[yeya24]

@yeya24 I think we need to find an approach to show "The file has been downloaded successfully by Dragonfly", not only to describe the behavior of log roration in doc. WDYT?

I agree with you. Do you have any suggestions? I think a way to do this is via dfdaemon. It can expose some information like api to let uses know if one image is successfully downloaded. However, this is architecture-related because currently dfdaemon itself is just a proxy.

[fengzixu]

I think we need to define and output necessary info about downloading images through Dragonfly.

1. Begin to download
2. Downloading
3. Download successfully or failed

We can output related info in the log. That's a simple way.
But I think if we need to supply an interface or command to check the record of downloading.

For example, we can process dfclient list records.

{
      "ImageName": "nginx",
      "ImageID": "4sda43rsdfsda",
      "tag": "latest",
      "StartedAt": 2019-09-19,
      "UpdatedAt": 2019-09-20,
}

@yeya24 @starnop @inoc603 WDYT?

[starnop]

also cc/ @lowzj

@fengzixu Thanks for your good idea.
However, dfdaemon is just a proxy to hijack the http request form container engines such as docker, containerd, etc. So it's hard for dfdaemon to get the complete info of the image.

In addition, dfdaemon is not only a proxy for the image download requests but also for file download requests actually.

[fengzixu]

So adding some output to illustrate status of downloading images in logs is an available solution. Is it right? @starnop

[yeya24]

So adding some output to illustrate status of downloading images in logs is an available solution. Is it right? @starnop

You mean output this information to the stdout of dfdaemon? I don't think it is readable. When there are multiple dfget requests.

[starnop]

So adding some output to illustrate status of downloading images in logs is an available solution. Is it right? @starnop

In fact, there isn't a good idea to solve this problem for me at present. And I will try to find a better solution. And please share your solution here if you have any ideas. @fengzixu @yeya24 @inoc603 @lowzj

[fengzixu]

So adding some output to illustrate status of downloading images in logs is an available solution. Is it right? @starnop

You mean output this information to the stdout of dfdaemon? I don't think it is readable. When there are multiple dfget requests.

I'm not sure. I think we need to find a way to illustrate the status of downloading images. For end-users, we can let them check images downloaded by Dragonfly through docker images. It means that we need to revise the document of QuickStart. But I don't think that is a good approach. Because of checking image status needs to cross two tools. That is inconvenient.

For developers of Dragonfly, illustrating the status of every downloading image is necessary. So we need to think about it more.

[yeya24]

As we have mentioned about combining dfdaemon and dfget to one single binary, I think it is OK to add some apis in dfdaemon later.

[starnop]

@fengzixu #926 has solved this issue actually.

For developers of Dragonfly, illustrating the status of every downloading image is necessary. So we need to think about it more.

I totally agree with you. However, as mentioned above, it's almost impossible to get the image download status for Dragonfly itself. And IMOP, we can make it clear using third tools like opentracing.

In addition, dfget will report the metrics to supernode when it success to download a file. So we can get something from supernode:

• check the log /home/admin/supernode/logs/dfget.log
• check the metrics form supernode API

[fengzixu]

@starnop Just need to revise our doc of QuickStart. :) Let me modify it.

[yeya24]

@fengzixu #926 has solved this issue actually.

For developers of Dragonfly, illustrating the status of every downloading image is necessary. So we need to think about it more.

I totally agree with you. However, as mentioned above, it's almost impossible to get the image download status for Dragonfly itself. And IMOP, we can make it clear using third tools like opentracing.

In addition, dfget will report the metrics to supernode when it success to download a file. So we can get something from supernode:

• check the log /home/admin/supernode/logs/dfget.log
• check the metrics form supernode API

This reminds me about the docs of monitoring dfget. I haven't add it yet. In order to see the logs , supernode must start in debug mode(-D)

[inoc603]

@fengzixu will the analyze sub command I proposed in #806 meet your needs?

[inoc603]

If I remember correctly, dfdaemon only pass layer data to docker daemon when the whole layer is downloaded. When using docker pull, progress bar for each layer either stops at zero, or reach 100% in an instant. If we implement some kind of streaming mechanism, it will be more clear that the image is being downloaded.

[starnop]

If we implement some kind of streaming mechanism, it will be more clear that the image is being downloaded.

I totally agree with you. In addition, we will not need to write the file to disk and read it again to dfdaemon, which will reduce the IO pressure of peer node.