Last call of @info seems not to be flushed

[xukai92]

The last call of @info seems not to be flushed. Is this tested or I did something wrong?

[PhilipVinc]

Do you have an example?
It should flush after every event is serialized.

It's not tested, though.
I should test it, that's true.

[xukai92]

The main codes are

@showprogress "Distilling" for i in 1:n_lessons
        s = score(knowledge)
        gs = Tracker.gradient(() -> s, ps)
        Tracker.update!(opt, ps, gs)
        @info "train" score=Flux.data(s)
    end
    
acc = mean(onecold(ds.student(get_Z(ds, knowledge))) .== onecold(get_probability(knowledge)))
@info "train" finish_msg="Distillation with $acc towards knowledge." log_step_increment=0

The last @info never shows in my TFBoard. Is it related to log_step_increment=0 I set?

[PhilipVinc]

Hmm. I cannot reproduce this on MacOs with tensor board 1.10.0.

Could you check that it is not TensorBoard that is not refreshing for some reason?
Simply start TensorBoard after you have logged that last message. Do you see it?

Alternatively, if you checkout the branch read_serialized you can use the (un-exported) command map_events.
It's not really well documented and tested yet, but you should be able to extract the last event and see what's inside.

[PhilipVinc]

You could try running the following

tbl = TBLogger()
# do your stuff

events = []
TensorBoardLogger.map_events(x->push!(events,x), TensorBoardLogger.logdir(tbl))
display(last(events))
display(last(events).summary.value)

what's the output?

[xukai92]

Simply start TensorBoard after you have logged that last message. Do you see it?

In fact this works! What exactly happens here?

[PhilipVinc]

Eh. I have no idea.
I have noticed that sometimes TensorBoard does not refresh (even if you push the refresh button).
I only saw it happening when you log subsequent messages with the same step.

It's possible that we do something that tensor board does not like.
But if they don't document how we should talk to TB, I can only take guesses.

[xukai92]

All right. I guess I will close this issue for now in thie case.

[PhilipVinc]

By the way, this seems known.
Seems it was fixed in v1.15