Cannot Reproduce the MOT17 Scores #1156
Comments
emirhanbayar
changed the title
|
Thank you for sharing these experiment results @emirhanbayar
I have not checked the config parameters for StrongSORT in a long time, I suggest you to start there
According to my OSNet experiments |
|
👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. |
How did you train to get your own weight file? |
|
I did not train anything. If you mean the reID weight file (osnet_x0_25_msmt17.pt). I chose it from the ones that are listed at: https://github.com/mikel-brostrom/yolo_tracking/blob/master/boxmot/appearance/reid_model_factory.py |
|
First of all. Which detector are you using? |
|
MOT17 train detections from: https://drive.google.com/drive/folders/1zzzUROXYXt8NjxO1WUcwSzqD-nn7rPNr The authors say that they used YOLOX. |
Ok. Then you are using the right detections 😄.
|
I had recorded the following results: I newly ran the code with the following parameters: And got the following result:
No. |
|
Have you tried: https://drive.google.com/drive/folders/1wdG-vJOMGynf5QjqEa1jNZNYtz0PQqPu ? Check this: dyhBUPT/StrongSORT#74 |
I had already modified the ground truth accordingly. Now, I downloaded them from the link that you shared. The result is exactly the same. An update: I fed both detections and features that are shared by the authors of StrongSORT as input and kept the same parameters as in your implementation. On the MOT17 train, I got: HOTA: 63.105 Changed parameters as in the original implemenataion of StrongSORT: Then, on the MOT17 train, I got: HOTA: 65.353 I am not sure. Maybe, I am making a mistake. You can inspect how I modified your implementation by looking at my fork at: https://github.com/emirhanbayar/yolo_tracking To run the code:
|
|
Great! We are getting closer... 😄
You can now try |
Actually, hota 65.4 is obtained by feeding reid features from the ones that pre-extracted and shared by authors of StrongSORT. Neither detector nor thr reID model is used from boxmot. |
|
I am unfamiliar with this data
Are the features normalized? |
|
Sorry for the late reply. The entries are generally between -2 and 4. Some examples of distributions of the feature vectors from the detections from MOT17 sequence 2:
You can see the distributions for each of the detections at: https://drive.google.com/file/d/1Ef8zms1Ph5JOn5kRjpRYc7kdYHIe3lCw/view?usp=sharing |
|
I just noticed that the MOTA score (66.354) you get is quite low compared to the reported one (77.0). This could be due to some misconfiguration in the Kalman Fillter. You could try importing the official one and that one instead. |
|
Aaand you need to normalize the embeddings for better IDF1 results 😄 |
|
Hi,guys,I have partially addressed your issue by setting "track.time_since_update" of "strong_sort" to >2, resulting in an HOTA of 68.184 on MOT17. The deviation from the results in the paper is less than 1, indicating that there is still room for optimization. The reason why adjusting "time_since_update" proves to be effective in enhancing HOTA is currently under exploration. If you have any findings on this matter, I would appreciate the opportunity for further discussion. |
You have to comment out the part from if tlwh[0] < 0: to tlwh[1] = 0, I'm not sure why you want to do this, but no doubt this reduces HOTA |
|
yeah, that's it, change >= to > |
|
Just intuition here. Have not tested any of this. I believe this will cause to output the last detected bbox for 2 frames more. So when a track fails in associating with a detection, the last detected bbox will be output'ed anyways instead of not giving an output at all. Could be beneficial for when the track is lost for a few frames. Let me know if you experiment with this idea! |
|
Your intuition is correct that increasing this number will make the output more tracks, but in fact, if you look at strongsort's results, you'll see that the optimal HOTA output txt file size is larger than the boxmot output txt file size, in other words, strongsort's optimal result produces more and longer tracks. In my tests, setting this number to 1 to 3 would produce the same HOTA, 68.184, but the output trajectory would be different, i.e., the larger the number, the more the trajectory. I think understanding the meaning of this number would help improve strongsort and other trackers in boxmot
…---Original---
From: ***@***.***>
Date: Fri, Dec 15, 2023 22:32 PM
To: ***@***.***>;
Cc: ***@***.******@***.***>;
Subject: Re: [mikel-brostrom/yolo_tracking] Cannot Reproduce the MOT17 Scores(Issue #1156)
I believe this will cause to output the last detected bbox for 2 frames more. So when a track fails in associating with a detection, the last detected bbox will be output'ed anyways instead of not giving an output at all. Could be beneficial for when the track is lost for a few frames.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
In my tests, the optimal result for this number should be 1 to 3(> instead of >=), and the meaning of this number should be the maximum number of frames the track has not been updated. Now, I think we should look at other parameters to determine why boxmot's HOTA is less than the paper value, after all, it would be very strange to use strongsort's detection and features directly to produce different results.
ps: Instead of ecc, I saved the ecc results from boxmot and ran them in strongsort, producing similar results
…---Original---
From: ***@***.***>
Date: Fri, Dec 15, 2023 22:32 PM
To: ***@***.***>;
Cc: ***@***.******@***.***>;
Subject: Re: [mikel-brostrom/yolo_tracking] Cannot Reproduce the MOT17 Scores(Issue #1156)
I believe this will cause to output the last detected bbox for 2 frames more. So when a track fails in associating with a detection, the last detected bbox will be output'ed anyways instead of not giving an output at all. Could be beneficial for when the track is lost for a few frames.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
I believe the remaining performance increase can be easily obtained by fine-tuning. |
|
I have also noticed that modifying For making predictions ahead some modifications are needed |
|
👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. |
Search before asking
Question
Hello,
I encounter an inconsistency between your implementation's HOTA scores and those stated in the paper of StrongSORT.
Setup:
Tracker: StrongSORT (did not change any parameter)
Detections: from https://drive.google.com/drive/folders/1Zk6TaSJPbpnqbz1w4kfhkKFCEzQbjfp_?usp=sharing
Re-ID: osnet_x0_25_msmt17.pt
HOTA scores after running your implementation:
MOT-17-Train: 63.8
MOT-17-Test: 59.69
HOTA scores claimed in the original paper:
MOT-17-Train: 68.2 (StrongSORT v4 in Table 1)
MOT-17-Test: 63.5 (StrongSORT online in Table 4)
I suspected this results from using a different Re-ID and switched to using features provided by the authors of StrongSORT. However, this even reduced the HOTA score on the MOT-17-Train.
Do you have any idea why this is happening? I would appreciate any comment on this.
Thanks in advance.
The text was updated successfully, but these errors were encountered: