Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize tag query by avoiding multi-query #223

Merged
merged 2 commits into from
Apr 18, 2023

Conversation

Arkatufus
Copy link
Contributor

@Arkatufus Arkatufus commented Apr 17, 2023

Changes

Use SQL built-in aggregate string to retrieve journal row and tags in 1 go

Latest dev Benchmarks

Tag Count TagMode Mean Error StdDev Gen0 Gen1 Allocated
10 Csv 1,712.480 ms 22.7837 ms 21.3119 ms - - 390.94 KB
100 Csv 1,720.164 ms 21.3537 ms 19.9743 ms - - 1368.78 KB
1000 Csv 1,763.240 ms 30.8206 ms 28.8296 ms 1000.0000 - 10675.73 KB
10000 Csv 1,887.153 ms 35.8259 ms 35.1858 ms 12000.0000 3000.0000 103176.36 KB
10 TagTable 3.262 ms 0.0645 ms 0.1096 ms 42.9688 3.9063 355.72 KB
100 TagTable 4.935 ms 0.0969 ms 0.1420 ms 156.2500 31.2500 1307.93 KB
1000 TagTable 24.775 ms 0.4383 ms 0.3885 ms 1312.5000 656.2500 10923.02 KB
10000 TagTable 418.739 ms 5.3480 ms 5.0025 ms 12000.0000 3000.0000 106984.68 KB

This PR's Benchmarks

Tag Count TagMode Mean Error StdDev Gen0 Gen1 Gen2 Allocated
10 Csv 1,760.393 ms 27.1970 ms 25.4401 ms - - - 449.35 KB
100 Csv 1,766.355 ms 25.0182 ms 23.4021 ms - - - 1368.33 KB
1000 Csv 1,755.960 ms 33.8171 ms 34.7276 ms 1000.0000 - - 10733.67 KB
10000 Csv 1,905.026 ms 22.3564 ms 20.9122 ms 12000.0000 3000.0000 - 103175.1 KB
10 TagTable 2.336 ms 0.0389 ms 0.0344 ms 39.0625 3.9063 - 341.51 KB
100 TagTable 3.943 ms 0.0705 ms 0.0660 ms 148.4375 31.2500 - 1262.26 KB
1000 TagTable 18.597 ms 0.3570 ms 0.3506 ms 1281.2500 718.7500 - 10481.17 KB
10000 TagTable 184.446 ms 3.3447 ms 2.9650 ms 14000.0000 3666.6667 1666.6667 103154.93 KB

@Arkatufus
Copy link
Contributor Author

Another optimization I tried is by passing the original journal query into the tag query like so:

var tagRows = await connection
    .GetTable<JournalTagRow>()
    .Where(r => r.OrderingId.In(rowQuery.Select(jr => jr.Ordering)))
    .Select(r => new TagRow
    {
        OrderingId = r.OrderingId,
        TagValue = r.TagValue
    })
    .ToListAsync();

This actually degrades the performance:

Tag Count TagMode Mean Error StdDev Gen0 Gen1 Gen2 Allocated
10 Csv 1,752.93 ms 18.677 ms 15.596 ms - - - 450.28 KB
100 Csv 1,722.95 ms 11.452 ms 8.941 ms - - - 1368.69 KB
1000 Csv 1,726.12 ms 9.396 ms 8.789 ms 1000.0000 - - 10734.09 KB
10000 Csv 1,895.89 ms 20.333 ms 19.020 ms 12000.0000 3000.0000 - 103175 KB
10 TagTable 20.60 ms 0.208 ms 0.194 ms 31.2500 - - 379.39 KB
100 TagTable 23.35 ms 0.466 ms 0.518 ms 156.2500 31.2500 - 1320.66 KB
1000 TagTable 43.39 ms 0.850 ms 0.795 ms 1250.0000 666.6667 - 10836.95 KB
10000 TagTable 451.57 ms 7.244 ms 6.049 ms 13000.0000 4000.0000 1000.0000 105942.11 KB

row = jr,
tags = tagTable
.Where(r => r.OrderingId == jr.Ordering)
.StringAggregate(";", r => r.TagValue)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use SQL built-in string aggregate function to retrieve all tags in one go

@Aaronontheweb
Copy link
Member

@Arkatufus looks like this broke something across several Db implementations

@Arkatufus
Copy link
Contributor Author

@Arkatufus looks like this broke something across several Db implementations

Should be fixed now

Copy link
Member

@Aaronontheweb Aaronontheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb Aaronontheweb merged commit 35eca99 into akkadotnet:dev Apr 18, 2023
@Arkatufus Arkatufus self-assigned this Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

2 participants