Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created a new configuration entry to enable saving keychains charms to the local DB. #227

Closed
wants to merge 7 commits into from

Conversation

wangxingzhen
Copy link

#226 (review)

Support caching Charms to local DB via configuration options

@gaelsuv
Copy link

gaelsuv commented Oct 19, 2024

Can you demonstrate how the charms cache is working with the new config parameter set to true? I've tried inspecting the same charm multiple times and the bot is fetching it again and again from the GC, not the underlying db. I know the reason, but I'm curious to see if you observe the same behavior or not.

@wangxingzhen
Copy link
Author

Can you demonstrate how the charms cache is working with the new config parameter set to true? I've tried inspecting the same charm multiple times and the bot is fetching it again and again from the GC, not the underlying db. I know the reason, but I'm curious to see if you observe the same behavior or not.

I just tested it multiple times and everything is working correctly; data is being fetched from the database. Here are the steps I followed: I executed the command gh pr checkout 227 to check out the current PR code, then modified config.js to set cache_charms to true. If you have any other steps or environment settings, please share them, and we can troubleshoot together.

When I first called http://127.0.0.1:9160/?url=steam://rungame/730/76561202255233023/+csgo_econ_action_preview%20S76561198291921905A39944764900D5098182160541447191, PostgreSQL generated this data:
WX20241020-105620@2x

Pay attention to the keychains where i is currently 26. I changed it to 25 in PostgreSQL, and then when I called http://127.0.0.1:9160/?url=steam://rungame/730/76561202255233023/+csgo_econ_action_preview%20S76561198291921905A39944764900D5098182160541447191, the value of sticker_id changed to 25:
2
3

This is the main code change in this PR. Please check if your code matches mine.
4
5
6
7

@gaelsuv
Copy link

gaelsuv commented Oct 20, 2024

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?

Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.

The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.

Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both?
SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);

Thanks again for your help.

@wangxingzhen
Copy link
Author

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?

Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.

The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.

Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);

Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

@wangxingzhen
Copy link
Author

After careful consideration and testing, I have decided to close this PR. The issue with caching charms stems from the unique index (i_unique_item), and I believe a more comprehensive approach is needed.

@gaelsuv
Copy link

gaelsuv commented Oct 20, 2024

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.

Thanks!

@wangxingzhen
Copy link
Author

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.

Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

@gaelsuv
Copy link

gaelsuv commented Oct 21, 2024

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.

At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)?
I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

@wangxingzhen
Copy link
Author

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.

At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)? I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

Although I don't know the exact reason, here's my interpretation: The (defindex, paintindex, paintwear, paintseed) combination never changes, whereas the "a" field can change frequently, such as when applying or scraping stickers, or when transferring an item to another user. I believe this is why the author chose to use these attributes for the unique index rather than the "a" column, to maintain consistency in identifying the item.

@gaelsuv
Copy link

gaelsuv commented Oct 21, 2024

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.
At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)? I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

Although I don't know the exact reason, here's my interpretation: The (defindex, paintindex, paintwear, paintseed) combination never changes, whereas the "a" field can change frequently, such as when applying or scraping stickers, or when transferring an item to another user. I believe this is why the author chose to use these attributes for the unique index rather than the "a" column, to maintain consistency in identifying the item.

The field changing when the item is transferred is not A if I'm not wrong, but rather S, which in the items table happens to be the value stored as "ms" (it is also the attribute whose changes are being tracked -together with the price- under the history table by the "SCD-like" trigger).
Are you sure the "a" field is changing when applying/scraping stickers? Didn't know about that.

@wangxingzhen
Copy link
Author

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.
At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)? I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

Although I don't know the exact reason, here's my interpretation: The (defindex, paintindex, paintwear, paintseed) combination never changes, whereas the "a" field can change frequently, such as when applying or scraping stickers, or when transferring an item to another user. I believe this is why the author chose to use these attributes for the unique index rather than the "a" column, to maintain consistency in identifying the item.

The field changing when the item is transferred is not A if I'm not wrong, but rather S, which in the items table happens to be the value stored as "ms" (it is also the attribute whose changes are being tracked -together with the price- under the history table by the "SCD-like" trigger). Are you sure the "a" field is changing when applying/scraping stickers? Didn't know about that.

Yes, I'm sure. The "a" field is actually the asset_id. When you apply or scrape stickers, the asset_id changes. Additionally, when an item is gifted or transferred to another user, both the asset_id and the steam64_id (which corresponds to the "s" field) change.

@gaelsuv
Copy link

gaelsuv commented Oct 21, 2024

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.
At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)? I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

Although I don't know the exact reason, here's my interpretation: The (defindex, paintindex, paintwear, paintseed) combination never changes, whereas the "a" field can change frequently, such as when applying or scraping stickers, or when transferring an item to another user. I believe this is why the author chose to use these attributes for the unique index rather than the "a" column, to maintain consistency in identifying the item.

The field changing when the item is transferred is not A if I'm not wrong, but rather S, which in the items table happens to be the value stored as "ms" (it is also the attribute whose changes are being tracked -together with the price- under the history table by the "SCD-like" trigger). Are you sure the "a" field is changing when applying/scraping stickers? Didn't know about that.

Yes, I'm sure. The "a" field is actually the asset_id. When you apply or scrape stickers, the asset_id changes. Additionally, when an item is gifted or transferred to another user, both the asset_id and the steam64_id (which corresponds to the "s" field) change.

Ok, that explains why the merge logic is using the surrogate key (defindex, paintindex, paintwear, paintseed): it is the set of the only attributes that can never change for a CS item.
That said, since charms do not have a paintwear (and neither a paintseed nor a paintindex), we're effectively talking about commodities. In other terms, there will be n instances of the same charm (e.g.: Big Kev) with the same pattern (82034) which are truly the same object. The only thing that differentiates them is the inventory they are in.

Is it right?

@wangxingzhen
Copy link
Author

Thanks for the in-depth analysis. Did you also try to inspect a new different charm after this one in the test above (a=39944764900)?
Asking because I can confirm that even in my environment (I had a fresh pull from the last commit on your repo and I created a new local postgres db: https://github.com/wangxingzhen/csgo_inspect/tree/cache_charms) the first charm being inspected is successfully cached and retrieved from the db on subsequent requests.
The problem I have arises as soon as I inspect a new different charm, which is not being saved into the items table because its incremental/update key (defindex, paintindex, paintwear, paintseed) is exactly the same of the previously cached charms.
Could you please show me whether indeed in your items table, after inspecting two different charms, you can find them both? SELECT * FROM <your_schema>.items WHERE a IN (<charm1 a param>, <charm2 a param>);
Thanks again for your help.

Thank you for your insights. After testing, I found that the issue arises from the unique index (i_unique_item). I believe there is still a lot of work to do, and simply adding caching for charms is not the right approach. I will close this PR for now, and I hope we can collaborate on finding a better solution to avoid this issue in the future.

Actually addressing the issue should be pretty easy. I'll leave here two different approaches we can follow to properly handle charms cache in a future PR:

  1. Unique index expansion: since we're already storing the keychains array into the jsonb dedicated new column, a basic working solution should consist in adding the keychains.pattern value to the unique index (keychains->0->>'pattern'). Same should be done for the ON CONFLICT clause of the INSERT query.
  2. Since the pattern is for charms what the paintseed is for skins, we can store the pattern into the paintseed column. This should be enough in order to not have to modify the unique index.

I would be happy to contribute directly to the PR, but not being particularly familiar with JS, I will just leave you these two suggestions here hoping they will be useful.
Thanks!

Thank you for your suggestions! However, both approaches might not be suitable because the table also stores steam64_id, asset_id, and d. It’s quite likely that different users could have charms with the same pattern, which would lead to conflicts. We need to consider a more robust solution to handle this scenario.

You are completely right, sorry for the wrong indications.
At this point though I have a genuine question: why has the ON CONFLICT clause been originally set to (defindex, paintindex, paintwear, paintseed)? I understand that tuple, before charms were a thing, uniquely identified a given item and this makes it a totally valid surrogate key for the incremental/update logic. But why not using the "a" column directly i.e. the natural key of the CS items? Do we have inspectable items in CS for which one or more of those 4 attributes can change over time? If not, I don't see the reason yet to use the tuple instead of the primary key directly for merge operations.

Although I don't know the exact reason, here's my interpretation: The (defindex, paintindex, paintwear, paintseed) combination never changes, whereas the "a" field can change frequently, such as when applying or scraping stickers, or when transferring an item to another user. I believe this is why the author chose to use these attributes for the unique index rather than the "a" column, to maintain consistency in identifying the item.

The field changing when the item is transferred is not A if I'm not wrong, but rather S, which in the items table happens to be the value stored as "ms" (it is also the attribute whose changes are being tracked -together with the price- under the history table by the "SCD-like" trigger). Are you sure the "a" field is changing when applying/scraping stickers? Didn't know about that.

Yes, I'm sure. The "a" field is actually the asset_id. When you apply or scrape stickers, the asset_id changes. Additionally, when an item is gifted or transferred to another user, both the asset_id and the steam64_id (which corresponds to the "s" field) change.

Ok, that explains why the merge logic is using the surrogate key (defindex, paintindex, paintwear, paintseed): it is the set of the only attributes that can never change for a CS item. That said, since charms do not have a paintwear (and neither a paintseed nor a paintindex), we're effectively talking about commodities. In other terms, there will be n instances of the same charm (e.g.: Big Kev) with the same pattern (82034) which are truly the same object. The only thing that differentiates them is the inventory they are in.

Is it right?

yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants