Spellbook migration to DuneSQL is complete 👏

[jeff-dude]

a massive thank you to all the wizards who stepped in to help migrate spells without us asking, it's much appreciated 🙏
also, thank you to all for the patience, as we worked through the large effort to migrate spellbook to the DuneSQL engine.

what steps have been taken and/or will be taken to finalize migration?

• turned off production orchestration on spark engine
• removed all spark steps in the CI tests attached to each PR
• backed up repo to preserve legacy scripts, for emergency usage only
• dune team will remove all legacy files from the repo
• dune team will remove any spark-specific code in macros
• dune team will remove all dunesql tags
• dune team will remove code which requires usage of alias() in alias property & remove from all existing spells
• dune team will remove references in the readme that are tied only to the migration effort

any other info forgotten here will be tracked in comments below.

what does this mean for wizard contributions moving forward?

• ensure all branches are up-to-date and in sync with dune main branch
• no more modifications to legacy files
• no more dummy legacy files on new spells
• once the dune team removes all dunesql tag and alias() functions on existing code, new spells will no longer require this -- for now, continue to use, we will track in comments when we can stop

what happens next?

• continue building spells, we will continue to help push along and push into production 🤝
• dune team will shift focus to two main areas over the next few weeks/months:
  • building evm chain transfers & balances spells -- we're starting fresh and building together with help from the community
    • join us in conversation and track here: Migrate and fix token transfers #4521 EVM Balances main issue #4523
  • redesigning major sector-level spells to be more efficient in orchestration, less reliant on metadata at all levels, leveraging macros for repeatable code, new and/or modified columns and plenty more
    • join us in conversation and track here: [proposal] sector-level spells redesign #4637
  • improving our production orchestration

[jeff-dude]

one minor caveat:
there was a small fraction of spells which were not migrated. the majority of these had little-to-no usage on the dune app in recent queries. one exception is the transfers/balances spells, which we didn't migrate directly, as the ongoing effort to rebuild linked above is happening.

if anyone comes across a spell they'd expect on dunesql, but isn't, please do reach out here and we can look to help.

[Sean-Cartwright]

The priority_fee_per_gas variable (spell is it called?) is not in the latest version of Dune, so I cannot pull the block rewards for a query I want to run. Is this an example of this issue?

[jeff-dude]

can you be a bit more specific? what is the exact table name (and link to spell, if you can)?

[jeff-dude]

• dune team will remove all legacy files from the repo
• dune team will remove any spark-specific code in macros

we have taken the steps to remove legacy files/macros/any references to legacy.
PR here: #4648

we will monitor to ensure no failures and/or side effects, but if anything comes up, tag us here 🙏

[jeff-dude]

• dune team will remove all dunesql tags
• dune team will remove code which requires usage of alias() in alias property & remove from all existing spells

these are now both complete and we are monitoring both CI runs & production to ensure all is smooth.
PR here: #4663

if anything comes up, plz tag us here

[jeff-dude]

fyi @0xRobin @Hosuke
we no longer need tags for dunesql or alias() function wrapped around the spell name in the alias config

[jeff-dude]

• dune team will remove references in the readme that are tied only to the migration effort

PR here: #4676

[Sean-Cartwright]

Thanks for the help, Looking now, it seems like it has already been added back to the new system. It was in ethereum.transactions Here is an example of an old query with it Ethereum Block Rewards & Fee Summary - Last 15 Days (dune.com) <https://dune.com/queries/834270/1458638>: with eth_tx as (select block_time, block_number, nonce, gas_used, (gas_price-coalesce(*priority_fee_per_gas*,0))/1e9 base_fee_gwei, coalesce(priority_fee_per_gas,0)/1e9 *priority_fee_per_gas_gwei* from ethereum.transactions I should be able to do my query now, thanks. - R Sean Cartwright

…

On Mon, Nov 6, 2023 at 2:07 PM jeff ***@***.***> wrote: can you be a bit more specific? what is the exact table name (and link to spell, if you can)? — Reply to this email directly, view it on GitHub <#4662 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDVQ4RWZ6UGPNWODLEX2RQTYDE7RDAVCNFSM6AAAAAA6MSI5RWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIOJRGY2TG> . You are receiving this because you commented.Message ID: ***@***.*** com>

[jeff-dude]

got it! glad it has been resolved.

for future reference, ethereum.transactions isn't a spell, but rather a raw base table. if you find issues in these tables, we'll need to raise in discord support instead 👍