Query: Deploying hello world to solana-test-validator on M1 fails #18033

fdeantoni · 2021-06-17T03:37:01Z

Problem

I'm running Solana master with the example Hello World program and on OSX x86 things run as expected. I can deploy the hello world program to the test validator. However, when doing the exact same on M1, it fails at the "Finalize transaction" step:

$ solana program deploy dist/program/helloworld.so
===================================================================
Recover the intermediate account's ephemeral keypair file with
`solana-keygen recover` and the following 12-word seed phrase:
===================================================================
spring kick mail one box meat august educate car ghost orient woman
===================================================================
To resume a deploy, pass the recovered keypair as
the [PROGRAM_ADDRESS_SIGNER] argument to `solana deploy` or
as the [BUFFER_SIGNER] to `solana program deploy` or `solana write-buffer'.
Or to recover the account's lamports, pass it as the
[BUFFER_ACCOUNT_ADDRESS] argument to `solana program close`.
===================================================================
Error: Deploying program failed: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I was hoping if anybody would have a hint where I should look to try and resolve this.

This is what I did to get my environment started:

Set up BPF SDK

$ git clone https://github.com/solana-labs/solana
$ export PROJECT_ROOT=`pwd`/solana
$ cd $PROJECT_ROOT/sdk/cargo-build-bpf
$ cargo install --path .
$ cd $PROJECT_ROOT
$ ln -s $PROJECT_ROOT/sdk ~/.cargo/bin/sdk

Build the CLI and run configure

$ cd $PROJECT_ROOT/cli
$ cargo build
$ cd $PROJECT_ROOT/keygen
$ cargo build
$ $PROJECT_ROOT/target/debug/solana  config set --url localhost
$ $PROJECT_ROOT/target/debug/solana-keygen new

Run the test validator

$ cd $PROJECT_ROOT/validator
$ ./solana-test-validator

Build hello world
Edit the example-helloworld/src/program-rust/Cargo.toml to match $PROJECT_ROOT deps, and then:

$ git clone https://github.com/solana-labs/example-helloworld
$ cd example-helloworld
$ npm install
$ npm run build:program-rust

Everything compiles ok and seems to run fine except for the "Finalize transaction" part which fails on M1. In the validator.log I see no errors, only a warning:

WARN  solana_core::window_service] Window does not seem to be receiving data. Ensure port configuration is correct...

I do not see this warning when running on OSX x86.

Any hints where I should look to try and fix this?

Proposed Solution

None - just a query.

The text was updated successfully, but these errors were encountered:

neildotvn · 2021-08-24T03:20:52Z

Any update on this? I have the same problem.

gxxcastillo · 2021-08-26T04:11:33Z

Similar issue, I'm on a Mac with an M1 Chip. I've tried different demos and all of them have the same issue when I try to deploy my program. Basically it just gets stuck at [0/1] Finalizing transaction...

validator.logs has this being called repeatedly: solana_rpc::send_transaction_service] Retrying transaction: when I grep using my transaction id.

I've tried building + deploying with my terminal running Rosetta and run into the same problem regardless, so not sure if this is related.

gxxcastillo · 2021-08-29T07:26:26Z

I found my issue: I had installed almost everything under Rosetta. Checking my rustup toolchain via rustup toolchain list I discovered I had somehow managed to install rust for arm64 instead of x86. After reinstalling everything, I've confirmed my default toolchain is now stable-x86_64-apple-darwin and everything is now building 😃😌

monomadic · 2021-08-31T06:51:05Z

@gxxcastillo okay but that isn't really fixing the issue it's avoiding it. I'm getting the same issue on M1 - funny thing is, it was deploying perfectly earlier - but I can't confirm if this was the x86 version.

I get the feeling this occurs when part of your toolchain is x86 and some is arm, but can't confirm yet. I'll keep digging.

Qumeric · 2021-09-04T19:11:51Z

I also use M1 mac and I have

Error: Account address has insufficient funds for spend (0.4265436 SOL) + fee (0.001355 SOL)

Does it look like the same issue? 🤔

Although npm run build:program-c causes the same error.

gxxcastillo · 2021-09-04T19:40:16Z

@Qumeric, make sure 1) You're connected to the right cluster and 2) That you've got enough SOL in your account

Qumeric · 2021-09-05T11:58:17Z

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

artursapek · 2021-09-05T17:50:22Z

pretty sad the fastest blockchain doesn't work on the fastest laptops :(

Krishna-Psftware · 2021-09-22T04:55:26Z

Similar issue even am facing while requesting airdrop

Error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

Krishna-Psftware · 2021-09-22T06:35:12Z

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Krishna-Psftware · 2021-09-22T06:36:01Z

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.

(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Did you resolve this issue?

crmanik · 2021-09-27T04:06:42Z

Hello, I tried following:

installed rust in rosetta terminal.
compiled Solana tools from source
Able to start Solana-test-validator
Confirmed endpoint is right.

But still I get this damn error:

error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I'm really stuck, any pointers are how to move forward?

dcolley · 2021-09-30T08:59:27Z

There is a Docker image dedicated to mac-m1 - https://github.com/metaspan/solana-docker-mac-m1
I have the same problem and my logs show the following:

[2021-09-30T08:56:11.948913055Z INFO  solana_rpc::send_transaction_service] Retrying transaction: 41yweJXm4inkN7BZ5TfwyHV3FUYaZ84YMmMp386BNKgSnLtn9Jiha8YCABNsfJjoDSnyYYAkZLMJdZ8gAUx41aC
thread 'solana-banking-stage-tx' panicked at 'JIT is only supported on x86_64', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/solana_rbpf-0.2.11/src/jit.rs:738:13
stack backtrace:
   0: std::panicking::begin_panic
   1: solana_rbpf::jit::JitCompiler::new
   2: solana_rbpf::jit::JitProgram<E,I>::new
   3: <solana_rbpf::elf::EBpfElf<E,I> as solana_rbpf::vm::Executable<E,I>>::jit_compile
   4: solana_bpf_loader_program::create_executor
   5: solana_bpf_loader_program::process_loader_upgradeable_instruction
   6: solana_bpf_loader_program::process_instruction_common
   7: solana_runtime::message_processor::MessageProcessor::process_instruction
   8: solana_runtime::message_processor::MessageProcessor::process_message
   9: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
  10: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
  11: solana_runtime::bank::Bank::load_and_execute_transactions
  12: solana_core::banking_stage::BankingStage::process_and_record_transactions
  13: solana_core::banking_stage::BankingStage::process_packets_transactions
  14: solana_core::banking_stage::BankingStage::process_packets
  15: solana_core::banking_stage::BankingStage::process_loop

dcolley · 2021-09-30T11:29:35Z

Seems this is linked: qmonnet/rbpf#48

codenjobs · 2021-10-06T20:34:51Z

If you are ok to use the terminal with Intel architecture, you can refer to this blog post How to make Solana Test Validator work with a Macbook with M1 chip instead.

amalcaraz · 2021-11-07T15:40:28Z

I have the same issue with an M1 mac and found a workaround starting the validator with the "--bpf-program" option.
Not doing the deployment process, but at least I could have the program in my testnet and keep working with it

solana-test-validator --bpf-program ADDRESS target/deploy/PROGRAM.so

newhouseb · 2021-11-08T00:45:02Z

From perusing the discord, I've figured out what was up (as alluded to in other threads):

Adding --log to the test-validator command and watching the deploy, you'll see the following

[2021-11-08T00:40:59.691095000Z INFO  solana_core::heaviest_subtree_fork_choice] marking fork starting at: (4505, DAmaeMFkDVRaeyuD9HDfvCsy4JDbJDui1YHvrzMsH5Ao) valid candidate
thread 'solana-banking-stage-tx' panicked at 'JIT is only supported on x86_64', /Users/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/solana_rbpf-0.2.11/src/jit.rs:738:13
stack backtrace:

As you can see: the BPF JIT is not going to work on arm64. Fortunately you can run things without the JIT by adding --no-bpf-jit to the test-validator command and then you'll be good to go.

fanatid · 2021-11-09T07:48:03Z

#21215 should help to run test validator without explicit --no-bpf-jit flag.

newhouseb · 2021-11-15T17:45:38Z

Also heads up that as you try to get the hello world to actually work you'll run into further issues due to BPF parameter alignment as mentioned here: #21269

nphenley · 2021-11-19T00:33:44Z

I have found my issue. I built Solana from sources (because otherwise, I faced another M1 related issue) and used ./run.sh as it is stated in https://github.com/solana-labs/solana/blob/master/README.md. However, what helped me is to run ./solana/target/debug/solana-test-validator instead.
(Also I tried to do solana airdrop 100 previously but it stuck on [0/1] Finalizing transaction... as described in some previous reply)

Did you find a solution to this?

chihkaiyu · 2021-11-19T13:53:19Z

I build Solana tool suite from source and add --no-bpf-jit flag to solana-test-validator. Now at least I can deploy the example program to local validator.

Before that, I can't even run solana-test-validator. Using the Solana CLI which is built from source, the CLI works but then it's stuck when deploying the program.

$ solana program deploy dist/program/helloworld.so                                                                                                                                                          
                                                                                                      
===========================================================================                           
Recover the intermediate account's ephemeral keypair file with                                        
`solana-keygen recover` and the following 12-word seed phrase:                                        
===========================================================================                           
gas tiger plastic knife speak uncle monkey rebel boat suspect agree concert                           
===========================================================================                           
To resume a deploy, pass the recovered keypair as the                                                 
[BUFFER_SIGNER] to `solana program deploy` or `solana write-buffer'.                                  
Or to recover the account's lamports, pass it as the                                                  
[BUFFER_ACCOUNT_ADDRESS] argument to `solana program close`.                                          
===========================================================================                           
Error: Deploying program failed: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I resume the failed deployment but it still stuck and finally reach maximum retries.

$ solana program deploy --buffer ~/.config/solana/id.json ./dist/program/helloworld.so 
Blockhash expired. 5 retries remaining             
Blockhash expired. 4 retries remaining             
Blockhash expired. 3 retries remaining             
Blockhash expired. 2 retries remaining             
Blockhash expired. 1 retries remaining             
Error: Data writes to account failed: Custom error: Max retries exceeded

diginator · 2021-12-01T03:33:08Z

Did you get through this? $ solana program deploy and $anchor deplay are both getting stuck at:
Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded

Vkzem · 2021-12-11T07:19:33Z

Hello, I tried following:

installed rust in rosetta terminal.

compiled Solana tools from source

Able to start Solana-test-validator

Confirmed endpoint is right.

But still I get this damn error:

error: unable to confirm transaction. This can happen in situations such as transaction expiration and insufficient fee-payer funds

I'm really stuck, any pointers are how to move forward?

did you manage to solve this problem? also facing the same even when trying to do a devnet airdrop

diginator · 2021-12-11T16:09:25Z

So, I think might be a network error. I switched to a different network and since the devnet wallet is currently drained, I tried using testnet and it totally worked.
Btw, I tried using testnet earlier when was stuck at the deployment but it wouldn't work.
Changing the network connection is what got it working for me.

Vkzem · 2021-12-11T23:25:20Z

thanks mate, I think you're exactly right - plus looks like the devnet issue is resolved now, too

n0x1m · 2022-01-06T16:22:13Z

I have a reverse setup and similar issue on v1.9.2, I build on a Mac x86_64 and deploy to a Raspberry Pi test validator, it never succeeds but retries to send transactions and then times out, requiring me to close the buffer.

Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded

How can I debug this?

diginator · 2022-01-06T17:14:45Z

This is a network issue and you're probably facing this now because the Solana network is down. Check at status.solana.com and try again later. If it still happens, switch your network.

t-nelson · 2022-01-06T22:32:53Z

This is a network issue and you're probably facing this now because the Solana network is down. Check at status.solana.com and try again later. If it still happens, switch your network.

They're obviously running a private cluster. None of the public clusters will influence it, congestion or otherwise

I have a reverse setup and similar issue on v1.9.2, I build on a Mac x86_64 and deploy to a Raspberry Pi test validator, it never succeeds but retries to send transactions and then times out, requiring me to close the buffer.
Blockhash expired. 5 retries remaining
Blockhash expired. 4 retries remaining
Blockhash expired. 3 retries remaining
Blockhash expired. 2 retries remaining
Blockhash expired. 1 retries remaining
Error: Data writes to account failed: Custom error: Max retries exceeded
How can I debug this?

Have you tried to deploy from the rPi directly to rule out network configuration issues.

n0x1m · 2022-01-08T08:17:10Z

Have you tried to deploy from the rPi directly to rule out network configuration issues.

if I scp the binary to the rpi I can deploy it from there without issues. Networking is fine otherwise. It appears some issue with the tooling crossing the architectures (?) but I don't know how to debug this further.

dcolley · 2022-03-03T08:41:46Z

After upgrade to latest tools I have a different issue, and google leads me back here... (I forgot about this issue).

Running Ubuntu 18.04 arm64 in Docker on Mac M1

root@04598e084504:~/test# anchor --version
anchor-cli 0.22.1
root@04598e084504:~/test# rustc --version
rustc 1.59.0 (9d1b2106e 2022-02-23)
root@04598e084504:~/test# rustup --version
rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.59.0 (9d1b2106e 2022-02-23)`
root@04598e084504:~/test# solana --version
solana-cli 1.9.9 (src:devbuild; feat:3246413280)
root@04598e084504:~/test# yarn --version
1.22.17
root@04598e084504:~/test# uname -a
Linux 04598e084504 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

$ anchor init test
yarn install v1.22.17
warning package.json: No license field
info No lockfile found.
warning No license field
[1/4] Resolving packages...
warning @project-serum/anchor > @solana/web3.js > rpc-websockets > [email protected]: CircularJSON is in maintenance only, flatted is its successor.
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > [email protected]" has incorrect peer dependency "mocha@^3.X.X || ^4.X.X || ^5.X.X || ^6.X.X || ^7.X.X || ^8.X.X".
[4/4] Building fresh packages...
success Saved lockfile.
Done in 16.56s.
test initialized

$ cd test
$ anchor build
BPF SDK: /opt/solana-1.9.9/bin/sdk/bpf
cargo-build-bpf child: rustup toolchain list -v
cargo-build-bpf child: cargo +bpf build --target bpfel-unknown-unknown --release
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

More info:

# rustup toolchain list -v
stable-aarch64-unknown-linux-gnu (default)	/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu
bpf	/opt/solana-1.9.9/bin/sdk/bpf/dependencies/bpf-tools/rust

It seems related to the toolchain for aarch64

nathanleclaire · 2022-03-06T01:08:10Z

Same issue here, can't get test validator to successfully receive programs in docker container running on m1 mac. Native mac binary works fine.

nathanleclaire · 2022-04-18T21:30:45Z

Deploys are working fine for me now with solana-test-validator running in a container on M1 using this Dockerfile and this Docker run command.

shawnwang0715 · 2022-06-12T12:02:47Z

$ cd test
$ anchor build
BPF SDK: /opt/solana-1.9.9/bin/sdk/bpf
cargo-build-bpf child: rustup toolchain list -v
cargo-build-bpf child: cargo +bpf build --target bpfel-unknown-unknown --release
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

More info:

# rustup toolchain list -v
stable-aarch64-unknown-linux-gnu (default)	/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu
bpf	/opt/solana-1.9.9/bin/sdk/bpf/dependencies/bpf-tools/rust

It seems related to the toolchain for aarch64

@dcolley I have the same issue of qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory in docker on M1 Mac. Did you find out the solution?

dcolley · 2022-06-12T14:21:14Z

@shawnwang0715 no, I switched to ubuntu on x64 instead

shawnwang0715 · 2022-06-12T14:45:48Z

@shawnwang0715 no, I switched to ubuntu on x64 instead

Could you please describe how do you build on ubuntu x64? I used your dockerfile and tried to run cargo build-bpf inside the container but still failed. More details here: metaspan/solana-docker-mac-m1#4

dcolley · 2022-06-12T19:12:16Z

@shawnwang0715
In my experience, arm64 (incl. Mac M1) is still too niche for many projects - especially those working with Rust. The low-level dependencies aren't available and I found myself debugging too long to try to get it working. After 40+ hours (over some weeks) I pulled the plug and got a £500 server.
I am running ubuntu x64 on bare-metal. Just follow the instructions for ubuntu as per the docs.

steveluscher · 2022-07-16T04:59:25Z

I'm going to close this issue, but in doing so we're going to lose all of this valuable investigation and solution-finding.

Can someone please re-post the question and a summary of the answer to https://solana.stackexchange.com/questions/ask so that people who have these sorts of problems in the future can find their way to it? Thanks!

gxxcastillo mentioned this issue Aug 28, 2021

start validator solana-test-validator Library not loaded error #17222

Closed

dcolley mentioned this issue Sep 30, 2021

Problem with anchor deploy metaspan/solana-docker-mac-m1#1

Open

This was referenced Oct 31, 2021

Solana-test-validator error on mac M1 chip #19743

Closed

Installing from source on an M1 Mac #20783

Closed

matteosantama mentioned this issue Jan 29, 2022

Unable to deploy IDL coral-xyz/anchor#1033

Closed

nathanleclaire mentioned this issue Mar 7, 2022

Can't deploy programs to the localnet validator started by workbench workbenchapp/solana-workbench#26

Closed

steveluscher closed this as completed Jul 16, 2022

gartnera mentioned this issue Jun 28, 2024

Build multi-architecture solana test validator docker image zeta-chain/node#2398

Closed

Query: Deploying hello world to solana-test-validator on M1 fails #18033

Query: Deploying hello world to solana-test-validator on M1 fails #18033

Comments

fdeantoni commented Jun 17, 2021

Problem

Proposed Solution

neildotvn commented Aug 24, 2021

gxxcastillo commented Aug 26, 2021 • edited Loading

gxxcastillo commented Aug 29, 2021

monomadic commented Aug 31, 2021 • edited Loading

Qumeric commented Sep 4, 2021 • edited Loading

gxxcastillo commented Sep 4, 2021

Qumeric commented Sep 5, 2021 • edited Loading

artursapek commented Sep 5, 2021

Krishna-Psftware commented Sep 22, 2021

Krishna-Psftware commented Sep 22, 2021 • edited Loading

Krishna-Psftware commented Sep 22, 2021

crmanik commented Sep 27, 2021

dcolley commented Sep 30, 2021

dcolley commented Sep 30, 2021

codenjobs commented Oct 6, 2021

amalcaraz commented Nov 7, 2021 • edited Loading

newhouseb commented Nov 8, 2021

fanatid commented Nov 9, 2021

newhouseb commented Nov 15, 2021

nphenley commented Nov 19, 2021

chihkaiyu commented Nov 19, 2021

diginator commented Dec 1, 2021

Vkzem commented Dec 11, 2021

diginator commented Dec 11, 2021 • edited Loading

Vkzem commented Dec 11, 2021

n0x1m commented Jan 6, 2022

diginator commented Jan 6, 2022

t-nelson commented Jan 6, 2022

n0x1m commented Jan 8, 2022

dcolley commented Mar 3, 2022

Running Ubuntu 18.04 arm64 in Docker on Mac M1

nathanleclaire commented Mar 6, 2022 • edited Loading

nathanleclaire commented Apr 18, 2022

shawnwang0715 commented Jun 12, 2022 • edited Loading

dcolley commented Jun 12, 2022

shawnwang0715 commented Jun 12, 2022

dcolley commented Jun 12, 2022

steveluscher commented Jul 16, 2022

gxxcastillo commented Aug 26, 2021 •

edited

Loading

monomadic commented Aug 31, 2021 •

edited

Loading

Qumeric commented Sep 4, 2021 •

edited

Loading

Qumeric commented Sep 5, 2021 •

edited

Loading

Krishna-Psftware commented Sep 22, 2021 •

edited

Loading

amalcaraz commented Nov 7, 2021 •

edited

Loading

diginator commented Dec 11, 2021 •

edited

Loading

nathanleclaire commented Mar 6, 2022 •

edited

Loading

shawnwang0715 commented Jun 12, 2022 •

edited

Loading