Check if you are running arm64 or amd64 brew
With the switch to m1. Most people will still be running the amd64 version of brew, which will be going through a translation layer all the time.
Run
which brew
If you get /opt/homebrew/bin/brew
Then you are using the correct arm version.
If you get /usr/local/bin/brew
You are still using the amd64 version.
You can follow the following if you want to switch.