[安装FlashAttention] CUDA版本 和 Nvidia驱动版本
nvidia-smi 查看driver api 的CUDA版本
听说这个是本机能装到的最高版本
那这样看来我最高能装到12.4。
nvcc -V 查看当前runtime api的CUDA版本
还是古老的11.5版本，没办法啊，FlashAttention老是说不支持？
安装Torch时选择的CUDA版本
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
我是使用这个命令安装的，也就是CUDA版本是11.8，在conda环境安装的runtime api是CUDA11.8版本的。
flash attention报错如下：
看来flash attention用的是系统的那个CUDA runtime api，而不是conda环境的，所以他说我的CUDA版本太低了。。。。
那我只能升级系统CUDA版本咯
参考别人的安装过程
- https://blog.csdn.net/m0_52583356/article/details/138150039
- https://blog.csdn.net/h3c4lenovo/article/details/119003405
可能是这个 CUDA12.1 是之前就在的，我安装并没有安装成功，因为这个时间节点是2023。。。
但是我自己重新开一个窗口，发现 nvcc -V还是低版本的
原来是自己的 bashrc 没有配置路径：
打开 ~/.bashrc 文件，输入以下内容
export PATH=/usr/local/cuda-12.1/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
我这一步就搞定了，不需要像https://blog.csdn.net/h3c4lenovo/article/details/119003405 这个链接里面说的那样，还要安装cudnn。
记得 source ~/.bashrc
但是还是没办法安装。。。。
一开始我以为是我 torch 安装的 CUDA toolkit11.8，nvcc -V是12.1会冲突，然后我把torch也换成了CUDA12.1的，但是还是报了神奇的错误。。。。
我明明有torch但是他说我没有
(FlashAttn) Guanjq@server-4080-8:~/flash-attention$ pip install flash-attn --use-pep517
Collecting flash-attn
Using cached flash_attn-2.7.4.post1.tar.gz (6.0 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
Traceback (most recent call last):
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
main()
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-chrlj9e0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 334, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-chrlj9e0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 304, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-chrlj9e0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 522, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-chrlj9e0/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "<string>", line 22, in <module>
ModuleNotFoundError: No module named 'torch'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
(FlashAttn) Guanjq@server-4080-8:~/flash-attention$ python
Python 3.11.11 (main, Dec 11 2024, 16:28:39) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>
嘶，很神奇，不要用官网的那个安装命令
pip install flash-attn --no-build-isolation会报神奇的错误，但是pip install flash-attn就不报错，这个是什么意思…
然后就安装成功了？？？
那我试试
怎么不能正常使用啊
import torch
from flash_attn import flash_attn_qkvpacked_func, flash_attn_func
q = torch.ones((1, 10, 1, 768))
k = torch.ones((1, 20, 1, 768))
v = torch.ones((1, 20, 1, 768))
flash_attn_func(q, k, v, dropout_p=0.0, softmax_scale=None, causal=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 1201, in flash_attn_func
return FlashAttnFunc.apply(
^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 839, in forward
out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 96, in _flash_attn_forward
out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.fwd(
^^^^^^^^^^^^^^^^^^^
RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at "/opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/torch/include/c10/cuda/impl/CUDAGuardImpl.h":25, please report a bug to PyTorch.
这个是什么错误啊？？？？？？
奥奥，原来是FlashAttention只支持 CUDA 上面的tensor，但是tensor()生成默认是CPU。
但是兄弟！还没完！！
FlashAttention 甚至精度只能支持 Float16。。。
>>> v = torch.ones((1, 20, 1, 768)).cuda()
>>> k = torch.ones((1, 20, 1, 768)).cuda()
>>> q = torch.ones((1, 10, 1, 768)).cuda()
>>> flash_attn_func(q, k, v, dropout_p=0.0, softmax_scale=None, causal=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 1201, in flash_attn_func
return FlashAttnFunc.apply(
^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 839, in forward
out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 96, in _flash_attn_forward
out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.fwd(
^^^^^^^^^^^^^^^^^^^
RuntimeError: FlashAttention only support fp16 and bf16 data type
好的，兄弟！还没有完！！！
他说它的head dim 最高只能是256.。。。
>>> v = torch.ones((1, 20, 1, 768), dtype=torch.float16).cuda()
>>> q = torch.ones((1, 10, 1, 768), dtype=torch.float16).cuda()
>>> k = torch.ones((1, 20, 1, 768), dtype=torch.float16).cuda()
>>> flash_attn_func(q, k, v, dropout_p=0.0, softmax_scale=None, causal=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 1201, in flash_attn_func
return FlashAttnFunc.apply(
^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/miniconda3/envs/FlashAttn/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 839, in forward
out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/Guanjq/flash-attention/flash_attn/flash_attn_interface.py", line 96, in _flash_attn_forward
out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.fwd(
^^^^^^^^^^^^^^^^^^^
RuntimeError: FlashAttention forward only supports head dimension at most 256
好吧，那我使用多头注意力机制。。。
>>> v = torch.ones((1, 20, 8, 96), dtype=torch.float16).cuda()
>>> q = torch.ones((1, 20, 8, 96), dtype=torch.float16).cuda()
>>> k = torch.ones((1, 20, 8, 96), dtype=torch.float16).cuda()
>>> flash_attn_func(q, k, v, dropout_p=0.0, softmax_scale=None, causal=False)
这下终于调通乐。。。。