安装hanlpfull 遇到了问题

根据文档,我使用 pipi install hanlp[full] 命令安装完全版,但是最后到如下步骤的时候出错。根据文档,hanlp[full]应该安装tensonflow和c++等扩展组件。 这个要如何解决?我下载的时候,用的是清华源

——————————————————————————

Building wheels for collected packages: tokenizers
Building wheel for tokenizers (pyproject.toml) … error
ERROR: Command errored out with exit status 1:
command: ‘d:\py_proj_53git\kfai_nlp\venv\scripts\python.exe’ ‘d:\py_proj_53git\kfai_nlp\venv\lib\site-packages\pip_vendor\pep517\in_process_in_process.py’ buil
d_wheel ‘C:\Users\CHENGX~1\AppData\Local\Temp\tmp_y2gqtx3’
cwd: C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714
Complete output (81 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-3.6
creating build\lib.win-amd64-3.6\tokenizers
copying py_src\tokenizers_init_.py → build\lib.win-amd64-3.6\tokenizers
creating build\lib.win-amd64-3.6\tokenizers\models
copying py_src\tokenizers\models_init_.py → build\lib.win-amd64-3.6\tokenizers\models
creating build\lib.win-amd64-3.6\tokenizers\decoders
copying py_src\tokenizers\decoders_init_.py → build\lib.win-amd64-3.6\tokenizers\decoders
creating build\lib.win-amd64-3.6\tokenizers\normalizers
copying py_src\tokenizers\normalizers_init_.py → build\lib.win-amd64-3.6\tokenizers\normalizers
creating build\lib.win-amd64-3.6\tokenizers\pre_tokenizers
copying py_src\tokenizers\pre_tokenizers_init_.py → build\lib.win-amd64-3.6\tokenizers\pre_tokenizers
creating build\lib.win-amd64-3.6\tokenizers\processors
copying py_src\tokenizers\processors_init_.py → build\lib.win-amd64-3.6\tokenizers\processors
creating build\lib.win-amd64-3.6\tokenizers\trainers
copying py_src\tokenizers\trainers_init_.py → build\lib.win-amd64-3.6\tokenizers\trainers
creating build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\base_tokenizer.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\bert_wordpiece.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\byte_level_bpe.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\char_level_bpe.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\sentencepiece_bpe.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations\sentencepiece_unigram.py → build\lib.win-amd64-3.6\tokenizers\implementations
copying py_src\tokenizers\implementations_init_.py → build\lib.win-amd64-3.6\tokenizers\implementations
creating build\lib.win-amd64-3.6\tokenizers\tools
copying py_src\tokenizers\tools\visualizer.py → build\lib.win-amd64-3.6\tokenizers\tools
copying py_src\tokenizers\tools_init_.py → build\lib.win-amd64-3.6\tokenizers\tools
copying py_src\tokenizers_init_.pyi → build\lib.win-amd64-3.6\tokenizers
copying py_src\tokenizers\models_init_.pyi → build\lib.win-amd64-3.6\tokenizers\models
copying py_src\tokenizers\decoders_init_.pyi → build\lib.win-amd64-3.6\tokenizers\decoders
copying py_src\tokenizers\normalizers_init_.pyi → build\lib.win-amd64-3.6\tokenizers\normalizers
copying py_src\tokenizers\pre_tokenizers_init_.pyi → build\lib.win-amd64-3.6\tokenizers\pre_tokenizers
copying py_src\tokenizers\processors_init_.pyi → build\lib.win-amd64-3.6\tokenizers\processors
copying py_src\tokenizers\trainers_init_.pyi → build\lib.win-amd64-3.6\tokenizers\trainers
copying py_src\tokenizers\tools\visualizer-styles.css → build\lib.win-amd64-3.6\tokenizers\tools
running build_ext
running build_rust
Updating crates.io index
warning: unused manifest key: target.x86_64-apple-darwin.rustflags
Compiling cfg-if v1.0.0
Compiling autocfg v1.1.0
Compiling winapi v0.3.9
Compiling proc-macro2 v1.0.36
Compiling unicode-xid v0.2.2
Compiling syn v1.0.90
Compiling lazy_static v1.4.0
Compiling memchr v2.3.4
Running rustc --crate-name cfg_if --edition=2018 "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\cfg-if-1.0.0\src\lib.rs" --error-fo rmat=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=16 0aa69f99926d87 -C extra-filename=-160aa69f99926d87 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\t arget\x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f84 29f5a4c039cba8f0b34ad1714\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a 4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name autocfg "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.1.0\src\lib.rs" --error-format=json --j son=diagnostic-rendered-ansi,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=1656de8c46573c79 -C extra-filename=-1656de8c46573c79 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\ deps -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name unicode_xid "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-xid-0.2.2\src\lib.rs" --error-format= json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "fea ture=\"default\"" -C metadata=3fc24dc6545c18dd -C extra-filename=-3fc24dc6545c18dd --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3 d7f8429f5a4c039cba8f0b34ad1714\target\release\deps -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad17 14\target\release\deps --cap-lints allow
Running rustc --crate-name build_script_build --edition=2018 "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.36\buil d.rs" --error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "f eature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=b059d5c42c6aebe4 -C extra-filename=-b059d5c42c6aebe4 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\p ip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\build\proc-macro2-b059d5c42c6aebe4 -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp \pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name build_script_build "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.3.4\build.rs" --error-format= json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" - -cfg "feature=\"std\"" --cfg "feature=\"use_std\"" -C metadata=765b56e8d3b1d9d3 -C extra-filename=-765b56e8d3b1d9d3 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\p ip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\build\memchr-765b56e8d3b1d9d3 -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip- install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name lazy_static "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\lazy_static-1.4.0\src\lib.rs" --error-format= json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=f8067a4 77347dc90 -C extra-filename=-f8067a477347dc90 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target \x86_64-pc-windows-msvc\release\deps --target x86_64-pc-windows-msvc -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a 4c039cba8f0b34ad1714\target\x86_64-pc-windows-msvc\release\deps -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039 cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name build_script_build --edition=2018 "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.90\build.rs" -- error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\ "clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg " feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" --cfg "feature=\"visit\"" -C metadata=be918d20599dbcc3 -C extra-filename=-be918d20599 dbcc3 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\build\syn-be918d20599dbcc3 -L d ependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Running rustc --crate-name build_script_build "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-0.3.9\build.rs" --error-format= json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"cfg\"" --cfg "feature=\"consoleapi\"" --cfg "feature=\"errhandlingapi\"" --cfg "feature=\"evntrace\"" --cfg "feature=\"fileapi\"" --cfg "feature=\"handleapi\"" --cfg "feature=\ "impl-debug\"" --cfg "feature=\"impl-default\"" --cfg "feature=\"in6addr\"" --cfg "feature=\"inaddr\"" --cfg "feature=\"ioapiset\"" --cfg "feature=\"knownfolders\"" --cfg "feature=\"lmcons\"" --cfg "feature=\"minschannel\"" --cfg "feature=\"minwinbase\"" --cfg "feature=\"minwindef\"" --cfg "feature=\"mswsock\"" --cfg "feature= \"namedpipeapi\"" --cfg "feature=\"ntdef\"" --cfg "feature=\"ntsecapi\"" --cfg "feature=\"ntstatus\"" --cfg "feature=\"objbase\"" --cfg "feature=\"processenv\"" --c fg "feature=\"processthreadsapi\"" --cfg "feature=\"profileapi\"" --cfg "feature=\"schannel\"" --cfg "feature=\"securitybaseapi\"" --cfg "feature=\"shlobj\"" --cfg "feature=\"sspi\"" --cfg "feature=\"std\"" --cfg "feature=\"synchapi\"" --cfg "feature=\"sysinfoapi\"" --cfg "feature=\"timezoneapi\"" --cfg "feature=\"winbase\"" - -cfg "feature=\"wincon\"" --cfg "feature=\"wincrypt\"" --cfg "feature=\"windef\"" --cfg "feature=\"winerror\"" --cfg "feature=\"winioctl\"" --cfg "feature=\"winnt\" " --cfg "feature=\"winreg\"" --cfg "feature=\"winsock2\"" --cfg "feature=\"winuser\"" --cfg "feature=\"ws2def\"" --cfg "feature=\"ws2ipdef\"" --cfg "feature=\"ws2tc pip\"" -C metadata=7f002f974218cb1d -C extra-filename=-7f002f974218cb1d --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4 c039cba8f0b34ad1714\target\release\build\winapi-7f002f974218cb1d -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c03 9cba8f0b34ad1714\target\release\deps --cap-lints allow
Compiling libc v0.2.121
Running rustc --crate-name build_script_build "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\libc-0.2.121\build.rs" --error-format= json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" - -cfg "feature=\"std\"" -C metadata=84786c6b9e9d2a1b -C extra-filename=-84786c6b9e9d2a1b --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenize rs_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\build\libc-84786c6b9e9d2a1b -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3 d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Compiling cc v1.0.73
Running rustc --crate-name cc --edition=2018 "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\cc-1.0.73\src\lib.rs" --error-format=js on --json=diagnostic-rendered-ansi,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=e934d95930 f408fc -C extra-filename=-e934d95930f408fc --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\re lease\deps -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow
Compiling log v0.4.16
Running rustc --crate-name build_script_build "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\log-0.4.16\build.rs" --error-format=js on --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"std\"" -C meta data=2b98e7d5f42a0ee5 -C extra-filename=-2b98e7d5f42a0ee5 --out-dir C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34a d1714\target\release\build\log-2b98e7d5f42a0ee5 -L dependency=C:\Users\CHENGX~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\ target\release\deps --cap-lints allow
error: linker link.exe not found
|
= note: program not found

note: the msvc targets depend on the msvc linker but link.exe was not found

note: please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option

error: could not compile memchr due to previous error

Caused by:
process didn’t exit successfully: rustc --crate-name build_script_build "C:\Users\Cheng Xiaobin\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.3.4\bu ild.rs" --error-format=json --json=diagnostic-rendered-ansi,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" --cfg "feature=\"use_std\"" -C metadata=765b56e8d3b1d9d3 -C extra-filename=-765b56e8d3b1d9d3 --out-dir C:\Users\CHENGX ~1\AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\build\memchr-765b56e8d3b1d9d3 -L dependency=C:\Users\CHENGX~1\ AppData\Local\Temp\pip-install-r85i1do3\tokenizers_b3d7f8429f5a4c039cba8f0b34ad1714\target\release\deps --cap-lints allow (exit code: 1)
warning: build failed, waiting for other jobs to finish…
error: build failed
cargo rustc --lib --manifest-path Cargo.toml --target x86_64-pc-windows-msvc --release -v --features pyo3/extension-module – --crate-type cdylib
error: cargo failed with code: 101


ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

(venv) D:\py_proj_53git\kfai_nlp>

继续提问,我安装了vs 2019, 重启了pycharm,但是还是出现同样的错误。

已经解决了,说明一下,需要安装vs2019 ,并且要勾选c+=的选项

1赞

持续咨询一个问题,在win10下,pip install hanlp[full] 要求安装vs 2019 c++。 那么如果在linux下,是否也需要,要怎么安装呢?有老师可以解答一下吗?

HanLP[full]依赖的Huggingface tokenizers等类库以源码形式发布,可能需要编译器支持,这对大多数Windows用户而言可能不习惯,因为Windows是闭源系统。然而Linux是开源系统,软件习惯上以源码形式传播,编译器基本上要么已经默认安装了,要么一句命令就装上了,没有这么多麻烦事。

HanLP[full]是为专家开发的,专家基本都用Linux。

2赞

明白,也就是说,在linux上不存在这个问题,只是在win上面需要安装vs编译器,是吧?

是的。

其实tokenizers在Windows上发布了Python3.7以上的binary wheel,你把python升级到3.7以上也就不需要编译器了。

明白,但目前公司用的是python3.6,改动影响还蛮大的,所以就这样吧,先用着。
谢谢大神。

安装vs 2019就可以解决吗

我python已经是3.11.4,但是还是会报这个错误