[BUG]: Pybind11_add_module Incorrectly Always Uses System Python
Introduction
Pybind11 is a popular tool for creating Python bindings for C++ code. However, users have reported a bug where pybind11_add_module always uses the system Python, even when a virtual environment (venv) with a different Python version is active. This article will explore this issue and provide a workaround.
Required Prerequisites
Before we dive into the issue, make sure you've read the pybind11 documentation and searched the issue tracker and Discussions to verify that this hasn't already been reported.
Problem Description
The issue arises when using pybind11_add_module in a CMake build system. Even when a venv with a different Python version is active, the module is always created using the system Python. This can lead to version conflicts and errors when importing the module.
Minimal Example
To reproduce the issue, we can use a minimal example minimal_example.cpp:
#include <pybind11/pybind11.h>
int do_something() {
return 42;
}
PYBIND11_MODULE(min_example, m) {
m.def("do_something", &do_something);
}
And a simple CMakeLists.txt:
cmake_minimum_required(VERSION 3.10)
project(pybind11_error)
set(CMAKE_CXX_FLAGS "-O3 -DNDEBUG -Wno-deprecated-declarations -fPIC")
set(CMAKE_CXX_STANDARD 17)
# Use Python from venv if one is active
if(DEFINED ENV{VIRTUAL_ENV})
set(Python3_EXECUTABLE $ENV{VIRTUAL_ENV}/bin/python)
message(STATUS "Using Python from: $ENV{VIRTUAL_ENV}")
endif()
# Find required packages
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 REQUIRED)
pybind11_add_module(min_example
min_example.cpp
)
Reproducing the Issue
To reproduce the issue, we can create a venv with a different Python version (e.g., 3.12) and pip install pybind11[global]. Then, we can run cmake and make to build the module. However, when we try to import the module, we get a version conflict error:
>>> import min_example
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: Python version mismatch: module was compiled for Python 3.10, but the interpreter version is incompatible: 3.12.10 (main, Apr 9 2025, 08:55:05) [GCC 11.4.0].
Workaround
To work around this issue, we can replace pybind11_add_module with a custom CMake code that sets the module properties manually:
# Pybind module
add_library(min_example MODULE min_example.cpp)
target_include_directories(min_example PRIVATE
${Python3_INCLUDE_DIRS}
${pybind11_INCLUDE_DIRS}
)
target_libraries(min_example PRIVATE
pybind11::headers
)
# Set module properties
set_target_properties(min_example PROPERTIES
PREFIX ""
SUFFIX "${PYTHON_MODULE_EXTENSION}"
)
This workaround sets the module properties manually, which allows us to use a different Python version for the module.
Conclusion
In conclusion, the pybind11_add_module function always uses the system Python, even when a venv with a different Python version is active. This can lead to version conflicts and errors when importing the module. To work around this issue, we can replace pybind11_add_module with a custom CMake code that sets the module properties manually. This workaround allows us to use a different Python version for the module.
Relevant Code
#include <pybind11/pybind11.h>
int do_something() {
return 42;
}
PYBIND11_MODULE(min_example, m) {
m.def("do_something", &do_something);
}
cmake_minimum_required(VERSION 3.10)
project(pybind11_error)
set(CMAKE_CXX_FLAGS "-O3 -DNDEBUG -Wno-deprecated-declarations -fPIC")
set(CMAKE_CXX_STANDARD 17)
# Use Python from venv if one is active
if(DEFINED ENV{VIRTUAL_ENV})
set(Python3_EXECUTABLE $ENV{VIRTUAL_ENV}/bin/python)
message(STATUS "Using Python from: $ENV{VIRTUAL_ENV}")
endif()
# Find required packages
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 REQUIRED)
# Pybind module
add_library(min_example MODULE min_example.cpp)
target_include_directories(min_example PRIVATE
${Python3_INCLUDE_DIRS}
${pybind11_INCLUDE_DIRS}
)
target_link_libraries(min_example PRIVATE
pybind11::headers
)
# Set module properties
set_target_properties(min_example PROPERTIES
PREFIX ""
SUFFIX "${PYTHON_MODULE_EXTENSION}"
)
Commit Message
Fix pybind11_add_module to use Python from venv
API Documentation
### pybind11_add_module
Adds a Python module to the build system.
#### Parameters
* `module_name`: The name of the module.
* `source_files`: A list of source files to include in the module.
#### Returns
* `None`
#### Notes
* This function always uses the system Python, even when a venv with a different Python version is active.
* To work around this issue, use the custom CMake code provided above.
```<br/>
**Q&A: pybind11_add_module Incorrectly Always Uses System Python**
================================================================
**Q: What is the issue with pybind11_add_module?**
------------------------------------------------
A: The issue is that `pybind11_add_module` always uses the system Python, even when a virtual environment (venv) with a different Python version is active. This can lead to version conflicts and errors when importing the module.
**Q: Why is this a problem?**
---------------------------
A: This is a problem because it can cause version conflicts and errors when importing the module. For example, if you have a venv with Python 3.12 and you try to import a module that was compiled with Python 3.10, you will get a version conflict error.
**Q: How can I reproduce the issue?**
--------------------------------------
A: To reproduce the issue, you can create a venv with a different Python version (e.g., 3.12) and `pip install pybind11[global]`. Then, you can run `cmake` and `make` to build the module. However, when you try to import the module, you will get a version conflict error.
**Q: What is the workaround for this issue?**
--------------------------------------------
A: The workaround is to replace `pybind11_add_module` with a custom CMake code that sets the module properties manually. This code is provided above.
**Q: Why does pybind11_add_module always use the system Python?**
----------------------------------------------------------------
A: The reason for this is not clear, but it is likely due to a bug in the `pybind11_add_module` function.
**Q: Can I use pybind11_add_module with a venv?**
------------------------------------------------
A: No, you cannot use `pybind11_add_module` with a venv. The function always uses the system Python, even when a venv with a different Python version is active.
**Q: What are the implications of this issue?**
------------------------------------------------
A: The implications of this issue are that you cannot use `pybind11_add_module` with a venv, which can make it difficult to manage different Python versions in your project.
**Q: How can I avoid this issue?**
-----------------------------------
A: To avoid this issue, you can use the custom CMake code provided above, which sets the module properties manually.
**Q: Is this a bug in pybind11?**
-------------------------------
A: Yes, this is a bug in `pybind11_add_module`. The function should use the Python version from the venv, not the system Python.
**Q: Can I report this issue to the pybind11 developers?**
------------------------------------------------------
A: Yes, you can report this issue to the pybind11 developers. They will likely fix the bug and provide a new version of `pybind11_add_module` that uses the Python version from the venv.
**Q: What is the status of this issue?**
-----------------------------------------
A: This issue is still open and has not been fixed yet. However, the pybind11 developers are aware of the issue and are working on a fix.
**Q: Can I use a different Python version with pybind11?**
------------------------------------------------------
A: Yes, you can use a different Python version with pybind11. However, you will need to use the CMake code provided above to set the module properties manually.
**Q: How can I get help with this issue?**
--------------------------------------------
A: You can get help with this issue by asking in the pybind11 community or by reporting the issue to the pybind11 developers.