C++ - Python Binding
Introduction
- Binding is the act of creating a bridge between two pieces of code to interface between different OS or programming languages.
-
The following will present binding between C++ and Python, and how to embed a Python interpreter inside C++ code.
-
We will use PyBind11 for this purpose.
- The doc is available under here.
- The library consists of multiple wrappers around
Python.hthanks to macros, and permit cross-language communication.
Integrate (CMake)
You can embed the library in two different ways :
- By installing PyBind11 in local, and referencing it inside CMakeLists.txt with
find_package(pybind11 REQUIRED) -
By embedding it as a subfolder using
add_subdirectory(pybind11)(you will need to clone the whole repo and put in in your project, but prefer official releases found in the GitHub) -
Then, you can generate a Python extension file including code written in C++ (see further chapters). It will also be available for your embedded interpreter.
- The file will look as follow :
project(MyProject)
# Add PyBind11 as subfolder
add_subdirectory(pybind11)
# Add PyBind11 as installed somewhere on disk
# find_package(pybind11 REQUIRED)
# C++ app files
file(GLOB PROJECT_SOURCES
"main.cpp"
# ...
)
# Exec
add_executable(MyProject ${PROJECT_SOURCES})
# To embed interpreter and use basic modules
target_link_libraries(MyProject PRIVATE pybind11::embed pybind11::module)
# Our generated Python lib (see below chapters)
pybind11_add_module(MyPYModule pythonbindings.cpp MyModule.h MyModule.cpp) # all files required by your extension
# Neat little trick if you have some Python script in your project tree that you'd like to access with the embedded interpreter
# The file will be copied each time (build, run ...) so you can edit it freely and not worry to have the latest version with your app
add_custom_target(copyPyScript ALL
COMMAND ${CMAKE_COMMAND} -E copy
${CMAKE_SOURCE_DIR}/which/file/tocopy.py
${CMAKE_CURRENT_BINARY_DIR}
)
add_dependencies(MyProject copyPyScript)
Note
For the following chapters, we assume PyBind11 as a subfolder. Examples are taken from a custom Qt app.
Importing PyBind11 (frameworks using slots keyword)
- PyBind11 uses the keyword
slotsin quite a few places. Qt will not be happy for example, as itself uses this keyword for its Signal and Slots callback mechanism. - To import without worry, use the following lines that will temporarily disable the macro while importing PyBind11 :
// Cross names error with Qt - disable slots while including required files
#pragma push_macro("slots")
#undef slots
#include "pybind11/include/pybind11/pybind11.h"
#include "pybind11/include/pybind11/embed.h"
#pragma pop_macro("slots")
pybind11 namespace to a more friendly and simple py namespace :
namespace py = pybind11; // creates a namespace alias
Warning
- Avoid the declaration
using namespace pybind11;. - While you could more easily use PyBind11 related objects, you will pollute your namespace and may create errors.
- Also, it is easier to read code with
py::leading before Python-specific declarations.
Calling Python from C++
- To be able to use Python in C++, the
#include "pybind11/include/pybind11/embed.h"is required. - Let's say we have the following script called
mymodule.pythat we may use along with some other code :
def printHello(boolMirrorInput):
print("Hello")
return boolMirrorInput
PYBIND11_EMBEDDED_MODULE(mymodule_embedded, m)
{
// `m` is a `py::module_` which is used to bind functions and classes
m.def("printHello", [](bool mirrorinput)
{
std::cout << "Hello from Cpp !"" << std::endl;
return mirrorinput;
});
}
// Do some Python in C++
static void doPython()
{
// Function is not thread safe - only one thread can call it at a time - should mutex lock it
// 1) Start the interpreter and keep it alive
py::scoped_interpreter guard{};
// 2) Some Python code
// Simple
py::print("Simple line hello");
// More advanced
py::exec
(
R"(
kwargs = dict(name="advanced Python", avalue=3)
message = "Hello, {name}! I am {avalue}".format(**kwargs)
print(message)
)"
);
// 3) Import a Python module, 'sys' here
py::module_ sys_module = py::module_::import("sys");
// Get the max size of an integer
py::object result = sys_module.attr("maxsize")();
// 4) Import our own Python script (module name is name of the file.py)
py::module_ my_module = py::module_::import("mymodule");
// Call one of the function
result = my_module.attr("printHello")(true);
assert(result.cast<bool>()); // if not true, function did not return its input
// 5) Use our embedded module
py::module_ emb_module = py::module_::import("mymodule_embedded");
result = emb_module.attr("printHello")(false);
assert(!result.cast<bool>()); // if not false, function did not return its input
}
(1) Guard object
- The
guardobject will start the interpreter and keep it running WHILE THE OBJECT IS ALIVE. - When the object is destroyed (whenever its scope ends), the interpreter cannot be used anymore.
Warning
- Only ONE guard can be used at a time, otherwise a crash will occur.
- It would be possible to run multiple interpreters through the
CPython`` api, butwith caveats to consider.
Note
In C++, the scope strongly depends on the placement of the variable.
- A static var in a
.cppfile may always exist - A static var as a member of a class may only appear the first time an object of said class is instantiated
- A pointer will start to live on the
newcommand and finish on thedeleteone - A local variable will end after the function/if/do/for ... statement scope is finished
With this in head, you can create localized instantiation of the object to use during a small part of a function (e.g. if you use the interpreter in a multithreaded app, where multiple threads could want to use the interpreter (but remember to have a mutex, since only one guard can exist at a time !)), thanks to the syntax :
void myfunc()
{
// Do something
// ...
// Oh, I need Python !
// TODO : lock your mutex
{
py::scoped_interpreter guard{};
// do Python
}
// scope of guard is finished -> interpreter is terminated
// TODO : unlock your mutex
}
- While it can seem strange to have
{}without any prior statement like anifor so, that is how C++ works ! - An
if(condition) dosomething;on one line works like anifwith multiple lines inside brackets : theifwill execute what he sees next - thedosomethingfor the one-line version, and the brackets (a local scope where lies multiple lines of code) for theif(){}version.
(2) Python code in C++
You can use some built-in functions like py::print() for simple code, or even exec multiple lines of code with py::exec(), py::eval() ...
(3) Local Python modules
- A module is easily imported with the command
py::module_ mymodule = py::module_::import(char* modulename);. 2.- Afterward, you get a result calling the function by its name, through the previously imported module :
py::object result = sys_module.attr("maxsize")();. - If the function has no argument, let second parenthesis empty, else fill with the required arguments.
- Afterward, you get a result calling the function by its name, through the previously imported module :
- You must know the type of returned object to be able to cast it back to its C++ version :
int res = result.cast<int>().
Warning
Your code may crash for multiple reasons, giving you a hard-time finding the problem :
- On the acquisition of the guard :
- if another guard object already exists
- On the import of a module :
- if the module does not exist
- if the module is not valid (Python would itself be unhappy trying to run this script)
- if some dependencies cannot be found
- On the execution of a function :
- if the function does not exist
- if the passed arguments count does not correspond to the function (if no default arguments exist, they MUST be filled even if those are not useful for the function)
- if the type of the arguments mismatch
- if you try to pass pointers/objects of some C++ class that Python is not aware of, even if you just intend to stock them to further re-use with C++ code (like a Python server using callbacks on C++ code)
- On the cast of a result :
- if you try to cast to an unknown type for Python (e.g. one of your class pointer)
- if you try to cast to the wrong type
- if you try to cast the result from a miscalled function (calling
result = sys_module.attr("maxsize");instead ofresult = sys_module.attr("maxsize")();will not crash on this line but on a further cast, as we did not really call the function)
(4) Local script
- Your
script.pysolely needs to lie near your.exeor be added to the Pythonsyspath. The calls are the same as before. - The module is imported following the name of the file (e.g. for
my_superScript_pyp.y.py, you would need to usepy::module_::import("my_superScript_pyp.y");).
(5) Embedded module
- You can use an embedded module as you would with any other module, simply giving its
PYBIND11_EMBEDDED_MODULEname. - The generated Python extension is present near your
.exeinside your build folders.
Using C++ from Python
Creating a module
- You first need to build your C++ code that you will call from Python.
- We will create a simple example that will decouple the binding, the Python calls, and the executed functions (they will be written as
.hppformat for easier read).
First, the actual functions, pure C++ :
// cppcode.hpp
#ifndef CPPCODE_HPP
#define CPPCODE_HPP
#include <iostream>
#include <string>
class SomeCppCode
{
public:
static void say(std::string s)
{
std::cout << s << std::endl;
}
static int add(int a, int b)
{
return a + b;
}
private:
SomeCppCode() {}; // cannot instantiate
}
#endif // CPPCODE_HPP
- Standard C++ code, no special consideration.
Then, we have the Python calls through C++ :
// pythoncalls.hpp
#ifndef PYTHONCALLS_HPP
#define PYTHONCALLS_HPP
#pragma push_macro("slots")
#undef slots
#include "pybind11/include/pybind11/pybind11.h"
#include "pybind11/include/pybind11/embed.h"
#pragma pop_macro("slots")
namespace py = pybind11;
#include "cppcode.hpp"
class PythonCalls
{
public:
PythonCalls() : _last_a(0), _last_b(0), _last_said("") {}
void callSay(std::string s, copyToLast = True)
{
SomeCppCode::say(s);
if(copyToLast)
{
_last_said = s;
}
}
void callSayAgain()
{
callSay(_last_said, False);
}
int callAdd(int a, int b, copyToLast = True)
{
if(copyToLast)
{
_last_a = a;
_last_b = b;
}
return SomeCppCode::add(a,b);
}
int callAddFromInternal()
{
return callAdd(_last_a, _last_b, False);
}
public:
int _last_a, _last_b;
private:
std::string _last_said;
}
#endif // PYTHONCALLS_HPP
- Some more C++ code, but note that we have public and private members, and only from standard, known types (no custom class pointer or so).
Finally, we create our Python module :
// pythonbindings.hpp
#ifndef PYTHONBINDING_HPP
#define PYTHONBINDING_HPP
#include "pythoncalls.hpp"
PYBIND11_MODULE(MyCustomModule, m) // m is the Python module where code will lie
{
// Doc for help(MyCustomModule)
m.doc() = R"pbdoc(
My Python module
-------------------------
Contains a class capable of saying things and adding, while interacting through C++ code
)pbdoc";
// Expose class to Python - we only have one
py::class_<PythonCalls>(m,"MagicCpp")
.def(py::init<>()) // args would go inside <> if existed
.def("say",&PythonCalls::callSay,"Say what we tell him to", py::arg("message"), py::arg(copytolast) = True)
.def("sayAgain",&PythonCalls::callSayAgain,"Say again what we last said")
.def("add",&PythonCalls::callAdd,"Add our two vars", py::arg("a"), py::arg("b"), py::arg(copytolast) = True)
.def("addFromInternal",&PythonCalls::callAddFromInternal,"Add the two vars currently inside object")
// Expose vars
.def_readwrite("last_a", &PythonCalls::_last_a, py::return_value_policy::reference);
.def_readwrite("last_b", &PythonCalls::_last_a, py::return_value_policy::reference);
// .def_readwrite("last_str", &PythonCalls::_last_said, py::return_value_policy::reference) <- illegal, is private so unreachable (same for constructor that MUST exist and be public)
}
#endif // PYTHONBINDING_HPP
MyCustomModule.
* m.doc() allows to create a documentation for whenever we would call help(MyCustomModule).
* The py::class_<>(m,name) expose our C++ class to Python with given name.
-
We then expose all the different functions and parameters (note that we can give them Python names that do not correspond to the C++ names). They must be
public, else the binding will fail. -
For functions, note that we must specify all the args. For already defined args in C++, we must re-set them here since PyBind11 cannot access those.
-
For parameters, note the return policy. This one is very important, since it can :
- Make a copy in Python, so we have two separate entities (one in C++ side, one in Python) (we won't pollute C++ by changing the values in Python, but changes would not reflect as we try to do here)
- Create a reference, so Python and C++ vars are tied together
The policies can also tell which side is managing the destruction of the object (using pointers for example).
- Without defining it, a default policy is set, but may not do what you think it will.
- Different
.defexist following the variable type (if const, static ...) with.def_readonly,.def_readonly_static,.def_readwrite_static... - You can also access private variables if you provide getters and setters.
-
You can also bind to anonymous functions (see first example).
-
The bindings should be put in a separate
.cppfile, only referenced in CMake. Else, we could quickly have problems like "already defined" errors as it would be redefined by some includes. - They must be referenced only ONCE from the whole project for the linker to work.
Let's modify our CMake where the module is created :
pybind11_add_module(MyCustomModule pythonbindings.hpp pythoncalls.hpp) # all files required by your extension
And voilĂ ! You can now use your module in Python with the generated extension, or thanks to the embedded interpreter (even if calling the C++ code directly may be a better choice if no reason justify passing through Python).
In your script, you can simply do :
from MyCustomModule import MagicCpp
mc = MagicCpp()
mc.say("Hello");
mc.sayAgain();
mc.add(3,5);
mc.addFromInternal();
mc.last_a = 1;
mc.addFromInternal();
# mc.last_str = "hoy"; <- crash, last_str is unknown
# mc.a_new_property = 3; <- crash, new parameters are not authorized
# For the previous line to work, the PYBIND11_EMBEDDED_MODULE class definition must be changed from py::class_<PythonCalls>(m,"MagicCpp") to py::class_<PythonCalls>(m,"MagicCpp", py::dynamic_attr())
Mindset
- It is very important to work with the correct mindset.
- If you simply use binding to calls "static" functions (such as adding two numbers), the mindset is quite simple. *
- But if you want to go more advanced and interact with your running C++ application, it is not because it can do various things and embeds a Python interpreter that it means Python is aware of your app.
- Let's say that your Python code should send events to your app such that you implemented a simple callback system where Python calls a function in C++ that will further use some kind of pointer to interact with your code.
- In your app you set said pointer, you run your interpreter, a callback is sent from Python, but sadly nothing changes in your C++ app.
- Never we said to Python that we have an app running and which "pointer to do further things" we are using. In fact, we could run this code solely from Python, without our app, and it would not crash just because our app is not running.
- Yes we are creating/using Python code from C++, but as a module (we never created "specific code for our application").
-
It is necessary to decouple both. We first need to create some kind of interface between SOME C++ code and SOME Python code, and then tell Python WHICH is our specific code (through pointers for example).
-
Let's review a more advanced system, written to interface C++ and Python to control our C++ application while Python is acting as an XML-RPC server.
We will have :
- A
serverinterface.h/.cpp: the layer between our app and the C++ Python callback system, sole able to call callbacks from it - A
serverinterface_py.h/.cpp: the callback system used by Python and communicating with C++ - A
pythonbindings.cpp: the elements exposed to Python
ServerInterface
Header file :
#ifndef SERVERINTERFACE_H
#define SERVERINTERFACE_H
#include <string>
#include <vector>
#include <thread>
#include <atomic>
#include "server/serverinterface_py.h"
class ServerInterface
{
// Allows binding to access private functions
friend class ServerInterface_py;
public:
~ServerInterface();
// Singleton-related functions
static ServerInterface& getInstance();
private:
ServerInterface();
ServerInterface(ServerInterface const&);
void operator=(ServerInterface const&);
public:
/**
* @brief Callbacks when server asks to do something
* @param cb callback
*/
void registerDoSomething(DoSomethingCallback cb);
/**
* @brief Remove callback from list
* @param cb callback
*/
void removeDoSomething(DoSomethingCallback cb);
private:
// Where our python script will live
void _server_thread_work();
// To stop our thread
volatile std::atomic_bool _stop_thread;
private:
// All registered callbacks
std::vector<DoSomethingCallback> _dosomethingcallbacks;
// Our python script
std::thread _server_thread;
};
#endif // SERVERINTERFACE_H
#include "serverinterface.h"
#include <chrono>
ServerInterface::ServerInterface() : _stop_thread(0)
{
// At instanciation, launches the server on a separate thread
_server_thread = std::thread(&ServerInterface::_server_thread_work, this);
// No join (we would block this thread while the server is not closed) nor detach (should not be used, memleak and no use)
}
ServerInterface::~ServerInterface()
{
// Removes server thread
_stop_thread = 1;
_server_thread.join(); // will wait for the thread to terminate
}
ServerInterface &ServerInterface::getInstance()
{
static ServerInterface inst;
return inst;
}
void ServerInterface::registerDoSomething(DoSomethingCallback cb)
{
_dosomethingcallbacks.push_back(cb);
}
void ServerInterface::removeDoSomething(DoSomethingCallback cb)
{
uint8_t i = 0;
for(DoSomethingCallback& cbi : _dosomethingcallbacks)
{
if(cbi == cb)
{
_dosomethingcallbacks.erase(_dosomethingcallbacks.begin()+i);
break;
}
i++;
}
}
void ServerInterface::_server_thread_work()
{
// Create interpreter
py::scoped_interpreter guard{};
// Import our script server.py
py::module_ serv_int = py::module_::import("server");
// Create server
py::object result = serv_int.attr("createServerOnIP")("192.168.0.10");
assert(result.cast<bool>());
// Set callback handler <------ this is ESSENTIAL to communicate with OUR app and not ... well ... nothing
result = serv_int.attr("setCallbackHandler")(this);
assert(result.cast<bool>());
// Server work
while(!_stop_thread)
{
// Check new data
serv_int.attr("handle_request")();
// Wait a bit (server is not actively used, will give some more time to other threads)
std::this_thread::sleep_for(std::chrono::milliseconds(50));
}
}
- The most important line is
serv_int.attr("setCallbackHandler")(this);, where we are setting a pointer to THIS interface for callbacks. - In your main code, call
ServerInterface::getInstance().registerDoSomething()and pass it any function compatible with the callback signature (inserverinterface_py.h) to be processed on callback.
ServerInterface_py
Header file :
#ifndef SERVERINTERFACE_PY_H
#define SERVERINTERFACE_PY_H
#pragma push_macro("slots")
#undef slots
#include "pybind11/include/pybind11/pybind11.h"
#include "pybind11/include/pybind11/embed.h"
#pragma pop_macro("slots")
namespace py = pybind11;
// Callbacks defs for server interaction (C++ side)
typedef void (*DoSomethingCallback)();
// To be able to implement our pointer
class ServerInterface;
class ServerInterface_py
{
public:
ServerInterface_py();
/**
* @brief Launches all callbacks
* @return 1 if had an interface set
*/
bool onDoSomething();
public:
// Pointer to our APP interface, could be set as private and provide setters and getters
ServerInterface* servint_ptr;
};
#endif // SERVERINTERFACE_PY_H
#include "serverinterface_py.h"
#include "serverinterface.h"
ServerInterface_py::ServerInterface_py() : servint_ptr(nullptr) {}
bool ServerInterface_py::onDoSomething()
{
// If no interface reference (our APP), go back
if(servint_ptr == nullptr) { return 0; }
for(ScanBeginCallback& cb : servint_ptr->_scanbegincallbacks)
{
cb();
}
return 1;
}
PythonBindings
#include "../serverinterface_py.h"
#include "../serverinterface.h"
PYBIND11_MODULE(xmlrpc_interface, m)
{
// Doc for help(xmlrpc_interface)
m.doc() = R"pbdoc(
XML-RPC Server Interface
-------------------------
Interface Python XML-RPC server with C++ code
)pbdoc";
// Expose class to Python
// We NEED to expose this class - since we use a pointer to it, a crash would occur calling serv_int.attr("setCallbackHandler")(this); as Python would not know anything about it
// But we do not need to expose anything, since we are just setting a pointer to another pointer
py::class_<ServerInterface>(m,"ServInterface");
// True interface for Python
py::class_<ServerInterface_py>(m,"ServerInterface")
.def(py::init<>())
.def("onDoSomething",&ServerInterface_py::onDoSomething,"Inform C++ that we received the go to doSomething from server")
.def_readwrite("servint_ptr", &ServerInterface_py::servint_ptr, py::return_value_policy::reference); // we will change it inside Python, so expose it
// note that we pass by reference, so Python and C++ pointers are the same
}
servint_ptr (and so to pass it through a function), we muste declare it. Since we don't use it else than replacing it with another one, we expose none of its elements.
Server.py
Finally, we have our Python (3) server :
# Launch an XML-RPC server to handle URCap commands
import socket
from xmlrpc.server import SimpleXMLRPCServer
from socketserver import ThreadingMixIn
# Our own module
from xmlrpc_interface import ServerInterface
#Our multi-threaded XML-RPC server
class MultithreadedSimpleXMLRPCServer(ThreadingMixIn, SimpleXMLRPCServer):
pass
# XML-RPC server
server = None
# Interface to communicate with C++, using our module
serv_int = ServerInterface()
def createServerOnIP(ip):
"""Create server on given IP
Returns:
If server was created successfully
"""
global server
server = MultithreadedSimpleXMLRPCServer((ip, 30153), logRequests=False)
return initServer(server)
def handle_request():
"""Handles request for XML-RPC server (should be called in a specific C++ thread)
Returns:
None
"""
assert server is not None
server.handle_request()
# Helpers
def initServer(server):
"""Init server while exposing needed functions
Note:
to be called from one of the createServerxxx functions
Returns:
If could create
"""
server.RequestHandlerClass.protocol_version = "HTTP/1.1"
server.register_function(heartbeat, "heartbeat")
server.register_function(set_title, "set_title")
server.register_function(get_title, "get_title")
server.register_function(start_scan, "start_scan")
server.register_function(stop_scan, "stop_scan")
server.register_function(scan_message, "scan_message")
return server is not None
def getIPs():
"""Print local IPs
Returns:
IP list
"""
for ip in socket.gethostbyname_ex(socket.gethostname())[-1]:
print(ip)
return socket.gethostbyname_ex(socket.gethostname())[-1]
# Server functions
def setCallbackHandler(handler):
"""Set C++ callback handler
Note:
Required for correct callbacks
Returns:
True
"""
serv_int.servint_ptr = handler;
return True
def onDoSomething():
"""C++ / Python communication through callback interface when doSomething is received from any XML-RPC client
Returns:
If callback succeeded
"""
return serv_int.test()
pybind11_add_module(xmlrpc_interface serverinterface_py.h serverinterface.h serverinterface_py.cpp serverinterface.cpp pythonbindings.cpp)
Threading cons
- The previous chapter introduced a way to get callbacks to our running application.
- A problem persists : since our server (i.e. Python script + C++ server function) is running in a separate thread, the sole working callbacks will be static functions, or members functions of objects EXISTING inside our server thread.
- If you would like to update your UI on a callback, such callback is not compatible. It is necessary to code some system to communicate around our threads.
Message queue
- If you are not using a framework offering some asynchronous call system (a main loop running for a thread, executing events), you may want to go with standard libraries.
-
The following is a pseudo-recipe that you may use to implement such system :
-
Modify the callback registering system to support the form
std::function<>(e.g. transformvoid (*MyCallback)(int,int)tostd::function<void(int,int)>- Remember to modify the containers
std::vector<>to support it - The main goal here is to be able to have callbacks on specific objects
- Remember to modify the containers
- When you register your member class function, you can now use
std::bind(&Class::yourFunction, &ClassObject, _1, _2 ... /*if fct has args*/)to pass to the registering system while in your object- The bind will create a wrapper compatible with the
std::function<>embedding the object, such that we now know which function from which object is called - If you stop here, and tries to call those on the server callback, you will certainly encounter a crash, since we are trying to call objects that do not belong to the server thread.
- The bind will create a wrapper compatible with the
- You now have some kind of queue to be able to use the callbacks from main thread
- A vector is not the best : you may need to protect the data from simultaneous R/W, manage to remove already processed callbacks ...
std::queueis a bir better in such that you canpush()on server side andpop()on main thread side (so you will not read the same event twice), but it is not thread-safe either. You may want to extend it to add some mutex onpush()andpop()methods.
- Whenever your thread-safe queue is created, your last step is to periodically check it from the main thread (thanks to a timer, once in a while in the main event loop ...).
Qt callback mechanism
- Using Qt
emit, we have an already existing callback system. The problem being that PyBind11 seems hard to integrate with classes implementing Qt specific elements. -
The previous example has been modified such that the
ServerInterfaceis never referenced by the binding nor theserverinterface_py. -
After our server as handled a request, we are reading a vector of custom
PY_MSGScontaining which callback we had (and potential extra args) - As we do not set the specific
ServerInterfacethrough a pointer anymore (since our own C++ code will handle to dispatch the messages), we can add Qt elements to ourServerInterface - We can now
emitspecific signals (and so any object can directlyQObject::connect()to it).
The modified example is implemented in another project available on our GitLab.
