C++ - Python Binding#

Introduction#
Python.h
thanks to macros, and permit cross-language communication.Integrate (CMake)#
You can embed the library in two different ways :
By installing PyBind11 in local, and referencing it inside CMakeLists.txt with
find_package(pybind11 REQUIRED)
By embedding it as a subfolder using
add_subdirectory(pybind11)
(you will need to clone the whole repo and put in in your project, but prefer official releases found in the GitHub)
1project(MyProject)
2
3# Add PyBind11 as subfolder
4add_subdirectory(pybind11)
5# Add PyBind11 as installed somewhere on disk
6# find_package(pybind11 REQUIRED)
7
8# C++ app files
9file(GLOB PROJECT_SOURCES
10 "main.cpp"
11 # ...
12)
13# Exec
14add_executable(MyProject ${PROJECT_SOURCES})
15# To embed interpreter and use basic modules
16target_link_libraries(MyProject PRIVATE pybind11::embed pybind11::module)
17
18# Our generated Python lib (see below chapters)
19pybind11_add_module(MyPYModule pythonbindings.cpp MyModule.h MyModule.cpp) # all files required by your extension
20
21# Neat little trick if you have some Python script in your project tree that you'd like to access with the embedded interpreter
22# The file will be copied each time (build, run ...) so you can edit it freely and not worry to have the latest version with your app
23add_custom_target(copyPyScript ALL
24 COMMAND ${CMAKE_COMMAND} -E copy
25 ${CMAKE_SOURCE_DIR}/which/file/tocopy.py
26 ${CMAKE_CURRENT_BINARY_DIR}
27)
28add_dependencies(MyProject copyPyScript)
Note
For the following chapters, we assume PyBind11 as a subfolder. Examples are taken from a custom Qt app.
Importing PyBind11 (frameworks using slots
keyword)#
slots
in quite a few places. Qt will not be happy for example, as itself uses this keyword for its Signal and Slots callback mechanism.1// Cross names error with Qt - disable slots while including required files
2#pragma push_macro("slots")
3#undef slots
4#include "pybind11/include/pybind11/pybind11.h"
5#include "pybind11/include/pybind11/embed.h"
6#pragma pop_macro("slots")
Another C++ specific declaration is used to link the pybind11
namespace to a more friendly and simple py
namespace :
namespace py = pybind11; // creates a namespace alias
Warning
using namespace pybind11;
.py::
leading before Python-specific declarations.Calling Python from C++#
#include "pybind11/include/pybind11/embed.h"
is required.mymodule.py
that we may use along with some other code :1def printHello(boolMirrorInput):
2 print("Hello")
3 return boolMirrorInput
And an embedded module such as (see further chapters) :
1PYBIND11_EMBEDDED_MODULE(mymodule_embedded, m)
2{
3 // `m` is a `py::module_` which is used to bind functions and classes
4 m.def("printHello", [](bool mirrorinput)
5 {
6 std::cout << "Hello from Cpp !"" << std::endl;
7 return mirrorinput;
8 });
9}
Afterward, where the interpreter is needed, you can use (code explained below) :
1// Do some Python in C++
2static void doPython()
3{
4 // Function is not thread safe - only one thread can call it at a time - should mutex lock it
5
6 // 1) Start the interpreter and keep it alive
7 py::scoped_interpreter guard{};
8
9 // 2) Some Python code
10 // Simple
11 py::print("Simple line hello");
12 // More advanced
13 py::exec
14 (
15 R"(
16 kwargs = dict(name="advanced Python", avalue=3)
17 message = "Hello, {name}! I am {avalue}".format(**kwargs)
18 print(message)
19 )"
20 );
21
22 // 3) Import a Python module, 'sys' here
23 py::module_ sys_module = py::module_::import("sys");
24 // Get the max size of an integer
25 py::object result = sys_module.attr("maxsize")();
26
27 // 4) Import our own Python script (module name is name of the file.py)
28 py::module_ my_module = py::module_::import("mymodule");
29 // Call one of the function
30 result = my_module.attr("printHello")(true);
31 assert(result.cast<bool>()); // if not true, function did not return its input
32
33 // 5) Use our embedded module
34 py::module_ emb_module = py::module_::import("mymodule_embedded");
35 result = emb_module.attr("printHello")(false);
36 assert(!result.cast<bool>()); // if not false, function did not return its input
37}
(1) Guard object#
guard
object will start the interpreter and keep it running WHILE THE OBJECT IS ALIVE.Warning
CPython
api, but with caveats to consider.Note
In C++, the scope strongly depends on the placement of the variable.
A static var in a
.cpp
file may always existA static var as a member of a class may only appear the first time an object of said class is instantiated
A pointer will start to live on the
new
command and finish on thedelete
oneA local variable will end after the function/if/do/for … statement scope is finished
With this in head, you can create localized instantiation of the object to use during a small part of a function (e.g. if you use the interpreter in a multithreaded app, where multiple threads could want to use the interpreter (but remember to have a mutex, since only one guard can exist at a time !)), thanks to the syntax :
1void myfunc()
2{
3 // Do something
4 // ...
5
6 // Oh, I need Python !
7 // TODO : lock your mutex
8 {
9 py::scoped_interpreter guard{};
10 // do Python
11 }
12 // scope of guard is finished -> interpreter is terminated
13 // TODO : unlock your mutex
14}
{}
without any prior statement like an if
or so, that is how C++ works !if(condition) dosomething;
on one line works like an if
with multiple lines inside brackets : the if
will execute what he sees next - the dosomething
for the one-line version, and the brackets (a local scope where lies multiple lines of code) for the if(){}
version.(2) Python code in C++#
You can use some built-in functions like py::print()
for simple code, or even exec multiple lines of code with py::exec()
, py::eval()
…
(3) Local Python modules#
A module is easily imported with the command
py::module_ mymodule = py::module_::import(char* modulename);
.- Afterward, you get a result calling the function by its name, through the previously imported module :
py::object result = sys_module.attr("maxsize")();
.If the function has no argument, let second parenthesis empty, else fill with the required arguments. You must know the type of returned object to be able to cast it back to its C++ version :
int res = result.cast<int>()
.
Warning
Your code may crash for multiple reasons, giving you a hard-time finding the problem :
- On the acquisition of the guard :
if another guard object already exists
- On the import of a module :
if the module does not exist
if the module is not valid (Python would itself be unhappy trying to run this script)
if some dependencies cannot be found
- On the execution of a function :
if the function does not exist
if the passed arguments count does not correspond to the function (if no default arguments exist, they MUST be filled even if those are not useful for the function)
if the type of the arguments mismatch
if you try to pass pointers/objects of some C++ class that Python is not aware of, even if you just intend to stock them to further re-use with C++ code (like a Python server using callbacks on C++ code)
- On the cast of a result :
if you try to cast to an unknown type for Python (e.g. one of your class pointer)
if you try to cast to the wrong type
if you try to cast the result from a miscalled function (calling
result = sys_module.attr("maxsize");
instead ofresult = sys_module.attr("maxsize")();
will not crash on this line but on a further cast, as we did not really call the function)
(4) Local script#
script.py
solely needs to lie near your .exe
or be added to the Python sys
path. The calls are the same as before.my_superScript_pyp.y.py
, you would need to use py::module_::import("my_superScript_pyp.y");
).(5) Embedded module#
PYBIND11_EMBEDDED_MODULE
name..exe
inside your build folders.Using C++ from Python#
Creating a module#
.hpp
format for easier read). 1// cppcode.hpp
2#ifndef CPPCODE_HPP
3#define CPPCODE_HPP
4
5#include <iostream>
6#include <string>
7
8class SomeCppCode
9{
10 public:
11 static void say(std::string s)
12 {
13 std::cout << s << std::endl;
14 }
15
16 static int add(int a, int b)
17 {
18 return a + b;
19 }
20
21 private:
22 SomeCppCode() {}; // cannot instantiate
23}
24
25#endif // CPPCODE_HPP
1// pythoncalls.hpp
2#ifndef PYTHONCALLS_HPP
3#define PYTHONCALLS_HPP
4
5#pragma push_macro("slots")
6#undef slots
7#include "pybind11/include/pybind11/pybind11.h"
8#include "pybind11/include/pybind11/embed.h"
9#pragma pop_macro("slots")
10
11namespace py = pybind11;
12
13#include "cppcode.hpp"
14
15class PythonCalls
16{
17 public:
18 PythonCalls() : _last_a(0), _last_b(0), _last_said("") {}
19
20 void callSay(std::string s, copyToLast = True)
21 {
22 SomeCppCode::say(s);
23 if(copyToLast)
24 {
25 _last_said = s;
26 }
27 }
28
29 void callSayAgain()
30 {
31 callSay(_last_said, False);
32 }
33
34 int callAdd(int a, int b, copyToLast = True)
35 {
36 if(copyToLast)
37 {
38 _last_a = a;
39 _last_b = b;
40 }
41 return SomeCppCode::add(a,b);
42 }
43
44 int callAddFromInternal()
45 {
46 return callAdd(_last_a, _last_b, False);
47 }
48
49 public:
50 int _last_a, _last_b;
51
52 private:
53 std::string _last_said;
54}
55
56#endif // PYTHONCALLS_HPP
1// pythonbindings.hpp
2#ifndef PYTHONBINDING_HPP
3#define PYTHONBINDING_HPP
4
5#include "pythoncalls.hpp"
6
7PYBIND11_MODULE(MyCustomModule, m) // m is the Python module where code will lie
8{
9 // Doc for help(MyCustomModule)
10 m.doc() = R"pbdoc(
11 My Python module
12 -------------------------
13 Contains a class capable of saying things and adding, while interacting through C++ code
14 )pbdoc";
15
16 // Expose class to Python - we only have one
17 py::class_<PythonCalls>(m,"MagicCpp")
18 .def(py::init<>()) // args would go inside <> if existed
19 .def("say",&PythonCalls::callSay,"Say what we tell him to", py::arg("message"), py::arg(copytolast) = True)
20 .def("sayAgain",&PythonCalls::callSayAgain,"Say again what we last said")
21 .def("add",&PythonCalls::callAdd,"Add our two vars", py::arg("a"), py::arg("b"), py::arg(copytolast) = True)
22 .def("addFromInternal",&PythonCalls::callAddFromInternal,"Add the two vars currently inside object")
23 // Expose vars
24 .def_readwrite("last_a", &PythonCalls::_last_a, py::return_value_policy::reference);
25 .def_readwrite("last_b", &PythonCalls::_last_a, py::return_value_policy::reference);
26
27 // .def_readwrite("last_str", &PythonCalls::_last_said, py::return_value_policy::reference) <- illegal, is private so unreachable (same for constructor that MUST exist and be public)
28}
29
30#endif // PYTHONBINDING_HPP
MyCustomModule
.m.doc()
allows to create a documentation for whenever we would call help(MyCustomModule).py::class_<>(m,name)
expose our C++ class to Python with given name.public
, else the binding will fail.For functions, note that we must specify all the args. For already defined args in C++, we must re-set them here since PyBind11 cannot access those.
For parameters, note the return policy. This one is very important, since it can :
Make a copy in Python, so we have two separate entities (one in C++ side, one in Python) (we won’t pollute C++ by changing the values in Python, but changes would not reflect as we try to do here)
Create a reference, so Python and C++ vars are tied together
The policies can also tell which side is managing the destruction of the object (using pointers for example).
.def
exist following the variable type (if const, static …) with .def_readonly
, .def_readonly_static
, .def_readwrite_static
….cpp
file, only referenced in CMake. Else, we could quickly have problems like “already defined” errors as it would be redefined by some includes.1# Our generated Python lib
2pybind11_add_module(MyCustomModule pythonbindings.hpp pythoncalls.hpp) # all files required by your extension
And voilà ! You can now use your module in Python with the generated extension, or thanks to the embedded interpreter (even if calling the C++ code directly may be a better choice if no reason justify passing through Python).
In your script, you can simply do :
1from MyCustomModule import MagicCpp
2
3mc = MagicCpp()
4mc.say("Hello");
5mc.sayAgain();
6mc.add(3,5);
7mc.addFromInternal();
8mc.last_a = 1;
9mc.addFromInternal();
10# mc.last_str = "hoy"; <- crash, last_str is unknown
11# mc.a_new_property = 3; <- crash, new parameters are not authorized
12# For the previous line to work, the PYBIND11_EMBEDDED_MODULE class definition must be changed from py::class_<PythonCalls>(m,"MagicCpp") to py::class_<PythonCalls>(m,"MagicCpp", py::dynamic_attr())
Mindset#
A
serverinterface.h/.cpp
: the layer between our app and the C++ Python callback system, sole able to call callbacks from itA
serverinterface_py.h/.cpp
: the callback system used by Python and communicating with C++A
pythonbindings.cpp
: the elements exposed to Python
ServerInterface#
Header file :
1#ifndef SERVERINTERFACE_H
2#define SERVERINTERFACE_H
3
4#include <string>
5#include <vector>
6#include <thread>
7#include <atomic>
8
9#include "server/serverinterface_py.h"
10
11class ServerInterface
12{
13 // Allows binding to access private functions
14 friend class ServerInterface_py;
15
16public:
17 ~ServerInterface();
18 // Singleton-related functions
19 static ServerInterface& getInstance();
20private:
21 ServerInterface();
22 ServerInterface(ServerInterface const&);
23 void operator=(ServerInterface const&);
24
25
26public:
27 /**
28 * @brief Callbacks when server asks to do something
29 * @param cb callback
30 */
31 void registerDoSomething(DoSomethingCallback cb);
32 /**
33 * @brief Remove callback from list
34 * @param cb callback
35 */
36 void removeDoSomething(DoSomethingCallback cb);
37
38private:
39 // Where our python script will live
40 void _server_thread_work();
41 // To stop our thread
42 volatile std::atomic_bool _stop_thread;
43
44private:
45 // All registered callbacks
46 std::vector<DoSomethingCallback> _dosomethingcallbacks;
47 // Our python script
48 std::thread _server_thread;
49};
50
51#endif // SERVERINTERFACE_H
Code file :
1#include "serverinterface.h"
2#include <chrono>
3
4ServerInterface::ServerInterface() : _stop_thread(0)
5{
6 // At instanciation, launches the server on a separate thread
7 _server_thread = std::thread(&ServerInterface::_server_thread_work, this);
8 // No join (we would block this thread while the server is not closed) nor detach (should not be used, memleak and no use)
9}
10
11ServerInterface::~ServerInterface()
12{
13 // Removes server thread
14 _stop_thread = 1;
15 _server_thread.join(); // will wait for the thread to terminate
16}
17
18ServerInterface &ServerInterface::getInstance()
19{
20 static ServerInterface inst;
21 return inst;
22}
23
24void ServerInterface::registerDoSomething(DoSomethingCallback cb)
25{
26 _dosomethingcallbacks.push_back(cb);
27}
28
29void ServerInterface::removeDoSomething(DoSomethingCallback cb)
30{
31 uint8_t i = 0;
32 for(DoSomethingCallback& cbi : _dosomethingcallbacks)
33 {
34 if(cbi == cb)
35 {
36 _dosomethingcallbacks.erase(_dosomethingcallbacks.begin()+i);
37 break;
38 }
39 i++;
40 }
41}
42
43void ServerInterface::_server_thread_work()
44{
45 // Create interpreter
46 py::scoped_interpreter guard{};
47 // Import our script server.py
48 py::module_ serv_int = py::module_::import("server");
49 // Create server
50 py::object result = serv_int.attr("createServerOnIP")("192.168.0.10");
51 assert(result.cast<bool>());
52 // Set callback handler <------ this is ESSENTIAL to communicate with OUR app and not ... well ... nothing
53 result = serv_int.attr("setCallbackHandler")(this);
54 assert(result.cast<bool>());
55
56 // Server work
57 while(!_stop_thread)
58 {
59 // Check new data
60 serv_int.attr("handle_request")();
61 // Wait a bit (server is not actively used, will give some more time to other threads)
62 std::this_thread::sleep_for(std::chrono::milliseconds(50));
63 }
64}
serv_int.attr("setCallbackHandler")(this);
, where we are setting a pointer to THIS interface for callbacks.ServerInterface::getInstance().registerDoSomething()
and pass it any function compatible with the callback signature (in serverinterface_py.h
) to be processed on callback.ServerInterface_py#
Header file :
1#ifndef SERVERINTERFACE_PY_H
2#define SERVERINTERFACE_PY_H
3
4#pragma push_macro("slots")
5#undef slots
6#include "pybind11/include/pybind11/pybind11.h"
7#include "pybind11/include/pybind11/embed.h"
8#pragma pop_macro("slots")
9
10namespace py = pybind11;
11
12// Callbacks defs for server interaction (C++ side)
13typedef void (*DoSomethingCallback)();
14// To be able to implement our pointer
15class ServerInterface;
16
17class ServerInterface_py
18{
19 public:
20 ServerInterface_py();
21
22 /**
23 * @brief Launches all callbacks
24 * @return 1 if had an interface set
25 */
26 bool onDoSomething();
27
28 public:
29 // Pointer to our APP interface, could be set as private and provide setters and getters
30 ServerInterface* servint_ptr;
31};
32
33#endif // SERVERINTERFACE_PY_H
Code file :
1#include "serverinterface_py.h"
2#include "serverinterface.h"
3
4ServerInterface_py::ServerInterface_py() : servint_ptr(nullptr) {}
5
6bool ServerInterface_py::onDoSomething()
7{
8 // If no interface reference (our APP), go back
9 if(servint_ptr == nullptr) { return 0; }
10
11 for(ScanBeginCallback& cb : servint_ptr->_scanbegincallbacks)
12 {
13 cb();
14 }
15 return 1;
16}
Nothing fancy, simple callback system. The solely thing to note is the pointer to the interface that allows to link the callbacks to our application.
PythonBindings#
1#include "../serverinterface_py.h"
2#include "../serverinterface.h"
3
4PYBIND11_MODULE(xmlrpc_interface, m)
5{
6 // Doc for help(xmlrpc_interface)
7 m.doc() = R"pbdoc(
8 XML-RPC Server Interface
9 -------------------------
10 Interface Python XML-RPC server with C++ code
11 )pbdoc";
12
13 // Expose class to Python
14 // We NEED to expose this class - since we use a pointer to it, a crash would occur calling serv_int.attr("setCallbackHandler")(this); as Python would not know anything about it
15 // But we do not need to expose anything, since we are just setting a pointer to another pointer
16 py::class_<ServerInterface>(m,"ServInterface");
17 // True interface for Python
18 py::class_<ServerInterface_py>(m,"ServerInterface")
19 .def(py::init<>())
20 .def("onDoSomething",&ServerInterface_py::onDoSomething,"Inform C++ that we received the go to doSomething from server")
21 .def_readwrite("servint_ptr", &ServerInterface_py::servint_ptr, py::return_value_policy::reference); // we will change it inside Python, so expose it
22 // note that we pass by reference, so Python and C++ pointers are the same
23}
Comments are self-explanatory. We cannot pass an argument of undeclared type to Python. Since we intend to modify the servint_ptr
(and so to pass it through a function), we muste declare it. Since we don’t use it else than replacing it with another one, we expose none of its elements.
Server.py#
Finally, we have our Python (3) server :
1# Launch an XML-RPC server to handle URCap commands
2import socket
3from xmlrpc.server import SimpleXMLRPCServer
4from socketserver import ThreadingMixIn
5# Our own module
6from xmlrpc_interface import ServerInterface
7
8#Our multi-threaded XML-RPC server
9class MultithreadedSimpleXMLRPCServer(ThreadingMixIn, SimpleXMLRPCServer):
10 pass
11
12# XML-RPC server
13server = None
14# Interface to communicate with C++, using our module
15serv_int = ServerInterface()
16
17def createServerOnIP(ip):
18 """Create server on given IP
19
20 Returns:
21 If server was created successfully
22 """
23 global server
24 server = MultithreadedSimpleXMLRPCServer((ip, 30153), logRequests=False)
25 return initServer(server)
26
27
28def handle_request():
29 """Handles request for XML-RPC server (should be called in a specific C++ thread)
30
31 Returns:
32 None
33 """
34 assert server is not None
35 server.handle_request()
36
37
38# Helpers
39
40
41def initServer(server):
42 """Init server while exposing needed functions
43
44 Note:
45 to be called from one of the createServerxxx functions
46 Returns:
47 If could create
48 """
49 server.RequestHandlerClass.protocol_version = "HTTP/1.1"
50 server.register_function(heartbeat, "heartbeat")
51 server.register_function(set_title, "set_title")
52 server.register_function(get_title, "get_title")
53 server.register_function(start_scan, "start_scan")
54 server.register_function(stop_scan, "stop_scan")
55 server.register_function(scan_message, "scan_message")
56 return server is not None
57
58
59def getIPs():
60 """Print local IPs
61
62 Returns:
63 IP list
64 """
65 for ip in socket.gethostbyname_ex(socket.gethostname())[-1]:
66 print(ip)
67 return socket.gethostbyname_ex(socket.gethostname())[-1]
68
69
70# Server functions
71
72
73def setCallbackHandler(handler):
74 """Set C++ callback handler
75
76 Note:
77 Required for correct callbacks
78 Returns:
79 True
80 """
81 serv_int.servint_ptr = handler;
82 return True
83
84
85def onDoSomething():
86 """C++ / Python communication through callback interface when doSomething is received from any XML-RPC client
87
88 Returns:
89 If callback succeeded
90 """
91 return serv_int.test()
The CMake line is the following :
pybind11_add_module(xmlrpc_interface serverinterface_py.h serverinterface.h serverinterface_py.cpp serverinterface.cpp pythonbindings.cpp)
Threading cons#
Message queue#
- Modify the callback registering system to support the form
std::function<>
(e.g. transformvoid (*MyCallback)(int,int)
tostd::function<void(int,int)>
Remember to modify the containers
std::vector<>
to support itThe main goal here is to be able to have callbacks on specific objects
- Modify the callback registering system to support the form
- When you register your member class function, you can now use
std::bind(&Class::yourFunction, &ClassObject, _1, _2 ... /*if fct has args*/)
to pass to the registering system while in your object The bind will create a wrapper compatible with the
std::function<>
embedding the object, such that we now know which function from which object is calledIf you stop here, and tries to call those on the server callback, you will certainly encounter a crash, since we are trying to call objects that do not belong to the server thread.
- When you register your member class function, you can now use
- You now have some kind of queue to be able to use the callbacks from main thread
A vector is not the best : you may need to protect the data from simultaneous R/W, manage to remove already processed callbacks …
std::queue
is a bir better in such that you canpush()
on server side andpop()
on main thread side (so you will not read the same event twice), but it is not thread-safe either. You may want to extend it to add some mutex onpush()
andpop()
methods.
Whenever your thread-safe queue is created, your last step is to periodically check it from the main thread (thanks to a timer, once in a while in the main event loop …).
Qt callback mechanism#
emit
, we have an already existing callback system. The problem being that PyBind11 seems hard to integrate with classes implementing Qt specific elements.ServerInterface
is never referenced by the binding nor the serverinterface_py
.After our server as handled a request, we are reading a vector of custom
PY_MSGS
containing which callback we had (and potential extra args)As we do not set the specific
ServerInterface
through a pointer anymore (since our own C++ code will handle to dispatch the messages), we can add Qt elements to ourServerInterface
We can now
emit
specific signals (and so any object can directlyQObject::connect()
to it).
The modified example is implemented in another project available on our GitLab.