C++ - Python Binding#

C++ / Python logos

Introduction#

Binding is the act of creating a bridge between two pieces of code to interface between different OS or programming languages.
The following will present binding between C++ and Python, and how to embed a Python interpreter inside C++ code.

We will use PyBind11 for this purpose.
The doc is available under here.
The library consists of multiple wrappers around Python.h thanks to macros, and permit cross-language communication.

Integrate (CMake)#

You can embed the library in two different ways :

  1. By installing PyBind11 in local, and referencing it inside CMakeLists.txt with find_package(pybind11 REQUIRED)

  2. By embedding it as a subfolder using add_subdirectory(pybind11) (you will need to clone the whole repo and put in in your project, but prefer official releases found in the GitHub)

Then, you can generate a Python extension file including code written in C++ (see further chapters). It will also be available for your embedded interpreter.
The file will look as follow :
 1project(MyProject)
 2
 3# Add PyBind11 as subfolder
 4add_subdirectory(pybind11)
 5# Add PyBind11 as installed somewhere on disk
 6# find_package(pybind11 REQUIRED)
 7
 8# C++ app files
 9file(GLOB PROJECT_SOURCES
10    "main.cpp"
11    # ...
12)
13# Exec
14add_executable(MyProject ${PROJECT_SOURCES})
15# To embed interpreter and use basic modules
16target_link_libraries(MyProject PRIVATE pybind11::embed pybind11::module)
17
18# Our generated Python lib (see below chapters)
19pybind11_add_module(MyPYModule pythonbindings.cpp MyModule.h MyModule.cpp) # all files required by your extension
20
21# Neat little trick if you have some Python script in your project tree that you'd like to access with the embedded interpreter
22# The file will be copied each time (build, run ...) so you can edit it freely and not worry to have the latest version with your app
23add_custom_target(copyPyScript ALL
24    COMMAND ${CMAKE_COMMAND} -E copy
25    ${CMAKE_SOURCE_DIR}/which/file/tocopy.py
26    ${CMAKE_CURRENT_BINARY_DIR}
27)
28add_dependencies(MyProject copyPyScript)

Note

For the following chapters, we assume PyBind11 as a subfolder. Examples are taken from a custom Qt app.

Importing PyBind11 (frameworks using slots keyword)#

PyBind11 uses the keyword slots in quite a few places. Qt will not be happy for example, as itself uses this keyword for its Signal and Slots callback mechanism.
To import without worry, use the following lines that will temporarily disable the macro while importing PyBind11 :
1// Cross names error with Qt - disable slots while including required files
2#pragma push_macro("slots")
3#undef slots
4#include "pybind11/include/pybind11/pybind11.h"
5#include "pybind11/include/pybind11/embed.h"
6#pragma pop_macro("slots")

Another C++ specific declaration is used to link the pybind11 namespace to a more friendly and simple py namespace :

namespace py = pybind11; // creates a namespace alias

Warning

Avoid the declaration using namespace pybind11;.
While you could more easily use PyBind11 related objects, you will pollute your namespace and may create errors.
Also, it is easier to read code with py:: leading before Python-specific declarations.

Calling Python from C++#

To be able to use Python in C++, the #include "pybind11/include/pybind11/embed.h" is required.
Let’s say we have the following script called mymodule.py that we may use along with some other code :
1def printHello(boolMirrorInput):
2    print("Hello")
3    return boolMirrorInput

And an embedded module such as (see further chapters) :

1PYBIND11_EMBEDDED_MODULE(mymodule_embedded, m)
2{
3    // `m` is a `py::module_` which is used to bind functions and classes
4    m.def("printHello", [](bool mirrorinput)
5    {
6        std::cout << "Hello from Cpp !"" << std::endl;
7        return mirrorinput;
8    });
9}

Afterward, where the interpreter is needed, you can use (code explained below) :

 1// Do some Python in C++
 2static void doPython()
 3{
 4    // Function is not thread safe - only one thread can call it at a time - should mutex lock it
 5
 6    // 1) Start the interpreter and keep it alive
 7    py::scoped_interpreter guard{};
 8
 9    // 2) Some Python code
10        // Simple
11    py::print("Simple line hello");
12        // More advanced
13    py::exec
14    (
15        R"(
16            kwargs = dict(name="advanced Python", avalue=3)
17            message = "Hello, {name}! I am {avalue}".format(**kwargs)
18            print(message)
19        )"
20    );
21
22    // 3) Import a Python module, 'sys' here
23    py::module_ sys_module = py::module_::import("sys");
24        // Get the max size of an integer
25    py::object result = sys_module.attr("maxsize")();
26
27    // 4) Import our own Python script (module name is name of the file.py)
28    py::module_ my_module = py::module_::import("mymodule");
29        // Call one of the function
30    result = my_module.attr("printHello")(true);
31    assert(result.cast<bool>()); // if not true, function did not return its input
32
33    // 5) Use our embedded module
34    py::module_ emb_module = py::module_::import("mymodule_embedded");
35    result = emb_module.attr("printHello")(false);
36    assert(!result.cast<bool>()); // if not false, function did not return its input
37}

(1) Guard object#

The guard object will start the interpreter and keep it running WHILE THE OBJECT IS ALIVE.
When the object is destroyed (whenever its scope ends), the interpreter cannot be used anymore.

Warning

Only ONE guard can be used at a time, otherwise a crash will occur.
It would be possible to run multiple interpreters through the CPython api, but with caveats to consider.

Note

In C++, the scope strongly depends on the placement of the variable.

  • A static var in a .cpp file may always exist

  • A static var as a member of a class may only appear the first time an object of said class is instantiated

  • A pointer will start to live on the new command and finish on the delete one

  • A local variable will end after the function/if/do/for … statement scope is finished

With this in head, you can create localized instantiation of the object to use during a small part of a function (e.g. if you use the interpreter in a multithreaded app, where multiple threads could want to use the interpreter (but remember to have a mutex, since only one guard can exist at a time !)), thanks to the syntax :

 1void myfunc()
 2{
 3    // Do something
 4    // ...
 5
 6    // Oh, I need Python !
 7    // TODO : lock your mutex
 8    {
 9        py::scoped_interpreter guard{};
10        // do Python
11    }
12    // scope of guard is finished -> interpreter is terminated
13    // TODO : unlock your mutex
14}
While it can seem strange to have {} without any prior statement like an if or so, that is how C++ works !
An if(condition) dosomething; on one line works like an if with multiple lines inside brackets : the if will execute what he sees next - the dosomething for the one-line version, and the brackets (a local scope where lies multiple lines of code) for the if(){} version.

(2) Python code in C++#

You can use some built-in functions like py::print() for simple code, or even exec multiple lines of code with py::exec(), py::eval()

(3) Local Python modules#

  1. A module is easily imported with the command py::module_ mymodule = py::module_::import(char* modulename);.

  2. Afterward, you get a result calling the function by its name, through the previously imported module : py::object result = sys_module.attr("maxsize")();.
    If the function has no argument, let second parenthesis empty, else fill with the required arguments.
  3. You must know the type of returned object to be able to cast it back to its C++ version : int res = result.cast<int>().

Warning

Your code may crash for multiple reasons, giving you a hard-time finding the problem :

  • On the acquisition of the guard :
    1. if another guard object already exists

  • On the import of a module :
    1. if the module does not exist

    2. if the module is not valid (Python would itself be unhappy trying to run this script)

    3. if some dependencies cannot be found

  • On the execution of a function :
    1. if the function does not exist

    2. if the passed arguments count does not correspond to the function (if no default arguments exist, they MUST be filled even if those are not useful for the function)

    3. if the type of the arguments mismatch

    4. if you try to pass pointers/objects of some C++ class that Python is not aware of, even if you just intend to stock them to further re-use with C++ code (like a Python server using callbacks on C++ code)

  • On the cast of a result :
    1. if you try to cast to an unknown type for Python (e.g. one of your class pointer)

    2. if you try to cast to the wrong type

    3. if you try to cast the result from a miscalled function (calling result = sys_module.attr("maxsize"); instead of result = sys_module.attr("maxsize")(); will not crash on this line but on a further cast, as we did not really call the function)

(4) Local script#

Your script.py solely needs to lie near your .exe or be added to the Python sys path. The calls are the same as before.
The module is imported following the name of the file (e.g. for my_superScript_pyp.y.py, you would need to use py::module_::import("my_superScript_pyp.y");).

(5) Embedded module#

You can use an embedded module as you would with any other module, simply giving its PYBIND11_EMBEDDED_MODULE name.
The generated Python extension is present near your .exe inside your build folders.

Using C++ from Python#

Creating a module#

You first need to build your C++ code that you will call from Python.
We will create a simple example that will decouple the binding, the Python calls, and the executed functions (they will be written as .hpp format for easier read).

First, the actual functions, pure C++ :
 1// cppcode.hpp
 2#ifndef CPPCODE_HPP
 3#define CPPCODE_HPP
 4
 5#include <iostream>
 6#include <string>
 7
 8class SomeCppCode
 9{
10    public:
11        static void say(std::string s)
12        {
13            std::cout << s << std::endl;
14        }
15
16        static int add(int a, int b)
17        {
18            return a + b;
19        }
20
21    private:
22        SomeCppCode() {}; // cannot instantiate
23}
24
25#endif // CPPCODE_HPP
Standard C++ code, no special consideration.
Then, we have the Python calls through C++ :
 1// pythoncalls.hpp
 2#ifndef PYTHONCALLS_HPP
 3#define PYTHONCALLS_HPP
 4
 5#pragma push_macro("slots")
 6#undef slots
 7#include "pybind11/include/pybind11/pybind11.h"
 8#include "pybind11/include/pybind11/embed.h"
 9#pragma pop_macro("slots")
10
11namespace py = pybind11;
12
13#include "cppcode.hpp"
14
15class PythonCalls
16{
17    public:
18        PythonCalls() : _last_a(0), _last_b(0), _last_said("") {}
19
20        void callSay(std::string s, copyToLast = True)
21        {
22            SomeCppCode::say(s);
23            if(copyToLast)
24            {
25                _last_said = s;
26            }
27        }
28
29        void callSayAgain()
30        {
31            callSay(_last_said, False);
32        }
33
34        int callAdd(int a, int b, copyToLast = True)
35        {
36            if(copyToLast)
37            {
38                _last_a = a;
39                _last_b = b;
40            }
41            return SomeCppCode::add(a,b);
42        }
43
44        int callAddFromInternal()
45        {
46            return callAdd(_last_a, _last_b, False);
47        }
48
49    public:
50        int _last_a, _last_b;
51
52    private:
53        std::string _last_said;
54}
55
56#endif // PYTHONCALLS_HPP
Some more C++ code, but note that we have public and private members, and only from standard, known types (no custom class pointer or so).
Finally, we create our Python module :
 1// pythonbindings.hpp
 2#ifndef PYTHONBINDING_HPP
 3#define PYTHONBINDING_HPP
 4
 5#include "pythoncalls.hpp"
 6
 7PYBIND11_MODULE(MyCustomModule, m) // m is the Python module where code will lie
 8{
 9    // Doc for help(MyCustomModule)
10    m.doc() = R"pbdoc(
11        My Python module
12        -------------------------
13        Contains a class capable of saying things and adding, while interacting through C++ code
14    )pbdoc";
15
16    // Expose class to Python - we only have one
17    py::class_<PythonCalls>(m,"MagicCpp")
18        .def(py::init<>()) // args would go inside <> if existed
19        .def("say",&PythonCalls::callSay,"Say what we tell him to", py::arg("message"), py::arg(copytolast) = True)
20        .def("sayAgain",&PythonCalls::callSayAgain,"Say again what we last said")
21        .def("add",&PythonCalls::callAdd,"Add our two vars", py::arg("a"), py::arg("b"), py::arg(copytolast) = True)
22        .def("addFromInternal",&PythonCalls::callAddFromInternal,"Add the two vars currently inside object")
23        // Expose vars
24        .def_readwrite("last_a", &PythonCalls::_last_a, py::return_value_policy::reference);
25        .def_readwrite("last_b", &PythonCalls::_last_a, py::return_value_policy::reference);
26
27        // .def_readwrite("last_str", &PythonCalls::_last_said, py::return_value_policy::reference) <- illegal, is private so unreachable (same for constructor that MUST exist and be public)
28}
29
30#endif // PYTHONBINDING_HPP
The module is named MyCustomModule.
m.doc() allows to create a documentation for whenever we would call help(MyCustomModule).
The py::class_<>(m,name) expose our C++ class to Python with given name.

We then expose all the different functions and parameters (note that we can give them Python names that do not correspond to the C++ names). They must be public, else the binding will fail.
  • For functions, note that we must specify all the args. For already defined args in C++, we must re-set them here since PyBind11 cannot access those.

  • For parameters, note the return policy. This one is very important, since it can :

    • Make a copy in Python, so we have two separate entities (one in C++ side, one in Python) (we won’t pollute C++ by changing the values in Python, but changes would not reflect as we try to do here)

    • Create a reference, so Python and C++ vars are tied together

    The policies can also tell which side is managing the destruction of the object (using pointers for example).

Without defining it, a default policy is set, but may not do what you think it will.
Different .def exist following the variable type (if const, static …) with .def_readonly, .def_readonly_static, .def_readwrite_static
You can also access private variables if you provide getters and setters.

You can also bind to anonymous functions (see first example).

The bindings should be put in a separate .cpp file, only referenced in CMake. Else, we could quickly have problems like “already defined” errors as it would be redefined by some includes.
They must be referenced only ONCE from the whole project for the linker to work.

Let’s modify our CMake where the module is created :
1# Our generated Python lib
2pybind11_add_module(MyCustomModule pythonbindings.hpp pythoncalls.hpp) # all files required by your extension

And voilà ! You can now use your module in Python with the generated extension, or thanks to the embedded interpreter (even if calling the C++ code directly may be a better choice if no reason justify passing through Python).

In your script, you can simply do :

 1from MyCustomModule import MagicCpp
 2
 3mc = MagicCpp()
 4mc.say("Hello");
 5mc.sayAgain();
 6mc.add(3,5);
 7mc.addFromInternal();
 8mc.last_a = 1;
 9mc.addFromInternal();
10# mc.last_str = "hoy"; <- crash, last_str is unknown
11# mc.a_new_property = 3; <- crash, new parameters are not authorized
12# For the previous line to work, the PYBIND11_EMBEDDED_MODULE class definition must be changed from py::class_<PythonCalls>(m,"MagicCpp") to py::class_<PythonCalls>(m,"MagicCpp", py::dynamic_attr())

Mindset#

It is very important to work with the correct mindset.
If you simply use binding to calls “static” functions (such as adding two numbers), the mindset is quite simple.

But if you want to go more advanced and interact with your running C++ application, it is not because it can do various things and embeds a Python interpreter that it means Python is aware of your app.
Let’s say that your Python code should send events to your app such that you implemented a simple callback system where Python calls a function in C++ that will further use some kind of pointer to interact with your code.
In your app you set said pointer, you run your interpreter, a callback is sent from Python, but sadly nothing changes in your C++ app.
Never we said to Python that we have an app running and which “pointer to do further things” we are using. In fact, we could run this code solely from Python, without our app, and it would not crash just because our app is not running.
Yes we are creating/using Python code from C++, but as a module (we never created “specific code for our application”).
It is necessary to decouple both. We first need to create some kind of interface between SOME C++ code and SOME Python code, and then tell Python WHICH is our specific code (through pointers for example).

Let’s review a more advanced system, written to interface C++ and Python to control our C++ application while Python is acting as an XML-RPC server.
We will have :
  • A serverinterface.h/.cpp : the layer between our app and the C++ Python callback system, sole able to call callbacks from it

  • A serverinterface_py.h/.cpp : the callback system used by Python and communicating with C++

  • A pythonbindings.cpp : the elements exposed to Python

ServerInterface#

Header file :

 1#ifndef SERVERINTERFACE_H
 2#define SERVERINTERFACE_H
 3
 4#include <string>
 5#include <vector>
 6#include <thread>
 7#include <atomic>
 8
 9#include "server/serverinterface_py.h"
10
11class ServerInterface
12{
13    // Allows binding to access private functions
14    friend class ServerInterface_py;
15
16public:
17    ~ServerInterface();
18    // Singleton-related functions
19    static ServerInterface& getInstance();
20private:
21    ServerInterface();
22    ServerInterface(ServerInterface const&);
23    void operator=(ServerInterface const&);
24
25
26public:
27    /**
28    * @brief Callbacks when server asks to do something
29    * @param cb callback
30    */
31    void registerDoSomething(DoSomethingCallback cb);
32    /**
33    * @brief Remove callback from list
34    * @param cb callback
35    */
36    void removeDoSomething(DoSomethingCallback cb);
37
38private:
39    // Where our python script will live
40    void _server_thread_work();
41    // To stop our thread
42    volatile std::atomic_bool _stop_thread;
43
44private:
45    // All registered callbacks
46    std::vector<DoSomethingCallback> _dosomethingcallbacks;
47    // Our python script
48    std::thread _server_thread;
49};
50
51#endif // SERVERINTERFACE_H

Code file :

 1#include "serverinterface.h"
 2#include <chrono>
 3
 4ServerInterface::ServerInterface() : _stop_thread(0)
 5{
 6    // At instanciation, launches the server on a separate thread
 7    _server_thread = std::thread(&ServerInterface::_server_thread_work, this);
 8    // No join (we would block this thread while the server is not closed) nor detach (should not be used, memleak and no use)
 9}
10
11ServerInterface::~ServerInterface()
12{
13    // Removes server thread
14    _stop_thread = 1;
15    _server_thread.join(); // will wait for the thread to terminate
16}
17
18ServerInterface &ServerInterface::getInstance()
19{
20    static ServerInterface inst;
21    return inst;
22}
23
24void ServerInterface::registerDoSomething(DoSomethingCallback cb)
25{
26    _dosomethingcallbacks.push_back(cb);
27}
28
29void ServerInterface::removeDoSomething(DoSomethingCallback cb)
30{
31    uint8_t i = 0;
32    for(DoSomethingCallback& cbi : _dosomethingcallbacks)
33    {
34        if(cbi == cb)
35        {
36            _dosomethingcallbacks.erase(_dosomethingcallbacks.begin()+i);
37            break;
38        }
39        i++;
40    }
41}
42
43void ServerInterface::_server_thread_work()
44{
45    // Create interpreter
46    py::scoped_interpreter guard{};
47    // Import our script server.py
48    py::module_ serv_int = py::module_::import("server");
49    // Create server
50    py::object result = serv_int.attr("createServerOnIP")("192.168.0.10");
51    assert(result.cast<bool>());
52    // Set callback handler <------ this is ESSENTIAL to communicate with OUR app and not ... well ... nothing
53    result = serv_int.attr("setCallbackHandler")(this);
54    assert(result.cast<bool>());
55
56    // Server work
57    while(!_stop_thread)
58    {
59        // Check new data
60        serv_int.attr("handle_request")();
61        // Wait a bit (server is not actively used, will give some more time to other threads)
62        std::this_thread::sleep_for(std::chrono::milliseconds(50));
63    }
64}
The most important line is serv_int.attr("setCallbackHandler")(this);, where we are setting a pointer to THIS interface for callbacks.
In your main code, call ServerInterface::getInstance().registerDoSomething() and pass it any function compatible with the callback signature (in serverinterface_py.h) to be processed on callback.

ServerInterface_py#

Header file :

 1#ifndef SERVERINTERFACE_PY_H
 2#define SERVERINTERFACE_PY_H
 3
 4#pragma push_macro("slots")
 5#undef slots
 6#include "pybind11/include/pybind11/pybind11.h"
 7#include "pybind11/include/pybind11/embed.h"
 8#pragma pop_macro("slots")
 9
10namespace py = pybind11;
11
12// Callbacks defs for server interaction (C++ side)
13typedef void (*DoSomethingCallback)();
14// To be able to implement our pointer
15class ServerInterface;
16
17class ServerInterface_py
18{
19    public:
20        ServerInterface_py();
21
22        /**
23        * @brief Launches all callbacks
24        * @return 1 if had an interface set
25        */
26        bool onDoSomething();
27
28    public:
29        // Pointer to our APP interface, could be set as private and provide setters and getters
30        ServerInterface* servint_ptr;
31};
32
33#endif // SERVERINTERFACE_PY_H

Code file :

 1#include "serverinterface_py.h"
 2#include "serverinterface.h"
 3
 4ServerInterface_py::ServerInterface_py() : servint_ptr(nullptr) {}
 5
 6bool ServerInterface_py::onDoSomething()
 7{
 8    // If no interface reference (our APP), go back
 9    if(servint_ptr == nullptr) { return 0; }
10
11    for(ScanBeginCallback& cb : servint_ptr->_scanbegincallbacks)
12    {
13        cb();
14    }
15    return 1;
16}

Nothing fancy, simple callback system. The solely thing to note is the pointer to the interface that allows to link the callbacks to our application.

PythonBindings#

 1#include "../serverinterface_py.h"
 2#include "../serverinterface.h"
 3
 4PYBIND11_MODULE(xmlrpc_interface, m)
 5{
 6    // Doc for help(xmlrpc_interface)
 7    m.doc() = R"pbdoc(
 8        XML-RPC Server Interface
 9        -------------------------
10        Interface Python XML-RPC server with C++ code
11    )pbdoc";
12
13    // Expose class to Python
14        // We NEED to expose this class - since we use a pointer to it, a crash would occur calling serv_int.attr("setCallbackHandler")(this); as Python would not know anything about it
15        // But we do not need to expose anything, since we are just setting a pointer to another pointer
16    py::class_<ServerInterface>(m,"ServInterface");
17        // True interface for Python
18    py::class_<ServerInterface_py>(m,"ServerInterface")
19        .def(py::init<>())
20        .def("onDoSomething",&ServerInterface_py::onDoSomething,"Inform C++ that we received the go to doSomething from server")
21        .def_readwrite("servint_ptr", &ServerInterface_py::servint_ptr, py::return_value_policy::reference); // we will change it inside Python, so expose it
22            // note that we pass by reference, so Python and C++ pointers are the same
23}

Comments are self-explanatory. We cannot pass an argument of undeclared type to Python. Since we intend to modify the servint_ptr (and so to pass it through a function), we muste declare it. Since we don’t use it else than replacing it with another one, we expose none of its elements.

Server.py#

Finally, we have our Python (3) server :

 1# Launch an XML-RPC server to handle URCap commands
 2import socket
 3from xmlrpc.server import SimpleXMLRPCServer
 4from socketserver import ThreadingMixIn
 5# Our own module
 6from xmlrpc_interface import ServerInterface
 7
 8#Our multi-threaded XML-RPC server
 9class MultithreadedSimpleXMLRPCServer(ThreadingMixIn, SimpleXMLRPCServer):
10    pass
11
12# XML-RPC server
13server = None
14# Interface to communicate with C++, using our module
15serv_int = ServerInterface()
16
17def createServerOnIP(ip):
18    """Create server on given IP
19
20    Returns:
21        If server was created successfully
22    """
23    global server
24    server = MultithreadedSimpleXMLRPCServer((ip, 30153), logRequests=False)
25    return initServer(server)
26
27
28def handle_request():
29    """Handles request for XML-RPC server (should be called in a specific C++ thread)
30
31    Returns:
32        None
33    """
34    assert server is not None
35    server.handle_request()
36
37
38# Helpers
39
40
41def initServer(server):
42    """Init server while exposing needed functions
43
44    Note:
45        to be called from one of the createServerxxx functions
46    Returns:
47        If could create
48    """
49    server.RequestHandlerClass.protocol_version = "HTTP/1.1"
50    server.register_function(heartbeat, "heartbeat")
51    server.register_function(set_title, "set_title")
52    server.register_function(get_title, "get_title")
53    server.register_function(start_scan, "start_scan")
54    server.register_function(stop_scan, "stop_scan")
55    server.register_function(scan_message, "scan_message")
56    return server is not None
57
58
59def getIPs():
60    """Print local IPs
61
62    Returns:
63        IP list
64    """
65    for ip in socket.gethostbyname_ex(socket.gethostname())[-1]:
66        print(ip)
67    return socket.gethostbyname_ex(socket.gethostname())[-1]
68
69
70# Server functions
71
72
73def setCallbackHandler(handler):
74    """Set C++ callback handler
75
76    Note:
77        Required for correct callbacks
78    Returns:
79        True
80    """
81    serv_int.servint_ptr = handler;
82    return True
83
84
85def onDoSomething():
86    """C++ / Python communication through callback interface when doSomething is received from any XML-RPC client
87
88    Returns:
89        If callback succeeded
90    """
91    return serv_int.test()

The CMake line is the following :

pybind11_add_module(xmlrpc_interface serverinterface_py.h serverinterface.h serverinterface_py.cpp serverinterface.cpp pythonbindings.cpp)

Threading cons#

The previous chapter introduced a way to get callbacks to our running application.
A problem persists : since our server (i.e. Python script + C++ server function) is running in a separate thread, the sole working callbacks will be static functions, or members functions of objects EXISTING inside our server thread.
If you would like to update your UI on a callback, such callback is not compatible. It is necessary to code some system to communicate around our threads.

Message queue#

If you are not using a framework offering some asynchronous call system (a main loop running for a thread, executing events), you may want to go with standard libraries.
The following is a pseudo-recipe that you may use to implement such system :
  1. Modify the callback registering system to support the form std::function<> (e.g. transform void (*MyCallback)(int,int) to std::function<void(int,int)>
    • Remember to modify the containers std::vector<> to support it

    • The main goal here is to be able to have callbacks on specific objects

  2. When you register your member class function, you can now use std::bind(&Class::yourFunction, &ClassObject, _1, _2 ... /*if fct has args*/) to pass to the registering system while in your object
    • The bind will create a wrapper compatible with the std::function<> embedding the object, such that we now know which function from which object is called

    • If you stop here, and tries to call those on the server callback, you will certainly encounter a crash, since we are trying to call objects that do not belong to the server thread.

  3. You now have some kind of queue to be able to use the callbacks from main thread
    • A vector is not the best : you may need to protect the data from simultaneous R/W, manage to remove already processed callbacks …

    • std::queue is a bir better in such that you can push() on server side and pop() on main thread side (so you will not read the same event twice), but it is not thread-safe either. You may want to extend it to add some mutex on push() and pop() methods.

  4. Whenever your thread-safe queue is created, your last step is to periodically check it from the main thread (thanks to a timer, once in a while in the main event loop …).

Qt callback mechanism#

Using Qt emit, we have an already existing callback system. The problem being that PyBind11 seems hard to integrate with classes implementing Qt specific elements.
The previous example has been modified such that the ServerInterface is never referenced by the binding nor the serverinterface_py.
  1. After our server as handled a request, we are reading a vector of custom PY_MSGS containing which callback we had (and potential extra args)

  2. As we do not set the specific ServerInterface through a pointer anymore (since our own C++ code will handle to dispatch the messages), we can add Qt elements to our ServerInterface

  3. We can now emit specific signals (and so any object can directly QObject::connect() to it).

The modified example is implemented in another project available on our GitLab.

C++ Python Binding