Packaging tools into self contained executable with PyInstaller

Sometimes your project has useful internal tools, but building them can bit cumbersome for other teams which can lead to time being wasted trying to instruct how to get build environment setup and the tool compiled. It would be nice to distribute these internal tools as self contained executables for other teams. PyInstaller is a Python package that can create Linux and Windows binaries that extract the package contents and run a script/entrypoint. Here is a short tutorial how to package internal tools with PyInstaller.

If the tool you are packaging is a simple script that does not have external dependencies (for example non Python executables), packaging can be as easy as:

pyinstaller --onefile tool.py --name tool

But many times the tool uses some binaries that have been compiled and need to be added to the package. We can add extra files into to the package with --add-data:

# pyinstaller --onefile tool.py --name tool --add-data '<source>:<destination in package>'
pyinstaller --onefile tool.py --name tool --add-data 'file.txt:.'

Fixing paths

Above solution works quite well, but your tool will not automatically have access to the binary/file you included, as it most likely can not find the file.

To fix this, we need to add a wrapper script that modifies PATH environment variable and adds the extracted package path (or even sub directories) to the PATH.

When pyinstaller extracts the package content, it uses the tmp system directory and creates a random sub directory. Unfortunately __file__ global variable will not point to the correct location, but we need to check the correct location of the wrapper script with a small function:

import os
import sys

def get_script_location():
    if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
        # Running from PyInstaller
        # Return location of the extracted package
        location = os.path.realpath(sys._MEIPASS)
    else:
        # Normal python run
        # When running PyInstaller package this would incorrectly return
        # current working directory
        location = os.path.dirname(os.path.realpath(__file__))
    return location

We can now add the script path to the PATH:

def add_to_path(p):
    os.environ['PATH'] = p + os.pathsep + os.environ['PATH']

add_to_path(get_script_location())

and then run the original script:

import tool
tool.main()

Check out https://github.com/buq2/pyinstaller_example for full example and how to cross compile the packages.

Easy Conan + CMake template for C++ projects

Conan is bit confusing. It seems to give you a lot of freedom to make mistakes on creating and configuring your project. I want show how Conan and CMake have worked really well for me on my projects.

First we want to define some required packages for the project and generators which help us setup the build system. We can do this by creating conanfile.txt in the project root.

[requires]
    protobuf/3.17.1
    grpc/1.39.1

[generators]
    cmake_find_package

Now we need to do some CMake configuration. We want to use Conan, but also give freedom for the user of the project to choose if they want to handle packages themselves. So we start by creating a CMake file which checks if user has Conan installed.

find_program(conanexecutable "conan")
if(NOT conanexecutable)
    message(WARNING "Tool conan is not installed. Check README.md for build instructions without conan.")
else()
    message(STATUS "Found conan. Installing dependencies.")
    if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake")
        message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan")
        file(DOWNLOAD "https://raw.githubusercontent.com/conan-io/cmake-conan/master/conan.cmake"
                      "${CMAKE_BINARY_DIR}/conan.cmake")
    endif()
    include(${CMAKE_BINARY_DIR}/conan.cmake)
    conan_cmake_run(CONANFILE conanfile.txt
                    BASIC_SETUP
                    BUILD missing
                    CONFIGURATION_TYPES "Release")
    set(CMAKE_MODULE_PATH "${CMAKE_BINARY_DIR}" ${CMAKE_MODULE_PATH})
endif()

If Conan is found, the script downloads newest version of conan.cmake file from github which helps running Conan directly from the CMake. After including the file, we then execute conan_cmake_run for which we can set some configuration options such as file from which the Conan requirements, options and generators are read, if we want to build missing packages, and all build we want to use with multi build systems (such as Visual Studio). Finally we include build directory to CMAKE_MODULE_PATH such that CMakes FIND_PACKAGE can find FindXXX.cmake scripts which are generated by cmake_find_package Conan generator.

Finally in the main CMakeLists.txt file we include the above script and try to find the required packages for the project and later use the found targets.

include(cmake/conan_config.cmake)

find_package(Protobuf REQUIRED)
find_package(gRPC REQUIRED)

#...

target_link_libraries(main
    gRPC::grpc++_unsecure
    )

And that is it. If everything went smoothly, the project can be compiled with

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel 12

You can see the whole sample code in grpc-conan repo, here.