Can too much choice be a disadvantage? Build systems and dependency management for C++ sometimes might give the impression. Ever cloned a repository and had a really hard time to build the project? Chances are that the build scripts were over engineered or used a build system one was not familiar with. In this post I try to sum up how I currently manage my C++ based projects with an in my opinion reasonable amount of effort.

All samples I’m going to show can be found in this a repository with further comments within the scripts: github: conan-demo.

The Build System: cmake

The first topic we’re going to talk about is the build system. In the past autoconf, manually written Makefiles and others had been used but for the last decade the first choice has been cmake. In spite of newer alternatives like Meson or Bazel, cmake has made it into the de-facto industry standard. Of course this represents my opinion and experience and counter examples exist (boost comes to my mind).

Cmake is an extremely powerful and feature rich tool which leaves a lot of decisions to the user. Therefore there have been significantly different approaches on how to write clean cmake files. The one below is a part of the demo repository (src/libdemo/CMakeLists.txt) mentioned above and shows how one could manage a small library project.

cmake_minimum_required(VERSION 3.14)
project(demo CXX)

# set some common default compiler options
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_VISIBILITY_PRESET hidden)
set(CMAKE_VISIBILITY_INLINES_HIDDEN ON)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)

# explicitly list the source files instead of searching them
# https://stackoverflow.com/questions/1027247/is-it-better-to-specify-source-files-with-glob-or-each-file-individually-in-cmak
# https://cmake.org/cmake/help/v3.15/command/file.html?highlight=glob#filesystem
set(DEMO_SOURCES ${CMAKE_SOURCE_DIR}/src/demo.cpp)
set(DEMO_HEADERS ${CMAKE_SOURCE_DIR}/include/demo/demo.hpp)

# add the new library, specifying SHARED or STATIC is possible
# if not specified it will depend on whether BUILD_SHARED_LIBS is set to ON or OFF
add_library(demo ${DEMO_SOURCES} ${DEMO_HEADERS})

target_include_directories(demo PUBLIC
    "${CMAKE_SOURCE_DIR}/include"
    "${CMAKE_BINARY_DIR}/exports")

Without going into too much detail, let’s just have a brief look at the script. We start with some global compiler definitions. Create the library project. Generate the exports header file and specify the install paths. The last line includes the test project if requested. For most projects this should be good enough to start with the development. For sure it will grow with the project itself. One detail which is not covered is how to include third party libraries. This is necessary in the next file (src/libdemo/tests/CMakeLists.txt).

find_package(GTest REQUIRED)

set(DEMO_TEST_SOURCES   "${CMAKE_CURRENT_SOURCE_DIR}/demo_tests.cpp"
                        "${CMAKE_CURRENT_SOURCE_DIR}/main.cpp")

add_executable(demo_tests ${DEMO_TEST_SOURCES})
target_link_libraries(demo_tests PUBLIC demo GTest::GTest)

This file shows how to search for the googletest dependency and creates the test executable which links against it. The find_package functionality is the feature which utilises the FindXXX.cmake scripts which are a part of the project or cmake itself. The FindXXX.cmake script includes the necessary compiler parameters, linking and include paths. Fortunately for us is that we can install googletest from most linux repositories and do not have to bother with building it. Meaning we will link and compile against the system library. On Windows we would have to build googletest ourselves and then point the CMAKE_MODULE_PATH to the install directory.

The important thing here is that we didn’t add any information on how we manage the dependencies to the cmake file. We used built-in functionality to find the third party library. With arguments like CMAKE_MODULE_PATH and writing or reusing other FindGTest.cmake scripts we are able to control the dependency resolution without having to modify the build files themselves. This will also be useful for more complex dependency management, since FindXXX.cmake scripts can be generated.

Dependency Management: conan

Usually a project doesn’t depend on just one third party library or external component and most likely it also doesn’t exclusively target Linux. Meaning we have to manage, maintain and resolve build issues, binary distribution and multiple different target platforms. Doing this is for a large project is as mentioned a daunting task and the main reason for today’s blog post. While cmake is in wide spread use, coming across projects with loads of binaries stored inside the project structure is far from a rarity. There are multiple projects which try to solve this issue. Some of those are Hunter, vcpkg and conan. Personally I have used conan quite a bit and so far the experience has been very positive. While it already is very flexible and powerful, being able to write all conan related code in Python is the icing on the cake. Vcpkg does not appear to offer the same flexibility back when I checked it out (admittedly quite some time ago) and Hunter has quite a few limitations when attempting to build projects which do not use cmake.

That said let’s dive right into conan and how to set it up. As inclined conan is entirely based on python, meaning one can easily install it using pip. After an installation the default configuration files should be generated. I usually simply run a “conan install .” command since it will do exactly that. Ignore the error message it spits out because of the incomplete install command.

After this I recommend a small manual change in the configuration if Linux systems will be targeted. Conan by default does not differentiate between Linux distributions, meaning we have to add an additional settings parameter. It is not strictly necessary but I recommend it since Linux distributions aren’t always ABI compatible. Especially when compiling against more than just the libc and libstdc++ binaries provided by the host system. The settings file can be found in your user directory “~/.conan/settings.yml”. It is a yaml file containing all settings to differentiate packages built for different platforms. If you look at the example shown here: gists.github.com: settings.yml, a simple distro setting can resolve the issue.

Having put this out of the way, it is a good idea to prepare ones set-up with another simplification. In it’s most basic way conan allows users to specify all settings (to specify the target platform) with every single command. Obviously we’d like to avoid this unnecessary effort. Conan offers a way to sum these settings up in so called profiles. A profile contains the settings for a single platform target. In the conan-demo repository a few examples are given: github.com: conan-demo/profiles. The Android build profiles are especially interesting since they show how one can set additional parameters like environment variables and some cmake options. This is particularly useful for cross compilation. It would easily be possible to specify all environment variables for classic cross compilation without cmake.

With sorting out the settings and profiles, the last and possibly most interesting part is the conanfile.py script. This script has to be created for every package or library which consumes packages, ergo it fulfils two purposes. On the one hand it allows to define dependencies and on the other allows to build and package projects. The scripts features are well documented here: documentation. An implementation for libdemo (pkg/libdemo/conanfile.py) is shown below.

python conanfile.py


from conans import ConanFile, CMake, tools
from conans.errors import ConanInvalidConfiguration

class DemoConan(ConanFile):
    name = "demo"
    version = "1.0"
    license = "MIT"
    url = "https://github.com/MichaEiler/conan-demo"
    settings = "os", "compiler", "arch", "build_type"

    # exports_sources = ["example_change.patch"]

    options = { "shared": [True, False] }
    default_options = { "shared": False }

    git_name = "conan-demo"
    git_url = "https://github.com/MichaEiler/conan-demo.git"
    git_tag = "v1.0"
    src_path = "conan-demo/src/libdemo/"

    def configure(self):
        # check the selected settings
        if self.settings.os == "Arduino":
            raise ConanInvalidConfiguration("This library only compiles on platforms with full STL support.")

    def build_requirements(self):
        # this is where you can specify packages like the android-ndk, nasm and other build tools
        # self.requires("nasm/1.x.y")
        pass

    def requirements(self):
        # define dependencies here:
        #self.requires("boost/1.73.0@user/channel")
        #self.options["boost"].shared = self.option.shared
        pass

    def source(self):
        self.run(f"git clone {self.git_url}")
        self.run(f"cd {self.git_name} && git checkout {self.git_tag}")

        # in case of required changes to some source code or build files within the git repository
        # add the patch to exports_sources and apply it here
        # this will also work on windows, since we are using a python/conan patch tool implementation
        # https://docs.conan.io/en/latest/reference/tools.html#tools-patch
        # tools.patch(base_path=self.src_path, patch_file="example_change.patch")

    def build(self):
        cmake = CMake(self)
        cmake.definitions["BUILD_SHARED_LIBS"] = "ON" if self.options.shared else "OFF"
        cmake.definitions["DEMO_ADD_TESTS"] = "OFF"
        # many other options are set automatically by CMake class
        # for scenarios where cross compilation is important, those should have been already set in the conan profile
        cmake.configure(source_folder=self.src_path)
        cmake.build()

        # the default cmake options are set in a way that a good CMakeLists.txt file already puts the resulting files into
        # the correct folders within the conan package (e.g. package/lib/, package/include/, ...)
        cmake.install()
        

    def package(self):
        # most copy operations can be handled by cmake install, in case additional files need to be copied or the CMakeLists.txt
        # does not feature an install option, this can be done here
        if self.settings.os == "Windows" and self.settings.build_type == "Debug":
            self.copy("*.pdb", src="./Debug/", dst="bin/", keep_path=False)

    def package_info(self):
        # remember the cmake generated export header definitions in libdemo?
        # let's make sure that in case of a static library the correct defines are set when consuming this package
        if not self.options.shared:
            self.cpp_info.defines.append("DEMO_STATIC_DEFINE")

        self.cpp_info.libs.append(self.name)

        # define the name in the automatically generated find_package scripts
        self.cpp_info.names["cmake_find_package"] = "Demo"
        self.cpp_info.names["cmake_find_package_multi"] = "Demo"

        # linking system libraries in case necessary can be done here
        #if self.settings.os == "Linux":
        #    self.cpp_info.system_libs.append("pthread")

Since libdemo is a very basic library the script doesn’t do too much and explains quite a few things in the comments instead. Important for packaging libdemo is the generic header information with the package name and version, as well as the settings required by the package. The package doesn’t require any dependencies, therefore we do not have to specify any in the requirements method. The source method calls git to clone the repository. The build method is slightly more interesting. It uses the cmake helper offered by the conan project to build the source code. Thanks to our profiles and toolchain files we do not have to specify a lot of additional cmake arguments. Most of them are already set by conan.

The script has been tested on Linux and Windows as well as with an Android cross build. The install logic will install the header and binary files automatically to the correct path inside the package since we made sure the cmake install operations are defined correctly. One more critical method is package_info where we have to specify important pre-processor defines, the compiled libraries (incl. their linking order(!)) as well as system dependencies.

With the script above we are now already able to build the package for all platforms for which we have a profile:

conan create . projectname/channelname -pr="<path>/profiles/fedora-x86_64-release -o shared=False

Once a package has been created it can be consumed by other projects. The demo repository also contains a small command line utility calling libdemo and interacting with the user. In contrast to the library project, the conanfile.py script shipped with the demo-tool (src/demo-tool/conanfile.py), only consumes other packages and does not build or package the resulting executable.

from conans import ConanFile, CMake, tools
from conans.errors import ConanInvalidConfiguration

class DemoToolConan(ConanFile):
    name = "demo-tool"
    settings = "os", "compiler", "arch", "build_type"
    generators = [ "cmake_find_package", "cmake_find_package_multi" ]

    def build_requirements(self):
        # this is where you can specify packages like the android-ndk, nasm and other build tools
        # self.requires("nasm/1.x.y")
        pass

    def requirements(self):
        # define dependencies here:
        self.requires("demo/1.0@projectname/channelname")
        self.options["demo"].shared = False

    def install(self):
        # in case some binary files from dependencies should be copied over
        # e.g. *.so, *.pdb files
        pass

Note two things in this script. The requirements method where we specify the dependency to the libdemo package and the generators in class members. In this case we want to generate the already mentioned FindXXX.cmake scripts, meaning we require the cmake_find_package generator. For Windows we also have to add the cmake_find_package_multi generator because of the visual studio projects requiring Debug and Release dependencies.

conan install .. -pr="<path/profiles/fedora-x86_64-release"
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_MODULE_PATH=`pwd` ..
make

Execute these two commands within a build subdirectory within the project. The resulting binary will work just fine on most Linux machines. Building for Android would also be possible, although in this case we are talking about an executable instead of a library and therefore cannot run it without rooting the phone (afaik). Building for Windows is only slightly different:

conan install .. -pr="<path>/profiles/win64-vs2019-release"
conan install .. -pr="<path>/profiles/win64-vs2019-debug"
cmake -G"Visual Studio 16 2019" -Ax64 -Thost=x64 -DCMAKE_MODULE_PATH=<builddir> ..
cmake --build . --config Release --target ALL_BUILD

The conan install command needs to be executed twice to download the dependencies for the Release and Debug configurations. For the interested reader: have a look at the build directory after running the conan install commands. You will find the generated FindXXX.cmake scripts for the cmake find_package feature there.

Notes

Due to the length of this post I left out some additional interesting tidbits. A few coming of the top of my mind are listed here:

The blog post has been very focused on cmake. I believe this is the most common situation for most projects. Nevertheless conan has no requirements which enforce the usage of cmake. The methods in the conanfile script can be freely adapted. It is entirely possible to just call make, compile a Visual Studio solution or use xcodebuild.
It also isn’t necessary to write all build scripts for open source dependencies. There are already public package scripts available for a significant number of projects: https://github.com/conan-io/conan-center-index/tree/master/recipes
Since I don’t have a macOS based machine available at the moment, I couldn’t test the script with macOS and iOS. Nevertheless this shouldn’t be a big problem if one utilises the toolchain files created by Alexander Widerberg: https://github.com/leetal/ios-cmake
The toolchain for Android can be even more streamlined. I just access an installed Android NDK in the toolchain file. It is of course also possible to package the ndk itself and list it under build_requires. There are already existing packages which can be reused for this task: https://github.com/bincrafters/conan-android_ndk_installer
The commands and scripts shown in this post also have package specific options. They can be freely adapted. The most basic use-case is a shared-flag to differentiate between static and shared libraries within packages.
I didn’t mention anything about managing the compiled packages. There is the free and open conan-server utility and the Artifactory Community Edition taking care of this. Both allow users to upload their packages to the server instance and later on download them on other machines. Think of them as repositories similar to the package repositories for Linux distributions.

Conclusion

Good things are rarely free. Meaning even with this approach creating all conan packages and clean cmake files still ends up in quite some effort. Nevertheless the results are in my opinion speaking for themselves. Especially larger projects with multi-platform support should consider using proper dependency management.

C++ Project Maintenance: Cmake and Conan by Example

The Build System: cmake

Dependency Management: conan

Notes

Conclusion