carlosrdrz.dev

Shared libs, rpath and the runtime linker

2024-04-21T19:40:00+02:00

If you are like me, maybe you understand that binaries can have statically or dynamically linked dependencies (shared libs) with each strategy having its own set of pros and cons. Maybe you also know that when a binary has shared libs, you can use ldd to learn more about which shared libraries that binary is using, and where they specifically are located in the file system.

This is the story of how troubleshooting why a binary refused to run, I ended up learning a ton more about shared libraries.

The problem

I was working on compiling the code for a project. The idea was to check if we could use our own compiling toolchain instead of CMake, which was the one commonly used for that project until then. This would bring a bunch of benefits because of integrations with lots of other tools.

We made good progress and managed to have something that successfully compiled, but the problem was that when we tried to run the binary, we were getting an error like this:

➜  ./run_server
./run_server: error while loading shared libraries: liblz4.so.1: cannot open shared object file: No such file or directory

Naturally, this being a problem with a shared library that couldn't be found, I expected ldd would indeed show that there was a library missing:

➜  ldd ./run_server
        linux-vdso.so.1 (0x00007ffd9dfa5000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fa0eb31a000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fa0eb300000)
...

Note: ... means there is (irrelevant) output omitted

Instead, it turns out that some of them were not found, but liblz4.so.1 specifically (the one the error was complaining about) was found! I found that very surprising. How come ldd finds it, but when we run the binary, it does not? Remember this, because we will come back to it later!

My initial thought was: okay, maybe the shared lib location is a symlink pointing to nowhere, or maybe it had the wrong permissions. The error very clearly says "No such file or directory" but I guess it doesn't hurt to check:

➜  ls -la /lib64/libz.so.1
lrwxrwxrwx 1 root root 14 Oct 30 02:16 /lib64/libz.so.1 -> libz.so.1.2.11
➜  ls -la /lib64/libz.so.1.2.11
-rwxr-xr-x 1 root root 102672 Oct 30 02:16 /lib64/libz.so.1.2.11

It's all good. Others should have permission to access the dir and read the library, which makes sense. Probably something much bigger would be broken if that wasn't the case.

Since it seems /lib64 does indeed contain that lib, I thought I could use LD_LIBRARY_PATH to force the binary to check in /lib64. I was pretty confident this wouldn't make any difference because that's a pretty standard directory for shared libs, and because ldd said the lib could be found there (which meant that it was already checking that path).

➜  LD_LIBRARY_PATH=/lib64/ ./run_server
./run_server: error while loading shared libraries: libthird-party_zlib_z.so: cannot open shared object file: No such file or directory

Surprisingly, too, it started complaining about a different lib (libthird-party_zlib_z.so), so indeed it seems /lib64 was not being used after all. (Remember this, too!)

Still, as we can see, there are more libraries not found anyway, so that didn't solve much.

I was running out of ideas, so I turned to the internet. Initially, some Google searches, until I eventually went to my favorite LLM for this sort of question. "Maybe it'll help?" - I thought.

It required some prompt engineering since the initial suggestions were way too basic, but after a while, it recommended I check if the binary had something called rpath set. It suggested a command to check that.

➜  readelf -d ./run_server | head -n 10

Dynamic section at offset 0xbf93cf8 contains 67 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/./__libs__]
 0x0000000000000001 (NEEDED)             Shared library: [liblz4.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
...

Oh, so it is set! What is this? A linker flag?

Let's check the man pages:

➜  man ld
...
       -rpath=dir
           Add a directory to the runtime library search path.  This is used when linking an ELF executable with shared objects.  All -rpath arguments are concatenated and passed to the runtime linker, which uses them to locate shared objects at
           runtime.
...
           The tokens $ORIGIN and $LIB can appear in these search directories.  They will be replaced by the full path to the directory containing the program or shared object in the case of $ORIGIN and either lib - for 32-bit binaries - or lib64
           - for 64-bit binaries - in the case of $LIB.
...

It turns out that when a binary has a rpath set, it adds that path to the runtime search paths (LD_LIBRARY_PATH, ldconfig, etc). The question is: why did my binary have a rpath? And why couldn't the linker find the lib in that directory?

I could only guess, but after spending some time learning about rpath and our toolchain, I concluded that all of this had to do with ensuring isolation between what I was building and whatever shared libs were installed in my system.

Instead of simply running my binary against the shared libs installed in the system, the compile toolchain knows exactly which versions of those shared libs to use (since they are marked as dependencies), so it drops them in a folder, and sets rpath to that folder to ensure those are used (instead of the system's).

The value of rpath in my binary was: $ORIGIN/./__libs__. The key thing to understand here is $ORIGIN, which means the path is relative to the binary path.

It turns out I broke all of this because I copied the binary from the location where the toolchain created the binary to somewhere in /tmp, completely unaware of the fact that the binary was (silently, I must say) referencing libraries with a relative path.

Once that was understood, the possible solutions were straightforward: Symlink the file instead of copying it, or simply use it from that original path.

The mystery is not fully solved

Having fixed my issue, I was happy I could continue with my original goal.

But there was a question in the back of my mind that was bothering me: Why was the binary complaining about not finding the shared lib, if ldd found it in /lib64?

Even accounting for the issue that I caused by copying the binary somewhere else, surely if ldd was finding the lib in /lib64, the binary should find it as well, and run with that version.

➜  ./run_server
./run_server: error while loading shared libraries: liblz4.so.1: cannot open shared object file: No such file or directory

➜  ldd ./run_server
        linux-vdso.so.1 (0x00007ffd9dfa5000)
        liblz4.so.1 => /lib64/liblz4.so.1 (0x00007fa0eb31a000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fa0eb300000)
[ ... ]

As mentioned, when running the binary, it complains liblz4.so.1 does not exist, but surprisingly ldd can find it in /lib64/liblz4.so.1. Isn't that weird?

To understand what is going on, I had to learn more about how these binaries load these shared libraries, so let's go and do that.

Usually, binaries deployed in Linux are ELF binaries. ELF binaries can be statically linked, or dynamically linked, which means libraries need to be linked at runtime. This process of linking shared libraries is done by a piece of software called the "linker runtime" (which usually is ld.so when using glibc).

The question here is: when executing an ELF binary, how do we know where to find the runtime linker?

It turns out every ELF binary includes a "program header" called .interp where that is specified. Here's an example using bash:

➜ readelf -l /usr/bin/bash

Elf file type is DYN (Shared object file)
Entry point 0x31d30
There are 13 program headers, starting at offset 64

Program Headers:
[...]
  INTERP         0x0000000000000318 0x0000000000000318 0x0000000000000318
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
[...]

Most binaries specify the default runtime linker, which is at /lib64/ld-linux-x86-64.so.2 in Linux.

Interestingly, you can call this runtime directly to get more information:

➜  /lib64/ld-linux-x86-64.so.2
/lib64/ld-linux-x86-64.so.2: missing program name
Try '/lib64/ld-linux-x86-64.so.2 --help' for more information.


➜  /lib64/ld-linux-x86-64.so.2 --help
Usage: /lib64/ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked 'ld.so', the program interpreter for dynamically-linked
ELF programs.  Usually, the program interpreter is invoked automatically
when a dynamically-linked executable is started.

You may invoke the program interpreter program directly from the command
line to load and run an ELF executable file; this is like executing that
file itself, but always uses the program interpreter you invoked,
instead of the program interpreter specified in the executable file you
run.  Invoking the program interpreter directly provides access to
additional diagnostics, and changing the dynamic linker behavior without
setting environment variables (which would be inherited by subprocesses).

  --list                list all dependencies and how they are resolved
  --verify              verify that given object really is a dynamically linked
                        object we can handle
  --inhibit-cache       Do not use /etc/ld.so.cache
  --library-path PATH   use given PATH instead of content of the environment
                        variable LD_LIBRARY_PATH
[ ... ]

This runtime linker is used to find out the dependencies (shared libraries) that are required for a binary to run. Then, depending on how it has been configured (using something like ldconfig) those libraries are resolved (found in the filesystem) and linked at runtime.

We can use this runtime linker to run any binaries. For example, this will resolve bash dependencies, resolve them, link them, and run bash, which ends up dropping me in a bash shell:

➜  /lib64/ld-linux-x86-64.so.2 /usr/bin/bash
[carlosrdrz@myserver ~]$

The thing is, to avoid having you prepend every command with its runtime linker, whatever runs the ELF binary does it for you. It reads that .interp section, finds the right runtime linker and calls it to run the binary, so the previous call is equivalent to simply doing:

➜  /usr/bin/bash
[carlosrdrz@myserver ~]$

Alright so now we know how things run, but how can we print the list of dependencies required for a binary to run? You might have noticed there are some arguments we can use in the runtime linker to get info about it, for example:

➜  /lib64/ld-linux-x86-64.so.2 --list /usr/bin/bash
        linux-vdso.so.1 (0x00007ffcff56a000)
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f7cac760000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f7cac400000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7cac8f4000)

That's useful! And at the same time looks very similar to what ldd provides. So, what is ldd doing?

I was surprised when I learned this:

➜  whereis ldd
ldd: /usr/bin/ldd /usr/share/man/man1/ldd.1.gz
➜  readelf -l /usr/bin/ldd
readelf: /usr/bin/ldd: Error: Not an ELF file - it has the wrong magic bytes at the start
➜  file /usr/bin/ldd
/usr/bin/ldd: Bourne-Again shell script, ASCII text executable

ldd is a script! I've always assumed it was a binary for some reason. We can read its code and see what it does, or even better, we could just run it with bash -x and see what it does.

➜  bash -x /usr/bin/ldd /usr/bin/bash
+ TEXTDOMAIN=libc
+ TEXTDOMAINDIR=/usr/share/locale
+ RTLDLIST='/lib/ld-linux.so.2 /lib64/ld-linux-x86-64.so.2 /libx32/ld-linux-x32.so.2'
+ warn=
[ ... ] reducing some output here for brevity [ ... ]
+++ LD_TRACE_LOADED_OBJECTS=1
+++ LD_WARN=
+++ LD_BIND_NOW=
+++ LD_LIBRARY_VERSION=
+++ LD_VERBOSE=
+++ /lib64/ld-linux-x86-64.so.2 /usr/bin/bash
        linux-vdso.so.1 (0x00007fff40151000)
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f3acff3c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3acfc00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3ad00d0000)
+ return 0
+ exit 0

I think there are two things to notice from that (or by analyzing the script directly):

There is a hardcoded list of possible runtime linkers
ldd is simply calling /lib64/ld-linux-x86-64.so.2 with some env variables, which makes it print the shared libs dependencies.

So we've just learned that it turns out you can do this to get the list of shared libs for a binary:

➜  LD_TRACE_LOADED_OBJECTS=1 /lib64/ld-linux-x86-64.so.2 /usr/bin/bash
        linux-vdso.so.1 (0x00007ffecb94b000)
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f181395f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1813600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1813af3000)

Or also this (because as I said, the right runtime linker will get called anyway)

➜  LD_TRACE_LOADED_OBJECTS=1 /usr/bin/bash
        linux-vdso.so.1 (0x00007ffecb94b000)
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f181395f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1813600000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1813af3000)

So... you might be guessing where am I going with all of this.

What happens if you specify your own .interp header in your binaries?

➜ readelf -l ./run_server

Elf file type is EXEC (Executable file)
Entry point 0x37bb540
There are 12 program headers, starting at offset 64

Program Headers:
[...]
  INTERP         0x00000000000002e0 0x00000000002002e0 0x00000000002002e0
                 0x0000000000000028 0x0000000000000028  R      0x1
      [Requesting program interpreter: /usr/local/platform/lib/ld.so]
[...]

When you specify your own runtime linker, it will be used for resolving all those shared libraries.

What you might not expect is that, if ldd does not have those runtime linkers in its hardcoded list of runtime linkers, it will simply try to use the default one, so its output will be completely useless to you. ldd will ask /lib64/ld-linux-x86-64.so.2 to print the shared libs, but ultimately, when running the binary, it is a different runtime linker linking those shared libraries!

Remember that weird thing where setting LD_LIBRARY_PATH to /lib64 when running our binary made it find one of the libs? Maybe now you can guess what was going on!

/lib64/ld-linux-x86-64.so is hardcoded to look for libs in some default paths, like /lib64, whereas /usr/local/platform/lib/ld.so (being a company-specific thing) was not (because of reasons I won't elaborate now).

When using ldd, the default runtime linker was being used, so it found the lib in /lib64, but when running the binary, the company runtime linker was used, and therefore the lib couldn't be found. When I set LD_LIBRARY_PATH to /lib64, it added that "default" dir to the runtime search dir, and made it available.

Learnings

Aside from the satisfaction of resolving a good old troubleshooting mystery, there are some good learnings here that could be useful in the future.

First, that whole thing about rpath is something we might find in the future. It's good to know that sometimes binaries reference shared libs using relative paths, and ldd is not very clear about that happening. Copying a binary to a different directory might make it not work!

Also, most importantly, the easiest way to get the source of truth from the runtime linker is to run your binary using LD env vars, since that ensures that the right runtime linker is giving you that information.

For example:

➜  LD_TRACE_LOADED_OBJECTS=1 $YOUR_BINARY

or:

➜  LD_DEBUG=libs $YOUR_BINARY

Those will print a lot of information about shared library usage in your binary, and you will be sure those are being printed by the real runtime linker.

It's worth clarifying that this assumes a glibc linker, whereas other projects (for example musl) use a different runtime linker that might not support these env vars and might have different mechanisms to figure this out.

You can also find more about this env vars with man ld.so.

Having said all of that, the main issue continued to be the same: I moved my binary to a different folder, it had an rpath, and most of the libraries were in that rpath directory, so they could not be found.

Truth be told, though, using ldd didn't help much with troubleshooting. In fact, it probably made things more difficult, since it wasn't aware of the binary using a different runtime linker, which made it give completely incorrect results.

A couple of warnings would have been nice! Maybe "rpath folder does not exist" or "runtime linker not found"? That would have been useful!

Adding devex to bure

2024-03-10T22:10:00+01:00

About twelve years ago, as a side project, I started working on a client-server game. It was inspired in games like Tibia and Habbo Hotel. The idea was you could log in the game with your own account (which you could create in an accompanying website), then create a character and chat with other people. It never wen't far, but I learnt a lot.

Five years later I decided to work a bit more on it, but I thought at the time that I had to reduce scope if I wanted to get something achievable, so I removed all of the networking stack and kept it as a single-player game, instead. In that iteration I refactored much of the code and introduced creatures, pathfinding and procedural maps.

I (naturally) never finished it anyway, but I was thinking now that it could be cool to write in retrospective about what I did, what I think were good or bad decisions and maybe try to add new features on top.

For a starting post in this series, let's focus on developer experience, so that future improvements are (hopefully) easier to make. We'll go over how to compile and run the project, fix a few issues along the way, and introduce a library to get nicer errors when something inevitably crashes.

The source code in this github repo. The whole engine and "game" (if it can even called one at this point) is written in C++ using SDL.

Compiling the project

The first step to try to get this running would be to compile the project, but unfortunately there are no documented dependencies anywhere. We can get those from the cmake config (CMakeLists.txt), though:

find_package(SDL2 REQUIRED)
find_package(SDL2_image REQUIRED)
find_package(SDL2_ttf REQUIRED)
find_package(LibXMLPP REQUIRED)
find_package(Glib REQUIRED)

There are also a bunch of cmake files in the cmake folder, which IIRC are used to find those libs in the system.

[crodriguez@fedora bure]$ ls -la cmake
total 80
drwxr-xr-x. 1 crodriguez crodriguez   442 mar  3 00:06 .
drwxr-xr-x. 1 crodriguez crodriguez   754 mar  3 00:38 ..
-rw-r--r--. 1 crodriguez crodriguez  5932 mar  3 00:06 CMakeParseArguments.cmake
-rw-r--r--. 1 crodriguez crodriguez  5512 mar  3 00:06 FindGlib.cmake
-rw-r--r--. 1 crodriguez crodriguez   934 mar  3 00:06 FindGlibmm.cmake
-rw-r--r--. 1 crodriguez crodriguez  1429 mar  3 00:06 FindLibXMLPP.cmake
-rw-r--r--. 1 crodriguez crodriguez 14836 mar  3 00:06 FindPackageHandleStandardArgs.cmake
-rw-r--r--. 1 crodriguez crodriguez  2029 mar  3 00:06 FindPackageMessage.cmake
-rw-r--r--. 1 crodriguez crodriguez  7705 mar  3 00:06 FindSDL2.cmake
-rw-r--r--. 1 crodriguez crodriguez  3795 mar  3 00:06 FindSDL2_image.cmake
-rw-r--r--. 1 crodriguez crodriguez  3241 mar  3 00:06 FindSDL2_net.cmake
-rw-r--r--. 1 crodriguez crodriguez  6651 mar  3 00:06 FindSDL2_ttf.cmake
-rw-r--r--. 1 crodriguez crodriguez 10713 mar  3 00:06 LibFindMacros.cmake

So let's install some of them (I'm using fedora 39 right now)

sudo dnf install SDL2-devel SDL2_image-devel SDL2_ttf-devel libxml2-devel glib-devel glibmm2.4-devel libxml++30-devel cmake

There was a really short script to build the whole thing:

[crodriguez@fedora bure]$ cat build.sh
#!/bin/bash

pushd build
cmake ..
make
popd

When doing so, we get a few warnings. Let's try to fix them:

CMake Deprecation Warning at engine/test/lib/googletest-1.8.0/googletest/CMakeLists.txt:48 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Warning (dev) at engine/test/lib/googletest-1.8.0/googletest/cmake/internal_utils.cmake:209 (find_package):
  Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
  are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
  the cmake_policy command to set the policy and suppress this warning.

Call Stack (most recent call first):
  engine/test/lib/googletest-1.8.0/googletest/CMakeLists.txt:60 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

They seem to be coming from googletest. I vendor'd this lib in engine/test/lib. It seems I'm using 1.8 while the lib is now in 1.14, but I'm going to give this version a try. I don't have many tests anyway, so I could look into upgrading later. These warns seem easy to fix.

There is only a couple of changes we need to do to the CMakeLists file in googletest. The first one is the min supported version of cmake listed in the file is 2.6.2, while we can change that to be at least 3.6 I'm currently using 3.27 anyway.

For the second one, you just run that command to get more info about it:

CMP0148
-------

.. versionadded:: 3.27

The ``FindPythonInterp`` and ``FindPythonLibs`` modules are removed.

These modules have been deprecated since CMake 3.12.
CMake 3.27 and above prefer to not provide the modules.
This policy provides compatibility for projects that have not been
ported away from them.

Projects using the ``FindPythonInterp`` and/or ``FindPythonLibs``
modules should be updated to use one of their replacements:

* ``FindPython3``
* ``FindPython2``
* ``FindPython``

The ``OLD`` behavior of this policy is for ``find_package(PythonInterp)``
and ``find_package(PythonLibs)`` to load the deprecated modules.  The ``NEW``
behavior is for uses of the modules to fail as if they do not exist.

This policy was introduced in CMake version 3.27.  CMake version
3.27.7 warns when the policy is not set and uses ``OLD`` behavior.
Use the ``cmake_policy()`` command to set it to ``OLD`` or ``NEW``
explicitly.

.. note::
  The ``OLD`` behavior of a policy is
  ``deprecated by definition``
  and may be removed in a future version of CMake.

I spent like 10 minutes using grep and reading cmake files just to realize the file is indeed mentioned in the error message (do read error messages completely folks).

The fix is simply changing this:

find_package(PythonInterp)

to:

find_package(Python)

Now we can run cmake and build without warnings.

We run the binary, and... it surprisingly works! That wasn't that hard after all.

But... after a bit, the program crashed with this error:

[2024-03-03 00:46:50] [ERROR] max iterations reached in a_star()

Stacktraces

I was getting a fair amount of fatal errors, and I remember trying to add a library called backward to help debug those situations. backward prints stack traces whenever there's a fatal error, but I could not made it work at the time. Let's give that another try now.

First, we'll make a reproducible segmentation fault. That should be easy:

...

  // initialize drawing system
  auto ds = std::make_unique<bure::systems::drawing_system>(std::move(gr));
  this->addSystem(std::move(ds));

  // Reproduce segmentation fault
  int* dir = 0x0;
  *dir = 10;

  // add ui renderer with manager
  _uiManager = std::make_shared<bure::ui::ui_manager>();
  auto uiRenderer = std::make_unique<bure::ui::ui_renderer>(_uiManager);

...

We compile and run:

[crodriguez@fedora bure]$ ./build.sh
~/dev/bure/build ~/dev/bure
-- Found Glibmm  : /usr/lib64/libglibmm-2.4.so
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /home/crodriguez/dev/bure/build
[ 17%] Built target bure-ui
[ 19%] Building CXX object CMakeFiles/bure-engine.dir/engine/engine.cpp.o
[ 21%] Linking CXX static library libbure-engine.a
[ 61%] Built target bure-engine
[ 84%] Built target client-lib
[ 86%] Linking CXX executable client
[ 88%] Built target client
[ 92%] Built target gtest
[ 96%] Built target gtest_main
[ 98%] Linking CXX executable client-test
[100%] Built target client-test
~/dev/bure
[crodriguez@fedora bure]$ ./build/client
Segmentation fault (core dumped)

Alright, so now let's figure out how to add backward so we can get nice stack traces.

Originally I tried to simply drop backward.(cpp/h) files in the source code, add it to my cmake targets and call it a day.

Today I'm going to try a different approach. I'll vendor the lib in a vendor folder (which will be the first step towards vendoring all libs anyway) and then use the CMakeFiles from the original lib to load it.

[crodriguez@fedora bure]$ mkdir vendor
[crodriguez@fedora bure]$ cd vendor
[crodriguez@fedora vendor]$ git clone git@github.com:bombela/backward-cpp.git

Then I add it in CMakeLists in my main client link stage. I add both the Interface target so the lib is available, and the Object target so there are triggers to print the trace whenever there is a fatal error.

add_subdirectory(./vendor/backward-cpp)

...

target_link_libraries(client
  client-lib
  bure-engine
  bure-ui
  ${SDL2_LIBRARIES}
  ${SDL2_IMAGE_LIBRARIES}
  ${SDL2TTF_LIBRARY}
  ${GLIB_LIBRARY}
  ${LibXMLPP_LIBRARIES}
  Backward::Interface   # <-- These two
  Backward::Object)

We compile and run and:

Stack trace (most recent call last):
#5    Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#4    Object "./build/client", at 0x405be4, in
#3    Object "/lib64/libc.so.6", at 0x7f996a76520a, in __libc_start_main
#2    Object "/lib64/libc.so.6", at 0x7f996a765149, in
#1    Object "./build/client", at 0x405e0b, in
#0    Object "./build/client", at 0x4312e8, in
Segmentation fault (Address not mapped to object [(nil)])
Segmentation fault (core dumped)

Progress! In the main README for backward it mentions you can install binutils-devel for the ultimate stack trace. I installed that, then ran it again, and:

Stack trace (most recent call last):
#5    Object "", at 0xffffffffffffffff, in
#4    Object "/home/crodriguez/dev/bure/build/client", at 0x4071c4, in _start
#3    Object "/lib64/libc.so.6", at 0x7fb20ba4620a, in __libc_start_main
#2    Object "/lib64/libc.so.6", at 0x7fb20ba46149, in __libc_start_call_main
#1    Source "/home/crodriguez/dev/bure/main.cpp", line 26, in main [0x4073eb]
         24:   // Init main game object
         25:   auto game = std::make_unique<example_game>();
      >  26:   bure::engine::get().init(std::move(game), resourcesPath, SCREEN_WIDTH,
         27:                            SCREEN_HEIGHT);
         28:
         29:   // Main game loop
#0    Source "/home/crodriguez/dev/bure/engine/engine.cpp", line 28, in bure::engine::init(std::unique_ptr<bure::game, std::default_delete<bure::game> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int) [0x435e72]
         26:   // Reproduce segmentation fault
         27:   int* dir = 0x0;
      >  28:   *dir = 10;
         29:
         30:   // add ui renderer with manager
         31:   _uiManager = std::make_shared<bure::ui::ui_manager>();
Segmentation fault (Address not mapped to object [(nil)])
Segmentation fault (core dumped)

Nice!

Let's try now a different sort of bug. Before, I did get an issue with the A* algorithm failing saying "too many steps". I wonder if that could create a stack trace now.

I changed the code to allow for only 10 iterations in the A* algorithm (before, it was 10K). Now, when we run the binary, we instantly get:

[crodriguez@fedora bure]$ ./build/client
[2024-03-03 10:42:15] [ERROR] max iterations reached in a_star()

No stacktrace, though.

That makes sense when you look at the code:

    if (iter++ > MAX_ITERATIONS) {
      bure::logger::error("max iterations reached in a_star()");
      exit(1);
    }

There are no fatal errors here. We simply log a message in stderr and exit with status code 1. If we want to print a stack trace in these sort of situations, we would need to figure out how backward gets triggered on error.

The default backward trigger is a signal (code here), so that's not going to work for us. We want to trigger stack traces on demand. Instead, we are going to define a new fatal function in the logging lib to print the log message, print a stacktrace and exit.

We are going to be using the backward lib, so we need to add backward to the include libs:

include_directories(${SDL2_INCLUDE_DIRS}
        ${SDL2_IMAGE_INCLUDE_DIRS}
        ${SDL2_TTF_INCLUDE_DIRS}
        ${GLIB_INCLUDE_DIRS}
        ${LibXMLPP_INCLUDE_DIRS}
        ${BACKWARD_INCLUDE_DIRS}
        ${gtest_SOURCE_DIR}/include
        ${gtest_SOURCE_DIR}
        engine)

We'll transform the call to something like this:

    if (iter++ > MAX_ITERATIONS) {
      bure::logger::fatal("max iterations reached in a_star()");
    }

Then define the function:

// logger.cpp

#include "backward.hpp"

using namespace backward;

[...]

void logger::fatal(std::string format...) {
  va_list args;
  va_start(args, format);
  log(level::fatal, format, args);
  va_end(args);

  Printer p;
  StackTrace st;
  st.load_here(32);
  st.skip_n_firsts(3);
  p.print(st);
  exit(1);
}

Now we get these nice stacktraces:

[2024-03-04 22:37:58] [FATAL] max iterations reached in a_star()
Stack trace (most recent call last):
#7    Object "", at 0xffffffffffffffff, in
#6    Object "/home/crodriguez/dev/bure/build/client", at 0x4071e4, in _start
#5    Object "/lib64/libc.so.6", at 0x7faa55c2220a, in __libc_start_main
#4    Object "/lib64/libc.so.6", at 0x7faa55c22149, in __libc_start_call_main
#3    Source "/home/crodriguez/dev/bure/main.cpp", line 32, in main
         29:   // Main game loop
         30:   while (!bure::engine::get().getGame()->isFinished()) {
         31:     bure::event_manager::get().pollEvent();
      >  32:     bure::engine::get().update();
         33:     SDL_Delay(SLEEP_BETWEEN_FRAMES_MS);
         34:   }
#2    Source "/home/crodriguez/dev/bure/engine/engine.cpp", line 74, in bure::engine::update()
         72:   for (auto&& e : getEntities()) {
         73:     if (_entitiesToRemove.find(&e.get()) == _entitiesToRemove.end()) {
      >  74:       e.get().update();
         75:     }
         76:   }
#1    Source "/home/crodriguez/dev/bure/entities/enemy_entity.cpp", line 60, in enemy_entity::update()
         57:           }
         58:
         59:           return result;
      >  60:         });
         61:
         62:     if (!path.empty()) {
         63:       auto next_pos = path.front();
#0    Source "/home/crodriguez/dev/bure/engine/utils/pathfinding.cpp", line 31, in bure::pathfinding::a_star(bure::map_coords, bure::map_coords, std::function<std::unordered_set<bure::map_coords, std::hash<bure::map_coords>, std::equal_to<bure::map_coords>, std::allocator<bure::map_coords> > (bure::map_coords, bure::map_coords)>)
         29:   while (!openSet.empty()) {
         30:     if (iter++ > MAX_ITERATIONS) {
      >  31:       bure::logger::fatal("max iterations reached in a_star()");
         32:     }
         33:
         34:     auto current = lowest_fscore(fScore, openSet);

That should be helpful in the future!

Building this blog

2024-02-28T09:15:00+01:00

This must be my third or fourth attempt at building, maintaining and writing in a blog. One of the things I wanted to try this time is not running an elaborate system for blogging, and instead trying to go with the simplest possible route.

In the past I tried running Wordpress and Ghost in a cloud instance, and even though it makes the writing experience nice for a bit, the whole thing goes stale quite quickly. You have to deal with updates, potential vulnerabilities, the whole plugin ecosystem, OS upgrades... You could go for a SaaS too, but I wanted to give simple static pages a try.

Most of those are built nowadays with Javascript, though, and I really would prefer not dealing with it and its ecosystem, if possible. I don't enjoy that aspect of the web that much. Instead, I tried to look for an alternative built in Python (which I'm dealing a lot with currently at work). That's how I found Pelican, which seems to be okay so far. I've modified a bit the CSS and templates from the default theme, and called it a day. The site is super simple so far, but it seems to also have support for things like syntax highlighting, SASS (if I ever wanted to go down that route, which I doubt) and some other stuff that might prove useful in the future. I guess I'll get there eventually. I might try to code some plugins myself, sometime.

As for the hosting, angelmm recommended Cloudflare Pages, which so far has been easy to use and setup. No concerns there, and it all falls in the free tier.

Anyway, one of the things I want to try differently this time is focusing more on content than on the blog itself. It's been the case multiple times already that I try to start a blog and end up spending much more time in the blogging platform than on actually writing content.

For me, the purpose of having this blog is to share more of my thoughts publicly (which I feel is one thing sometimes I struggle with), share ideas that could be useful to others, or clarify my thoughts on a topic (something I've found writing helps with). With those, it makes sense the focus should be in the content, not in the site.

The version you're seeing at the time of writing this is how it is going to stay for a while. I might add a few pages to add links to stuff I consider useful but don't fit on the article format (some ideas I'm shamelessly getting from this blog), but that's about it.