The risk, stated plainly

CentOS 7 is near end of life, and its stock gcc 4.8.5 and glibc 2.17 are old enough that some newer software simply won’t compile against them, including some components we run for algorithmic work. So upgrading the toolchain becomes unavoidable.

But let’s be clear up front: upgrading glibc is the single most dangerous step here. Upgrading gcc is relatively safe, because it can go into its own prefix and coexist with the system gcc, and you pick which one via PATH. glibc is different. It’s the base library that nearly every binary on the system links against, so if a replacement goes wrong you get anything from commands erroring out to a box that won’t boot and won’t even take ssh. Posts about someone upgrading glibc and bricking a production machine show up regularly; that’s not scaremongering.

We’ve had an incident from a glibc upgrade ourselves, and only got fluent at it after repeated practice driven by algorithm requirements. This carries real risk, so proceed carefully: rehearse on a machine you can snapshot and roll back, confirm it’s clean, and only then touch production. Before upgrading glibc, make sure you have an out-of-band rescue path (KVM/IPMI/cloud console), not just ssh.

The steps here follow this article, verified item by item on our side: https://chandrameenamohan.medium.com/building-nvidia-triton-inference-server-from-scratch-for-tensorflow-backend-873ef0a1f7f3

Target versions:

gcc    4.8.5  -->  9.3.0
glibc  2.17   -->  2.29

Before you start

Record the current versions so you have something to compare against afterward:

gcc -v
ldd --version

Then install the dependencies. Set up EPEL first:

yum install epel-release.noarch -y
yum install wget gcc gcc-c++ zlib-devel libcurl-devel openssl-devel bzip2-devel gmp-devel.x86_64 mpfr-devel.x86_64 libmpc-devel.x86_64 libSM-devel.x86_64 libxml2-devel gcc-gfortran.x86_64 autoconf automake boost-devel make patchelf bison -y

Adjust the dependency list to your system and installed libraries. patchelf is worth calling out: if some already-built binary can’t find the new loader or libraries after the upgrade, you can rescue it by editing the ELF’s interpreter and rpath. It’s a standard tool for cleaning up after a glibc upgrade.

Upgrade and verify

1. Upgrade gcc to 9.3.0

This step is relatively safe because it doesn’t touch the system gcc.

mkdir gcc-build
cd gcc-build

GCC_VERSION=9.3.0

# Download the chosen gcc version, here 9.3.0
wget https://ftp.gnu.org/gnu/gcc/gcc-${GCC_VERSION}/gcc-${GCC_VERSION}.tar.gz
# Extract
tar xzvf gcc-${GCC_VERSION}.tar.gz
# Create the build directory
mkdir obj.gcc-${GCC_VERSION}
# Enter the source tree and run download_prerequisites to fetch build dependencies
cd gcc-${GCC_VERSION}
./contrib/download_prerequisites

# Build and install from the sibling obj directory
cd ../obj.gcc-${GCC_VERSION}
../gcc-${GCC_VERSION}/configure --disable-multilib --enable-languages=c,c++
make -j 4   # match the job count to your CPU cores
make install

Don’t skip ./contrib/download_prerequisites: it pulls the matching versions of gmp/mpfr/mpc into the source tree so you don’t have to reconcile versions by hand. --disable-multilib drops 32-bit support, which you rarely need on a 64-bit box, and --enable-languages=c,c++ builds only C/C++ (not Fortran and friends), which saves a good deal of time. The version, install directory, and build directory are all yours to change.

2. Upgrade make to 4.3

glibc 2.29 requires make 4.3 or newer, and the make bundled with CentOS 7 is too old, so upgrade make first:

wget https://ftp.gnu.org/gnu/make/make-4.3.tar.gz
tar xf make-4.3.tar.gz
cd make-4.3
./configure --prefix=/usr/local
make
make install

Nothing tricky here, but remember make lands under /usr/local. Then yum remove make to drop the old version, and either adjust your environment or symlink the new make from /usr/local/bin and /usr/local/sbin into /usr/bin/ and /usr/sbin/, so that the glibc build picks up 4.3.

3. Upgrade glibc to 2.29 (high risk)

This is the genuinely dangerous step. The --prefix=/usr below installs the new glibc into the system’s main prefix, i.e. it replaces the glibc the system is currently running on, which is exactly where the risk lives. Confirm one more time before you start: the machine is snapshotted and you have an out-of-band rescue channel.

wget https://ftp.gnu.org/gnu/glibc/glibc-2.29.tar.gz
tar xf glibc-2.29.tar.gz
cd glibc-2.29
mkdir glibc-build
cd glibc-build
../configure --prefix=/usr && /usr/local/bin/make
/usr/local/bin/make install

One thing you must get right: use the make you just upgraded, so write it explicitly as /usr/local/bin/make rather than the old system make, or glibc 2.29 will fail to build on an insufficient make version. Nothing else is special.

Verify

ldconfig
gcc --version
ldd --version

gcc --version should report 9.3.0 and ldd --version should report 2.29. Run ldconfig first to refresh the shared-library cache, then verify.

Takeaway

Keep the tiers of risk straight: gcc and make can both go into their own prefix and coexist with the system, so a mistake is contained; glibc is different, and --prefix=/usr is replacing the base library for the whole system, where one wrong move can leave the box unbootable. So the two lines to hold are these: do the glibc step only on a machine you can snapshot and rescue out-of-band, and build glibc with the upgraded /usr/local/bin/make. Do those two, and this path is one we’ve verified repeatedly.