Skip to content

Conversation

@vtri950
Copy link

@vtri950 vtri950 commented Nov 27, 2025

Upgrade Buildroot to 2025.02.9 for GCC 13+ compatibility

Fixes #21967
Fixes #20993

Changes:

  • Upgrade BUILDROOT_BRANCH from 2025.02 to 2025.02.9 LTS

The newer Buildroot version includes updated package versions that are
compatible with modern GCC 13+ toolchains, eliminating build failures
on recent Linux distributions like Fedora 39+.

Tested on Fedora 43 with GCC 13.3.1.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 27, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @vtri950. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 27, 2025
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@nirs
Copy link
Contributor

nirs commented Nov 27, 2025

/ok-to-build-iso

Copy link
Contributor

@nirs nirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vtri950 thanks!

Makefile Outdated

INSTALL_SIZE ?= $(shell du out/minikube-windows-amd64.exe | cut -f1)
BUILDROOT_BRANCH ?= 2025.02
BUILDROOT_BRANCH ?= 2025.08.2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason not to use 2025.08?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2025.08.02 is the latest stable patch release

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on https://buildroot.org/download.html this version end of life is December 2025, so we have to update every month?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!
i have replaced it with the LTS version which contains the necessary fix.

@nirs
Copy link
Contributor

nirs commented Nov 28, 2025

failed iso build logs:
https://storage.cloud.google.com/minikube-builds/logs/21997/0cbf9da/iso_build.txt

Build failed with:

+ git remote add vtri950 git@github.com:vtri950/minikube.git
+ git fetch vtri950
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Build step 'Execute shell' marked build as failure

@medyagh any clue why the build tries to add this remote?

Copy link
Contributor

@nirs nirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vtri950 please rebase on master. The gluster package is already removed in master for long time. This may also fix the build iso job.

After rebase we see:

% git show --stat
commit 113d9e35755e9b5a7aea6537aded9ef3a56605bd (HEAD -> upgrade-buildroot-gcc13)
Author: Vidit Tripathi <viditt14@gmail.com>
Date:   Thu Nov 27 08:13:57 2025 +0530

    Upgrade Buildroot to 2025.08.2 for GCC 13+ compatibility

 Makefile                                             |  2 +-
 deploy/iso/minikube-iso/package/Config.in            |  1 -
 deploy/iso/minikube-iso/package/podman/Config.in     | 11 -----------
 deploy/iso/minikube-iso/package/podman/override.conf |  4 ----
 deploy/iso/minikube-iso/package/podman/podman.conf   |  1 -
 deploy/iso/minikube-iso/package/podman/podman.hash   |  7 -------
 deploy/iso/minikube-iso/package/podman/podman.mk     | 81 ---------------------------------------------------------------------------------
 7 files changed, 1 insertion(+), 106 deletions(-)

@vtri950
Copy link
Author

vtri950 commented Nov 30, 2025

failed iso build logs: https://storage.cloud.google.com/minikube-builds/logs/21997/0cbf9da/iso_build.txt

Build failed with:

+ git remote add vtri950 git@github.com:vtri950/minikube.git
+ git fetch vtri950
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Build step 'Execute shell' marked build as failure

@medyagh any clue why the build tries to add this remote?

the issue is here
https://github.com/kubernetes/minikube/blob/master/hack/jenkins/build_iso.sh#L88

@vtri950 vtri950 force-pushed the upgrade-buildroot-gcc13 branch from 0cbf9da to 8a09186 Compare November 30, 2025 04:19
@vtri950
Copy link
Author

vtri950 commented Nov 30, 2025

@vtri950 please rebase on master. The gluster package is already removed in master for long time. This may also fix the build iso job.

After rebase we see:

% git show --stat
commit 113d9e35755e9b5a7aea6537aded9ef3a56605bd (HEAD -> upgrade-buildroot-gcc13)
Author: Vidit Tripathi <viditt14@gmail.com>
Date:   Thu Nov 27 08:13:57 2025 +0530

    Upgrade Buildroot to 2025.08.2 for GCC 13+ compatibility

 Makefile                                             |  2 +-
 deploy/iso/minikube-iso/package/Config.in            |  1 -
 deploy/iso/minikube-iso/package/podman/Config.in     | 11 -----------
 deploy/iso/minikube-iso/package/podman/override.conf |  4 ----
 deploy/iso/minikube-iso/package/podman/podman.conf   |  1 -
 deploy/iso/minikube-iso/package/podman/podman.hash   |  7 -------
 deploy/iso/minikube-iso/package/podman/podman.mk     | 81 ---------------------------------------------------------------------------------
 7 files changed, 1 insertion(+), 106 deletions(-)

done

@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

/ok-to-build-iso

@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

/cc @afbjorklund

@vtri950 vtri950 force-pushed the upgrade-buildroot-gcc13 branch from 8a09186 to 6232705 Compare November 30, 2025 12:59
@vtri950 vtri950 changed the title Upgrade Buildroot to 2025.08.2 for GCC 13+ compatibility Upgrade Buildroot to 2025.02.8 LTS for GCC 13+ compatibility Nov 30, 2025
@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

Upgrade Buildroot to 2025.08.2 for GCC 13+ compatibility

Fixes #21967

Changes:

  • Upgrade BUILDROOT_BRANCH from 2025.02 to 2025.08.2

Latest push change version to 2025.02.8

  • Remove custom gluster package (requires deprecated Python 2)

This was already in master for a while

  • Remove custom podman package (now built-in to Buildroot 2025.08.2)

The newer Buildroot version includes updated package versions that are compatible with modern GCC 13+ toolchains, eliminating build failures on recent Linux distributions like Fedora 39+.

Tested on Fedora 43 with GCC 13.3.1.

Did you test latest push with with Fedora 43?

Please update the PR message to reflect the actual change. If you are still testing it you can convert it to draft.

@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

/ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @vtri950, building a new ISO failed for Commit 8a09186
See the logs at:
https://storage.cloud.google.com/minikube-builds/logs/21997/8a09186/iso_build.txt

@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

Hi @vtri950, building a new ISO failed for Commit 8a09186 See the logs at: https://storage.cloud.google.com/minikube-builds/logs/21997/8a09186/iso_build.txt

@vtri950 you can ignore this message, it is for the commit you replaced. We will get a new comment when the current build will complete.

The build log will be here:
https://storage.cloud.google.com/minikube-builds/logs/21997/6232705/iso_build.txt

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 30, 2025

It seems like this build issue is specific to Fedora*, but probably a good idea to update the OS minor anyway...

* The default build environment uses Debian, currently bookworm (12, oldstable): support/docker/Dockerfile

There was some earlier attempt to bump the Buildroot version, not sure what happened to that one?

source "$BR2_EXTERNAL_MINIKUBE_PATH/package/cni/Config.in"
source "$BR2_EXTERNAL_MINIKUBE_PATH/package/automount/Config.in"
source "$BR2_EXTERNAL_MINIKUBE_PATH/package/scheduled-stop/Config.in"
source "$BR2_EXTERNAL_MINIKUBE_PATH/package/podman/Config.in"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the podman package was added to Buildroot, until after 2025.02

Make sure that it is still enabled in the Buildroot config, and that it builds OK...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit ebbaac08e3f462bf023c12077f739ed1c0543eb2

Buildroot 2025.05

@nirs
Copy link
Contributor

nirs commented Nov 30, 2025

There was some earlier attempt to bump the Buildroot version, not sure what happened to that one?

We did not attempt to resolve this issue yet. @vtri950 you can add this issue to this PR.

@vtri950
Copy link
Author

vtri950 commented Nov 30, 2025

Upgrade Buildroot to 2025.08.2 for GCC 13+ compatibility
Fixes #21967
Changes:

  • Upgrade BUILDROOT_BRANCH from 2025.02 to 2025.08.2

Latest push change version to 2025.02.8

  • Remove custom gluster package (requires deprecated Python 2)

This was already in master for a while

  • Remove custom podman package (now built-in to Buildroot 2025.08.2)

The newer Buildroot version includes updated package versions that are compatible with modern GCC 13+ toolchains, eliminating build failures on recent Linux distributions like Fedora 39+.
Tested on Fedora 43 with GCC 13.3.1.

i have updated the description

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 30, 2025

This is something to watch out for in general, when replacing minikube packages with buildroot packages:

-config BR2_PACKAGE_PODMAN
-       bool "podman"
-       default y
config BR2_PACKAGE_PODMAN
	bool "podman"

That is, the minikube packages were selected by default - while buildroot packages need to be selected:

BR2_PACKAGE_PODMAN=y

Since minikube is already providing buildroot defconfig, having packages auto-select themselves was a mistake.
Having most packages default to using external pre-built binaries was also a mistake, albeit a more deliberate one.

It would have been nice to work closer with Buildroot upstream, and submit more packages for inclusion there.

[Socket]
SocketMode=0660
SocketUser=root
SocketGroup=podman
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to keep this configuration, otherwise it will not be possible to reach podman socket

@@ -1 +0,0 @@
d /run/podman 0770 root podman
Copy link
Collaborator

@afbjorklund afbjorklund Nov 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This as well, otherwise the directory containing the podman socket will be unreadable by the group

@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 30, 2025

The default podman configuration can only be accessed by root, so it will not work for docker@minikube

It would be possible to move this configuration outside the package though, and use systemd overrides.

/etc/systemd/system/podman.socket.d/override.conf

/etc/tmpfiles.d/podman.conf

Copy link
Contributor

@nirs nirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing shows that aarch64 iso fail on Fedora 43 and 42, so we cannot document it as working.

We can split the documentation update to a new PR or change it to document the issues on Fedora.

@vtri950 vtri950 force-pushed the upgrade-buildroot-gcc13 branch from ee66eb7 to 4e6ec0c Compare December 5, 2025 05:45
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 5, 2025
@vtri950
Copy link
Author

vtri950 commented Dec 5, 2025

Testing shows that aarch64 iso fail on Fedora 43 and 42, so we cannot document it as working.

i have removed the document part

We can split the documentation update to a new PR or change it to document the issues on Fedora.

can you open an issue with this finding?
i will create a fix for it and create the necessary documentation a well

@minikube-pr-bot

This comment has been minimized.

@minikube-pr-bot
Copy link

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment Test Name Flake Rate
KVM_Linux_crio (5 failed) TestFunctional/parallel/ImageCommands/ImageBuild(gopogh) 0.00% (chart)
KVM_Linux_crio (5 failed) TestFunctional/parallel/ImageCommands/ImageListShort(gopogh) 2.50% (chart)
KVM_Linux_crio (5 failed) TestCertExpiration(gopogh) 6.38% (chart)

Besides the following environments also have failed tests:

  • Docker_Linux_containerd_arm64: 34 failed (gopogh)

  • Docker_Linux_crio: 48 failed (gopogh)

  • Docker_Linux_crio_arm64: 57 failed (gopogh)

  • Docker_Windows: 34 failed (gopogh)

To see the flake rates of all tests by environment, click here.

Copy link
Contributor

@nirs nirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: afbjorklund, nirs, vtri950
Once this PR has been reviewed and has the lgtm label, please assign comradeprogrammer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@nirs
Copy link
Contributor

nirs commented Dec 7, 2025

We can split the documentation update to a new PR or change it to document the issues on Fedora.

can you open an issue with this finding? i will create a fix for it and create the necessary documentation a well

I created 2 issues:

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2025
@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 12, 2025

There is a new Buildroot 2025.02.9 released

https://buildroot.org/news.html

Consider also bumping the Linux version?

See buildroot/linux/linux.hash

@vtri950 vtri950 force-pushed the upgrade-buildroot-gcc13 branch from 4e6ec0c to 751b365 Compare December 15, 2025 06:12
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 15, 2025
@vtri950 vtri950 changed the title Upgrade Buildroot to 2025.02.8 LTS for GCC 13+ compatibility Upgrade Buildroot to 2025.02.9 LTS for GCC 13+ compatibility Dec 15, 2025
@vtri950
Copy link
Author

vtri950 commented Dec 15, 2025

Consider also bumping the Linux version?

can we do the linux upgrade in a separate issue?

@minikube-pr-bot

This comment has been minimized.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 15, 2025

can we do the linux upgrade in a separate issue?

You can separate it, the feedback was just that upstream includes a Linux upgrade with the rest of the OS release

https://github.com/buildroot/buildroot/blob/2025.02.8/linux/linux.hash

https://github.com/buildroot/buildroot/blob/2025.02.9/linux/linux.hash

It is not automatic, because minikube is not using the "latest version" (6.12.x) but a "custom version" (6.6.x).

https://github.com/buildroot/buildroot/blob/2025.02.x/linux/Config.in

BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="6.6.95"

@nirs
Copy link
Contributor

nirs commented Dec 15, 2025

We have another issue (#22031) for updating the kernel. Lets keep this change simple and merge it quickly.

@nirs
Copy link
Contributor

nirs commented Dec 15, 2025

/ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @vtri950, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21997 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 41.6s    │ 41.5s                  │
│ enable ingress │ 15.9s    │ 15.8s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 38.5s 42.4s 42.2s 41.0s 43.8s
Times for minikube (PR 21997) start: 41.2s 43.1s 40.6s 42.5s 40.1s

Times for minikube ingress: 15.7s 15.8s 15.8s 15.7s 16.3s
Times for minikube (PR 21997) ingress: 15.8s 15.8s 15.8s 15.9s 15.9s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21997 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.6s    │ 23.6s                  │
│ enable ingress │ 10.4s    │ 10.8s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 20.2s 23.7s 21.2s 22.8s 25.0s
Times for minikube (PR 21997) start: 24.4s 24.5s 23.1s 22.0s 24.0s

Times for minikube ingress: 10.6s 10.7s 9.6s 10.7s 10.7s
Times for minikube (PR 21997) ingress: 10.6s 10.7s 10.6s 11.6s 10.6s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21997 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 21.3s    │ 20.9s                  │
│ enable ingress │ 20.5s    │ 20.4s                  │
└────────────────┴──────────┴────────────────────────┘
Details

Times for minikube start: 20.9s 19.2s 22.6s 21.2s 22.4s
Times for minikube (PR 21997) start: 20.8s 18.8s 23.1s 19.7s 21.9s

Times for minikube ingress: 21.1s 21.2s 20.2s 20.2s 20.1s
Times for minikube (PR 21997) ingress: 20.2s 20.2s 20.2s 20.1s 21.2s

@nirs
Copy link
Contributor

nirs commented Dec 15, 2025

@vtri950 after updating buildroot, did you run make iso-menuconfig-x86_64 and make iso-menuconfig-aarch64? After running and exiting without making any change in menuconfig, there are some changes in the buildroot configs:

$ make iso-menuconfig-x86_64

$ git diff

$ git diff
diff --git a/deploy/iso/minikube-iso/configs/minikube_x86_64_defconfig b/deploy/iso/minikube-iso/configs/minikube_x86_64_defconfig
index 47fce0fa7..639a2fdf3 100644
--- a/deploy/iso/minikube-iso/configs/minikube_x86_64_defconfig
+++ b/deploy/iso/minikube-iso/configs/minikube_x86_64_defconfig
@@ -37,7 +37,6 @@ BR2_PACKAGE_XFSPROGS=y
 BR2_PACKAGE_ACPID=y
 BR2_PACKAGE_PARTED=y
 BR2_PACKAGE_SYSSTAT=y
-BR2_PACKAGE_LUAJIT=y
 BR2_PACKAGE_LZ4=y
 BR2_PACKAGE_LZ4_PROGS=y
 BR2_PACKAGE_CA_CERTIFICATES=y
$ make iso-menuconfig-aarch64

$ git diff
diff --git a/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig b/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig
index 103782e79..f3cd6346e 100644
--- a/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig
+++ b/deploy/iso/minikube-iso/configs/minikube_aarch64_defconfig
@@ -40,7 +40,6 @@ BR2_PACKAGE_XFSPROGS=y
 BR2_PACKAGE_ACPID=y
 BR2_PACKAGE_PARTED=y
 BR2_PACKAGE_SYSSTAT=y
-BR2_PACKAGE_LUAJIT=y
 BR2_PACKAGE_LZ4=y
 BR2_PACKAGE_LZ4_PROGS=y
 BR2_PACKAGE_CA_CERTIFICATES=y

Maybe luajit was removed from buildroot?

It is important to run this after the update so we can continue to update the config using menuconfig. Otherwise we will find this change on the next time we update buildroot configuration.

I'm running this on linux aarch64 vm - but usually we work on the iso configuration and build the iso on x86_64, so this may be a buildroot bug.

@afbjorklund do you know why we see this change?

I see that I run this on older buildroot 2025.02, testing again with 2025.02.9. Testing on the right buildroot show the same results, so this is not related to this change. I'll test again on x86_64.

@k8s-ci-robot
Copy link
Contributor

@vtri950: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
integration-vfkit-docker-macos-arm64 035679b link false /test integration-vfkit-docker-macos-arm64
integration-docker-crio-linux-x86-64 035679b link true /test integration-docker-crio-linux-x86-64
integration-kvm-containerd-linux-x86-64 035679b link true /test integration-kvm-containerd-linux-x86-64
integration-kvm-crio-linux-x86-64 035679b link true /test integration-kvm-crio-linux-x86-64

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Building minikube iso in docker fails iso: Update buildroot to bugfix release 2025.02.x

6 participants