Our third newsletter is a bit late -- but has an in-depth article on some as-yet-unexplained aspects of Funtoo related to metatools and our CDN. Definitely worth a read!

Each newsletter, we are going to try to feature an in-depth article combined with key Funtoo news for the month. 

Linux Kernel 6.1.20_p1 is Now Available

The sys-kernel/debian-sources-6.1.20_p1 has been unmasked on the meta-repo tree and is available for regular upgrade for everybody. This is basically a “bug fix” upgrade, as no new features or modules were announced. However, 1185 files have been touched by bug fix commits since v.6.1.12_p1, so you might want to consider upgrading your kernel to benefit from them.

The Funtoo CDN and Metatools

Most people are aware that Funtoo has its own CDN (Content Distribution Network), but few understand the role it has been playing in Funtoo and the potential it has for the future. Over the last month there have been interesting developments regarding the use of the CDN associated with Funtoo Metatools. Metatools lets Funtoo auto-generate up-to-date ebuilds from sites like GitHub. This month we saw the realization of a year-long effort which made Go and Rust packages a lot more efficient.

In this article, you will learn about the evolution of our content distribution, from the early days of Gentoo to the latest stage at which Funtoo finds itself right now, as well as some of the inner workings of Metatools and how to use the new extensions in your Go and Rust autogens.

Background

Historically, Linux distributions used a network of mirrors to distribute their packages, installation media images and so on. This would provide faster downloads for users all over the world who could select a close-by mirror — often inside the very institution they were working from — while also alleviating some of the load on the primary servers.

Gentoo made use of a traditional mirror network to distribute its installation media and distfiles but introduced an innovation: for the small files that make up the Portage tree, Gentoo started using the relatively new rsync protocol instead the traditional ftp and http, guaranteeing that only the files that needed update were downloaded, which made the updates a lot more efficient.

Funtoo inherited that system from Gentoo, but soon innovated again, adopting the then new “git protocol” instead of rsync. Now instead of downloading the files that changed, it downloads only the changes themselves (the git “deltas”), making the updates even more efficient and faster. Also, this meant that the master Portage tree could now be hosted on GitHub, doing away with the burden of maintaining a network of rsync mirror servers for the Portage tree. This move also made it feasible to group the previously monolithic Portage tree into logical “kits”, each in its independent repository within the “meta-repo”.

However, git by its very nature is not suitable for hosting binary files and therefore cannot be used to distribute installation media and pre-built packages, for example. Thus, just moving from rsync to git didn’t mean that Funtoo wouldn’t need to manage a mirror network anymore; installation media, stage tarballs, distfiles and occasional pre-built binaries still needed to be made available for download somehow. At this point, Daniel decided to reach out to CDN77, which generously offered to provide CDN resources to the Funtoo project.

The Funtoo CDN

A CDN (Content Distribution Network) serves the same purpose as the “mirror network”, but it’s hard to even try to compare what they really are. The best-maintained mirror network will look amateurish and won’t compare in terms of performance to a modern CDN service. The CDN77 service uses caching and high-speed links to make the files easily and rapidly downloadable worldwide, using several geographic endpoints that can cache files and communicate rapidly between each other.

For practical purposes, let’s just say that the user doesn’t need to select a “best mirror” and there are no outdated or down mirrors. From the user standpoint, all the available ISO images and stage tarballs appear to be under one single URL: https://build.funtoo.org/, and the distfiles referred to in the ebuilds appear under https://direct.funtoo.org. The CDN will transparently select the fastest route between the user and a server that can deliver the content at the moment of the request. The CDN does not only manage mirrors in strategic points around the world, but also the content can be cached by ISP’s, making it readily available at the fastest speed possible to the users who connect to them. It doesn’t matter if you are in Los Angeles, Bucharest, Jakarta, Buenos Aires or Cape Town, you will have the same fast and reliable experience downloading content from a CDN.

This move completely freed Funtoo from the need for any kind of mirror network for any purpose. At first, only the Funtoo stage tarballs and distfiles were uploaded to the CDN. But there were some distfiles referenced from some ebuilds from the Gentoo snapshots that were using Gentoo mirrors, which risked being altered or removed without notice. Realizing that, Daniel wrote a script to populate our CDN with a full collection of source code, so that this would not become an issue. Then the CDN was made the default “mirror” for all distfile sources and the “fastpull service” was added to the Funtoo Portage, providing users with a fast, reliable and universally accessible download point for all their source code.

But Funtoo is not about ebuilds from Gentoo snapshots. The real deal are the “autogens”, which in theory can generate anything — not just ebuilds — using the logic contained in a “generator”, and a Jinja template. Metatools is essentially an advanced Python-based API for creating ebuilds, which contains useful tools for automatically checking the latest or all the available versions of packages, downloading the sources and generating ebuilds based on information found online or contained inside the downloaded tarballs. The generator can also take parameters from a YAML file, thus allowing a single Python generator to generate ebuilds for hundreds of different packages.

It Starts With the Spider

Metatools itself has a highly-efficient Web spider which is used to download the sources for all autogenned ebuilds. When the autogen is run in developer mode, fastpull only downloads the source tarball to the local computer, but when it’s run on our official regen infrastructure, all distfiles grabbed by the spider are immediately made available on the CDN, so that source code is always available even if the original repository goes offline.

In 2022, the second-generation of our fastpull technology was released, which stores files indexed by their sha512 sum hash rather than their file names. Thus, Portage can request distfiles from our CDN by their sha512 sum hash, and then save them locally with their original file name.  This completely eliminates the possibility of having an infamous Portage “digest mismatch”.

Also in 2022, the ebuild-generation component of metatools gained a new and very powerful feature called “Dynamic Archives”, which allows the autogens to create their own tarballs. These can be modified or repackaged versions of the original source tarballs, maybe with the addition of some icons or documentation downloaded from different places. They can also be a “prepared” version of the sources, so that the generated ebuild can drop the dependencies that would then be needed to prepare the sources or build the documentation. They can be used to build tarballs from git clones that include git submodule sources, which are usually missing from the GitHub tarballs.

This leads us to the biggest story of March, 2023: the new golang and rust extensions to the github-1 generator that showcase the power of the Dynamic Archives within metatools.

The golang Extension

Metatools have had for some time the ability to peek into source tarballs for software written in Go Language to extract the gosum hashes and download urls for its dependencies, which could then be used to create an ebuild with a long SRC_URI, which would hold both the URL for the main package and also all its dependencies. This allowed us to have up-to-date ebuilds for packages that were written in the Go Language.

In order to use this generator, however, it was necessary to use a custom python autogen rather than a generic YAML one. Also, the resulting ebuild was sub-optimal, since it was based on the existing Gentoo go-modules.eclass which required listing every single required go module individually in SRC_URI. For many golang-based ebuilds, this resulted in hundreds of entries in SRC_URI. Even though our CDN is very efficient for downloads, Portage downloads files one at a time, so each entry in SRC_URI takes a minimum of a few seconds each to download. Add a hundred (or two!) entries in SRC_URI, and the download of sources could take 10 or more minutes! Quite annoying.

Fortunately, Funtoo doesn’t have to settle for that sub-optimal experience. Thanks to dynamic archives, invakid404 and drobbins were able to develop a solution. Rather than individually list each required go module in an ebuild, the autogen itself could create a single tarball which contained all necessary go modules, and this tarball would be magically populated on our CDN. The ebuild could now reference one additional file, rather than hundreds, and we were able to magically work around the Portage fetch performance issue. Emerging ebuilds for golang-based packages could be made fast again.

As of April 5, 2023, these dynamic golang autogens are now active in the main tree – net-misc/rclone is one example of such packages.

To make these improvements easier to use, the go-modules.eclass was optimized to transparently use Funtoo’s go-module bundles, and a new extension to the metatools github-1 generator was introduced. To make your golang-based autogen automatically create a “golang bundle” (tarball), just two additional lines are needed (the last two ones in the YAML below):

mypackage_rule:
    generator: github-1
    packages:
        - my package
            extensions:
                - golang

The rust extension

Also included in the harvester/2023-03 branch is the "rust" extension to the github-1 generator, which works in an analogous form to that of the golang extension, with a conjunction of new code added to the existing rust metatools sub and to the github-1 generator. harvester/2023-03 has been merged into "production Funtoo", meaning that this functionality is now fully active and in use.

Next Time

In the April issue, we’ll begin a series of tutorials on Pull Requests and the Funtoo Git Repository and how it works. You’ll learn about the Funtoo tools & metatools, the kit-fixups repository, and end with the git pull request workflow.
 

Welcome to our second official Funtoo Linux newsletter, covering the time period of February 2023! It’s exciting for us to get into a rhythm with newsletter releases. 

We have a much meatier newsletter this month, with recent technical updates, great news about Oleg aka angry_vincent, a profile of Kery, a new user – and his install challenges, and some really useful and detailed info about our bug tracker and how to use it well. Let’s start by covering the latest updates:

Latest Updates

Funtoo Linux now has debian-sources-6.1.12_p1 enabled by default. This appears to be working well with all the key packages that require kernel compatibility (zfs-kmod, nvidia-kernel-drivers).

Harvester’s “harvester/2023-01” branch has been merged into master, meaning that all harvester changes are now part of official releases. The most notable change is an upgrade of binutils to 2.39_p5, as well as a harfbuzz update. We now have “harvester/2023-03” active and ready for PR’s. Our focus for March – updating Xorg. Adriano Bosco (aka “adbosco”) is working on this at the moment.

Oleg and Ukraine

Those who have been a part of the Funtoo community for quite some time are aware that Oleg, aka angry_vincent, used to serve as the project lead for several years, where he competently supported the Funtoo user community with great support and assistance. 

You may not be aware, however, that Oleg lives in Ukraine, and has been very personally impacted by Russia’s invasion of Ukraine, even prior to the “special military operation”. By extension, there is a personal connection between the events in Ukraine and the Funtoo Linux project. Even though the conflict is very far away from most of us, it feels very close.

There is good news here, though. Daniel has been chatting with Oleg recently and Oleg will be assisting with maintenance and improvement of Funtoo from Scratch, and associated efforts, as he is able. We are really happy to have Oleg back in our community. If you see him around online, please welcome him back!

New Jira

We are now running a new version of Jira, which comes with its own set of changes. Let’s look at each one.

Comments

The latest comments on each issue are now at the top of the page, rather than the bottom. This is more convenient for some. However, this feature is still in its infancy, and even with the latest comment at the top of the page, there is a problem –  the “Add Comment” button will still be at the bottom of the page, so if you want to add to the discussion, you have to scroll all the way to the bottom of the page anyway. D’oh! And if it’s an issue with many comments, the latest ones will be hidden, so you can’t easily reference the latest comment to write a response. Double d’oh!

Atlassian, please fix this soon! You can toggle back having the most recent comment at the bottom of the page by clicking this button on the right top of the comments:

Workflow Cleanup

In a feature that is better thought-out, workflow steps are now organized under their own sub-menu, rather than being individual buttons. This is a welcome change which cleans up the UI quite a bit:

New Rules?

In addition to Jira itself, the official “rules” for creating an issue on the Funtoo Bug Tracker (https://bugs.funtoo.org) were updated by Daniel.

Previously, Daniel had a very gentle and metaphorical set of guidelines, which were in green, which appeared when the “Create” button was pressed on bugs.funtoo.org. Now, there are much stricter and direct rules that appear, which are in red. The underlying philosophy hasn’t changed, but the way the rules are being explained has gone through a dramatic change.

The new bug tracker rules are designed to set clear expectations on what is acceptable, to ensure that compliant bug reports are filed. The stricter rules are necessary to provide clear guidance to the community, but are not intended to discourage you from filing issues on the bug tracker. Just please follow these new rules, and you will be fine. Here is the new text:

If you are using personal overlays, custom USE settings or other non-standard things on your system, then your bug will be given low priority or may even be closed with little or no explanation. We are focused on supporting Funtoo users, who use Funtoo's official profiles with minor deviations only.
In your new issue, please include:

• A description of what you were trying to do -- not just the actual breakage. This is called giving us context.
• A description of what happened.
• A description of what you expected to happen.
• How you are being impacted by this issue.
• Please attach build logs and ego profile show output, if appropriate.
• For feature requests, new package additions or other improvements please include an explanation of why the issue is important to you and could be beneficial to the Funtoo community.
• Please include any other information that might help us reproduce the issue.

Please DON'T do the following:
DON'T attach emerge --info output -- instead please attach ego profile show output as is much more useful and will allow us to try to reproduce the issue.
DON'T use anything in Gentoo as justification to do or not do something in Funtoo -- Funtoo is a separate community with its own processes and development and we are under no obligation to do something just because Gentoo does it, or not do something because Gentoo doesn't do it.
Please see the Wolf Pack Philosophy for more background on these rules. Users who repeatedly and flagrantly ignore these rules when reporting bugs may be banned from the Funtoo bug tracker and related community resources.

It’s worth spending some time to discuss the rationale behind this change, and how Funtoo intends to use the bug tracker going forward.

Bug Tracker 101

Let’s dive into some deep thoughts about the bug tracker, which will help you to understand why we do things the way we do, and help you use the bug tracker as an expert.

The Funtoo Project utilizes the bug tracker in a quite peculiar way. Most projects will put in place strict rules in certain areas, but not others. For example, in most communities, you can receive unpleasant feedback from developers if you file a duplicate or if you let it show that you didn't read and understand the documentation thoroughly. On the Funtoo bug tracker, duplicates are welcome: they show that there are more people concerned about that issue, each one with their own personal experiences. On Funtoo, if you struggled enough trying to accomplish something to the point you thought you found a bug, then we consider it to be a bug, even if the issue was caused by something you did wrong. In the Funtoo ideal, everything should “just work” in the most intuitive way possible. If it doesn't, then that fact, in and of itself, is a bug. This is also different from most projects.

Our openness about user experience provides some temptation to use the bug tracker as the project’s main forum for discussing various ideas and improvements – and here is where it is important to understand a subtle but important rule that is enforced:  any idea proposed on the bug tracker as a feature request should include justification as to why this change would be beneficial to you as well as to the larger Funtoo community. This means you should not simply describe the technical change, but also make clear the motivation and reasoning behind it. This is very different from how most bug trackers operate, where pure technical discussions are the norm. In Funtoo, the “heart” of every issue should be a human story and with a clear explanation of how it benefits people. Not sure if others are interested in your idea? At least explain how it benefits you.

So, you have permission to discuss your personal needs on the bug tracker. But at the same time, Daniel doesn’t support the idea of using the bug tracker for general discussion – all comments on each issue should be focused on the specific issue at hand, and moving it towards resolution. And careful attention must be paid by all participants to not “hijack” the issue with tangential comments or ideas. The bug tracker is not an appropriate place for a “general brainstorming”-style discussion. 

If your idea is still in the brainstorming stage, Funtoo Linux does have forums at https://forums.funtoo.org, which are recommended for discussions, as well as Discord and Telegram servers/channels. These are great resources for soliciting feedback on your ideas.

So – did I mention that Funtoo rules were a bit peculiar? These subtle but important rules, when followed consistently, make a huge positive difference for the Funtoo community.

This user-centric way of using the bug tracker – with clear limits – tends to get misunderstood. Users will assume we allow free-flowing discussion, or think we are extremely strict and avoid reporting important issues that are annoying to them personally, thinking we don’t care. Neither one of these extremes is the reality at all. It’s probably fair to say that we are strict in how we use the bug tracker, but we are strictly focused on having a productive and positive force for improving the human experience with Funtoo.

Finally, it’s important to understand that in Funtoo – unlike in Gentoo – we are all using standardized profiles, not every possible USE flag combination – and that the bug tracker is primarily focused on improving our official profiles. Issues related to non-standard settings will be significantly deprioritized. On the other hand, if our official profiles aren’t working for you, that itself is a bug, and you should explain how our default settings are sub-optimal and how we should change them or improve Funtoo so you don’t need to use custom USE settings in the first place.

If in doubt – create an issue. We will give you feedback if it is best handled elsewhere.

New User – Kery

Kery joined the community on February 17th, when he was about to install Funtoo for the first time. He downloaded the GNOME stage3 Westmere tarball and started the installation process, but he would run into weird error messages, like chroot and even bash failing due to “illegal instruction”.

It turns out that Kery had chosen the correct sub-architecture tarball “Westmere tarball” for his Pentium P6200 (Arrandale variant of Westmere), but the fact is that P6200 lacks the extensions SSE4.1, SSE4.2, AES and PCLMUL, which GCC assumes are available on every Westmere chip. The solution was to choose the core2 tarball instead, which matches the Westmere Pentium P6200 actual available instruction set.

Basically, what happens here is that the chip manufacturer and the GCC developers have different opinions on what constitutes an “architecture”, or “sub-architecture” to use the Funtoo terminology. For the manufacturer, “architecture” means how the chip is physically built; for GCC, it means what instruction set is available.

Both are correct most of the time, but in the case of the Westmere chips, not all models ship with the same instruction set.  In the P6200 case, it has the same instruction set as what GCC considers to be a Core2 chip.

Kery also found out that our fchroot tool would silently fail when there was a CPU instruction set incompatibility between the binaries in the tarball and the physical hardware on which fchroot running.

After working through these issues, Kery managed to successfully install Funtoo on his P6200 box using the Core2 tarball, but when he tried to run any GNOME application, the entire session would crash and he’d be thrown back to the login screen. He had followed the instructions in the documentation in the order they were presented, but that didn’t lead to a working GNOME desktop as expected.

It took some digging, but he soon found out that the problem was that he hadn't installed the correct X driver. So, he went back to the documentation to find out how to do it. He learned that he needed to add the “gfxcard-intel” mix-in to get the driver he needed and run a full world update. It turns out that this mix-in, which is normally enabled by default, is not enabled for the core2_64 subarch.

Unfortunately that still didn't work. He couldn't even start the X server now. Eventually, he discovered that he needed the i915 mesa driver, which is provided by the gfxcard-intel-classic mix-in instead, which led to another world update. Basically, he had found another bug, but this time the bug was in the documentation. The “classic” driver enables the “xf86-video-intel” ebuild, which is required for some older Intel integrated graphics chips, but is no longer the recommended default, which is now provided by the modesetting driver enabled by the “gfxcard-intel” mix-in.

Once his video was working, another bug was found. Kery was using the “lxqt” stage3, and he found that LXQt was not being enabled as the default desktop environment. Another bug, and another thing for us to fix.

Probably most Funtoo user (or Gentoo user for that matter) have gone through some struggle when they installed the system for the first time.  What makes Kery's story different is that he happened to go down a path that hit a surprising number of bugs and he communicated everything, both his difficulties and his findings. Each roadblock has its own issue reported on the bug tracker, and we will be addressing every reported issue by improvements to our stages and documentation to prevent other users from encountering similar problems in the future. His efforts in reporting the issues will result in Funtoo being better for everyone. 

Thanks, Kery, for taking the time to do this! We’re going to be working on the issues you reported.

EOL

Thanks everyone for joining us for this month’s newsletter! Our plan is to make this newsletter a true “hub” for our distributed Funtoo community and appreciate you reading all the way to the bottom 🙂

A lot goes on in the Funtoo Community in a month.  Here are some highlights of what happened along the month of January 2023. And the first thing we have to announce this month is the start of this monthly newsletter itself!

Funtoo Newsletter

The lack of consistent and effective communication over time have led even active contributors to remain oblivious for months of certain developments have been put forth by other members. This can make things confusing and even frustrating.

In order to keep everybody up to speed with what's happening, including major software updates, infrastructure upgrades, community rules, new projects and future expectations we will be publishing this short newsletter every month.

Linux 6.1

sys-kernel/debian-sources was upgraded from v.5.18.16_p1 to v6.1.4_p1. This means that now we have '''Linux 6.1''' on Funtoo!  The upgrade was fast-tracked due to issues with Realtek Wi-Fi in the previous official kernel, 5.18.16_p1. You can read more about the Realtek Wi-Fi issues in Funtoo Bug https://bugs.funtoo.org/browse/FL-10937. This issue is confirmed fixed in v6.1.4_p1, but Linux 6.1 brings much more.

The jump from 5.18 to 6.1 brought 15 thousand non-merge commits, with a lot of improvements. Here are some highlights:

• Official support for Intel 4th gen Xeon and 13th gen Core (Raptor Lake) processor.
• Support for ARM-based laptops, such as the Lenovo ThinkPad X13s with the Qualcomm Snapdragon 8cx Gen 3 chip.
• Support for the new Intel Arc dedicated GPU's.
• Better performance for AMD Ryzen Threadripper and AMD EPYC.
• Better Btrfs performance.
• Countless new drivers, for devices ranging from GPU's to hardware sound and gamepads.

Whether or not you will see any difference or not depends a lot on your particular CPU, GPU and other devices, but some people did report that everything seemed to run faster on the new kernel.

Plasma-5.25.5 / KDE Gear 22.08.1

These are mostly bugfix releases that came out in 2022's last quarter.  After a good amount of internal testing, the new releases have been incorporated into Funtoo in January 2023.
Thanks to R0B for pushing this forward and all who tested it (Special thanks to Morphmex!) Some minor bugs were encountered after this update during last month, but they were quickly resolved.

Scanner Support for Next

Some dependencies had been “beard-trimmed” from Next-release and the scanner support was broken.

Now, all the Sane Frontends/Backends, XSane and scanner drivers are fully supported on Next, including OCR capabilities.

Development Languages

Funtoo now has Go v1.19.5 and Rust to 1.57.0. Python 2.7 is still available but is deprecated, as there's no upstream support anymore.  Some packages still require Python 2, but we are working towards upgrading or removing those packages.

OpenCL eselect deprecation

Funtoo has an eselect module for OpenCL, which allowed the user to choose among different OpenCL implementations (Mesa, AMD, Intel, NVIDIA, etc.).  However, in most cases that job can be done by dev-libs/ocl-icd and nvidia-drivers now installs files in different locations, which prevent eselect from working.

Some work has already been done towards removing the dependency on app-eselect/eselect-opencl from the ebuilds and we still have 9 ebuilds on the tree that do.  When that work is concluded, app-eselect/eselect-opencl itself will be removed from Funtoo.

Development Model Changes

Funtoo has long adopted the BDFL model (like the Linux Kernel Project), where the decision making was centralized around Daniel Robbins and the staff just followed along.

From December 2022, up to the first half of January 2023, a new system was introduced, where Daniel stepped down and BDFL and the decisions were taken by consensus among the staff, as Daniel focused on moving his family across the United States. In the Funtoo project, it was an interesting experience. There was very good work done by borisp, siris, coffnix, adbosco and others on a technical level -- and things looked like they were going in a good direction. But some problems were encountered.

Daniel jumped back into BDFL role by the end of January. Daniel saw evidence that too much was given to this team, too quickly, with not enough support, training and clear responsibilities. While they did an admirable job in many areas, 1.4-release had some unresolved breakage lasting several weeks, and the python packaging package had been broken and gone unfixed for some time.

While Daniel was originally very upset, he concluded in the end that the staff were put in a challenging situation, performed admirably especially considering the circumstances, and apologized for getting upset in the first place. He acknowledged that he often has unreasonable expectations. More work needs to be done to find the optimal model to allow community leaders to succeed.

This realization gave birth to additional development of a more defined community leadership model, which shall be gradually implemented throughout the next few months. It will likely be a mentorship model and as it takes shape, announcements will be made to recruit people interested in learning and contributing in different areas of the Funtoo Project, and an emphasis will be placed on skill building. Stay tuned!