[BSD] FreeBSD Project 2020 Q2 Status Report
DaVieS
davies at npulse.net
2020. Júl. 18., Szo, 00:12:10 CEST
FreeBSD Project Quarterly Status Report - Second Quarter 2020
Introduction
This report will be covering FreeBSD related projects between April and
June, and covers a diverse set of topics ranging from kernel updates
over userland and ports, as well to third-party work.
Some hilights picked with the roll of a d100 include, but are not
limited to, the ability to forcibly unmounting UFS when the underlying
media becomes inaccessible, added preliminary support for Bluetooth Low
Energy, a introduction to the FreeBSD Office Hours, and a repository of
software collections called potluck to be installed with the pot
utility, as well as many many more things.
As a little treat, readers can also get a rare report from the
quarterly team.
Finally, on behalf of the quarterly team, I would like to extend my
deepest appreciation and thank you to salvadore@, who decided to take
down his shingle. His contributions not just the quarterly reports
themselves, but also the surrounding tooling to many-fold ease the
work, are immeassurable.
We hope you find the report as interesting as we have,
Daniel Ebdrup Jensen (debdrup@), on behalf of the quarterly team.
__________________________________________________________________
FreeBSD Team Reports
* FreeBSD Foundation
* FreeBSD Core Team
* FreeBSD Release Engineering Team
* Cluster Administration Team
* Continuous Integration
* Ports Collection
* FreeBSD Office Hours
* Quarterly Status Reports Team
Projects
* FreeBSD on Microsoft HyperV and Azure
* Git Migration Working Group
* Lua Usage in FreeBSD
* Linux compatibility layer update
* NFS over TLS implementation
Kernel
* SoC audio framework and more audio drivers
* bhyve - NVMe emulation improvements
* Bluetooth Support
* DRM Drivers Update
* DTS Update
* ENA FreeBSD Driver Update
* Forcible Unmount of UFS/FFS Filesystems on Disk Failure
* i.MX 8M support
* Intel wireless and 11ac update
* amd64 5-Level Paging Structures support
* Not-transparent superpages
* NXP ARM64 SoC support
* amd64 pmap Fine-grained pv lists locking
* Lockless routing lookups and scalable multipath
* ZSTD Compression in ZFS
* CheriBSD 2020 Q2
Architectures
* Continuous Integration on !x86
* FreeBSD/RISC-V Project
Userland Programs
* Import of new implementation of bc and dc
* Binutils Retirement
* Run-Time Dynamic Linker improvements
* VHDX support in mkimg(1)
Ports
* Bastille
* KDE on FreeBSD
* Haskell on FreeBSD
* rtsx - Porting driver for Realtek SD card reader from OpenBSD
* Valgrind updates
Documentation
* FreeBSD Translations on Weblate
Miscellaneous
* FreshPorts
* PCI passthrough with bhyve on Intel and for OpenBSD guests
* SageMath
Third-Party Projects
* chaifi - a tool to simplify joining public WiFi networks
* MixerTUI
* Potluck - Flavour & Image Repository for pot
__________________________________________________________________
FreeBSD Team Reports
Entries from the various official and semi-official teams, as found in
the Administration Page.
FreeBSD Foundation
Contact: Deb Goodkin <deb at FreeBSDFoundation.org>
The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated
to supporting and promoting the FreeBSD Project and community
worldwide. Funding comes from individual and corporate donations and is
used to fund and manage software development projects, conferences and
developer summits, and provide travel grants to FreeBSD contributors.
The Foundation purchases and supports hardware to improve and maintain
FreeBSD infrastructure and provides resources to improve security,
quality assurance, and release engineering efforts; publishes marketing
material to promote, educate, and advocate for the FreeBSD Project;
facilitates collaboration between commercial vendors and FreeBSD
developers; and finally, represents the FreeBSD Project in executing
contracts, license agreements, and other legal arrangements that
require a recognized legal entity.
Here are some highlights of what we did to help FreeBSD last quarter:
COVID-19 Impact to the Foundation
Like other organizations, we put policies in place for all of our staff
members to work from home. We also put a temporary ban on travel for
staff members. We are continuing our work supporting the community and
Project, but some of our work and responses may be delayed because of
changes in some of our priorities and the impact of limited childcare
for a few of our staff members.
Partnerships and Commercial User Support
We help facilitate collaboration between commercial users and FreeBSD
developers. We also meet with companies to discuss their needs and
bring that information back to the Project. Not surprisingly, the stay
at home orders, combined with our company ban on travel during Q2 made
in-person meetings non-existent. However, the team was able to continue
meeting with our partners and commercial users virtually. These
meetings help us understand some of the applications where FreeBSD is
used.
Fundraising Efforts
Last quarter we raised $268,400! Thank you to the individuals and
organizations that stepped in to help fund our efforts. We'd like to
thank Netflix, employees of Nginx, Beckhoff Automation, and Mozilla
Foundation for their large contributions last quarter, which helped
bring our 2020 fundraising effort to $339k. We hope other organizations
will follow their lead and give back to help us continue supporting
FreeBSD.
These are trying times, and we deeply appreciate every donation that
has come in from $5 to $150,000. We're still here giving 110% to
supporting FreeBSD!
We are 100% funded by donations, and those funds go towards software
development work to improve FreeBSD, FreeBSD advocacy around the world,
keeping FreeBSD secure, continuous integration improvements, sponsoring
BSD-related and computing conferences (even the virtual events!), legal
support for the Project, and many other areas.
Please consider making a donation to help us continue and increase our
support for FreeBSD.
We also have the Partnership Program, to provide more benefits for our
larger commercial donors. Find out more information about the
partnership program and share with your companies!
OS Improvements
A number of FreeBSD Foundation grant recipients started, continued
working on, or completed projects during the second quarter. These
include:
* WiFi improvements
* Linuxulator application compatibility
* DRM / Graphics driver updates
* Zstd compression for OpenZFS
* Online RAID-Z expansion
* if_bridge performance improvements
You can find more details about most of these projects in other
quarterly reports.
Staff members also worked on a number of larger projects, including:
* Run-Time Dynamic Linker (rtld) improvements
* Improved FreeBSD support on Microsoft HyperV and Azure
* Fine-grained locking for amd64 pmap
* 5-level paging structures for amd64
* Non-transparent superpages
* Migration to a Git repository
* Tool chain modernization
Many of these projects also have detailed entries in other quarterly
report entries.
Staff members also put in significant effort in many ways other than
larger, individual projects. These include assisting with code reviews,
bug report triage, security report triage and advisory handling,
addressing syzkaller reports, and ongoing maintenance and bug fixes in
functional areas such as the tool chain, developer tools, virtual
memory kernel subsystem, low-level x86 infrastructure, sockets and
protocols, and others.
University of Waterloo Co-op
Foundation co-op students Colin, Tiger, and Yang completed their winter
2020 work term during the second quarter, and continued on with the
next school term in their respective programs. Although COVID-19
presented a unique challenge and prompted an abrupt transition to
remote work just over half way through the term, all three learned a
lot and provided positive contributions to the FreeBSD Project and to
the Foundation.
A few projects that were in progress or completed during the work term
were committed to the FreeBSD tree in the second quarter.
Continuous Integration and Quality Assurance
The Foundation provides a full-time staff member who is working on
improving continuous integration, automated testing, and overall
quality assurance efforts for the FreeBSD project.
During the second quarter of 2020, Foundation staff continued improving
the Project's CI infrastructure, monitoring regressions and working
with contributors to fix the failing build and test cases. The setting
up of VM host for CI jobs and staging environment is in progress. We
are also working with other teams in the Project for their testing
needs. For example, we added jobs for running full tests on non-x86
architectures. We are also working with many external projects and
companies to improve their support of FreeBSD.
See the FreeBSD CI section of this report for completed work items and
detailed information.
Supporting FreeBSD Infrastructure
The Foundation provides hardware and support to improve FreeBSD
infrastructure. Last quarter, we continued supporting FreeBSD hardware
located around the world. We started working on getting the new NYI
Chicago colocation facility prepared for some of the new FreeBSD
hardware we are planning on purchasing. NYI generously provides this
for free to the Project.
FreeBSD Advocacy and Education
A large part of our efforts are dedicated to advocating for the
Project. This includes promoting work being done by others with
FreeBSD; producing advocacy literature to teach people about FreeBSD
and help make the path to starting using FreeBSD or contributing to the
Project easier; and attending and getting other FreeBSD contributors to
volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD
presentations.
The FreeBSD Foundation sponsors many conferences, events, and summits
around the globe. These events can be BSD-related, open source, or
technology events geared towards underrepresented groups. We support
the FreeBSD-focused events to help provide a venue for sharing
knowledge, to work together on projects, and to facilitate
collaboration between developers and commercial users. This all helps
provide a healthy ecosystem. We support the non-FreeBSD events to
promote and raise awareness of FreeBSD, to increase the use of FreeBSD
in different applications, and to recruit more contributors to the
Project. As is the case for most of us in this industry, COVID-19 has
put our in-person events on hold. In addition to attending virtual
events, we are continually working on new training initiatives and
updating our selection of how-to guides to facilitate getting more
folks to try out FreeBSD.
Check out some of the advocacy and education work we did last quarter:
* Silver sponsor of BSDCan 2020. The event was held virtually, June
2-6, 2020
* Community Sponsor of Rootconf 2020. The event was held virtually,
June 19-20, 2020
* Annual FreeBSD Day, June 19. This year's celebration was postponed
in support of Juneteeth. However the activities surrounding FreeBSD
Day have been transformed into an ongoing series of online
sessions. See FreeBSD Fridays below for more information.
* Presented 27 Years of FreeBSD and Why You Should Get Involved as
part of a Linux Professional Institute series of webinars on June
24, 2020.
* Attended and presented at the virtual Open Source Summit 2020.
* Announced FreeBSD Fridays: A series of 101 classes designed to get
you started with FreeBSD. Find out more in the announcement
* Participated as an Admin for Google Summer of Code 2020
* Participated in the new FreeBSD Office Hours series including
holding our own Foundation led office hours. Videos from the one
hour sessions can be found on the Project's YouTube Channel. You
can watch ours here.
In addition to the information found in the Development Projects update
section of this report, take a minute to check out the latest update
blogs:
* 5x if_bridge Performance Improvement
* My Experience as a FreeBSD Foundation Co-Op Student
Keep up to date with our latest work in our Bi-Monthly newsletters.
Mellanox provided an update on how and why they use FreeBSD in our
latest Contributor Case Study.
We help educate the world about FreeBSD by publishing the
professionally produced FreeBSD Journal. As we mentioned previously,
the FreeBSD Journal is now a free publication. Find out more and access
the latest issues on the Journal site.
You can find out more about events we attended and upcoming events.
We have continued our work with a new website developer to help us
improve our website. Work is nearly complete to make it easier for
community members to find information more easily and to make the site
more efficient. We look forward to unveiling the refreshed site in Q3.
Foundation Board Meeting
Our annual board meeting was held on Tuesday June 2, 2020. We normally
hold this meeting the Tuesday before BSDCan, in Ottawa, Ontario,
Canada, but with the company travel ban, and the conference going
virtual, our meeting went virtual for the first time. The purpose of
the annual board meeting is to hold our board director and officer
elections, review work accomplished over the past year, and put
together strategic goals for the upcoming 12 months.
The board generally has two all-day board meetings each year, this one,
and a more informal one in January, typically held in Berkeley. Both
meetings allow us to connect, reevaluate and discuss new ideas, while
assessing what we should do to help the Project.
Some of our longer-term goals include Growing User and Developer
Communities, Developing Training and OS Course Content, Improving
desktop/laptop experience, Promoting FreeBSD (as you can see in all the
advocacy work listed above), and Improving Testing Capabilities.
Results of the director and officer elections were:
* Justin Gibbs (President)
* Benedict Reuschling (Vice President)
* Kirk McKusick (Treasurer)
* Philip Paeps (Secretary)
* Deb Goodkin (Assistant Secretary)
* Robert Watson (Director)
* Hiroki Sato (Director)
* George Neville-Neil (Director)
Find out more about the FreeBSD Foundation Board of Directors on our
website.
Legal/FreeBSD IP
The Foundation owns the FreeBSD trademarks, and it is our
responsibility to protect them. We also provide legal support for the
core team to investigate questions that arise.
Go to the FreeBSD Foundation's web site to find out how we support
FreeBSD and how we can help you!
__________________________________________________________________
FreeBSD Core Team
Contact: FreeBSD Core Team <core at FreeBSD.org>
The FreeBSD Core Team is the governing body of FreeBSD.
The Core Team held 10 meetings during the second quarter of 2020,
including a 2020-05-21 joint meeting with members of the FreeBSD
Foundation. Here are some highlights from that meeting:
* Deb requested guidance on how the Foundation can support the
community. Core and Foundation members believe that more developer
support is necessary to fill gaps in areas where commercial
customers do not provide backing. The clearest example of such a
gap is the desktop experience, including graphics and wireless
support. What makes this request different from past requests is
that rather than support for one-time projects, ongoing positions
are necessary for a consistently high-quality desktop experience.
"FreeBSD not being able to run on your laptop is the first step to
irrelevance." Ed Maste
* Both teams discussed topics for upcoming sessions of FreeBSD Office
Hours, informal FreeBSD video conferences that anyone can attend.
Everyone agreed that the Office Hours have been a useful way for
different parts of the Project to engage with each other and with
the wider community. Kudos to Allan Jude for initiating the Office
Hours and for everyone who has helped make them a success by
hosting or attending sessions.
* Both teams agreed that they should meet once per quarter.
The second annual community survey closed on 2020-06-16. The purpose of
the survey is to collect data from the public to help guide the
Project's efforts and priorities. As an example, last year's survey
results helped initiate the Project's conversion to Git. Thank you to
all who took the time to respond. The results will be released soon.
The Core-initiated Git Working Group continued to make progress, but
there are still some remaining issues to be worked out with the
translation from Subversion. Hopefully the new Git src repository will
be ready for use this summer. A beta version has been published for
people to test and a preliminary version of a Using Git for FreeBSD
Development primer will soon be ready to share. Core, the Git Working
Group, and Release Engineering are working towards the goal of
releasing 12.2 from the new Git repository.
Following the results of a Core-initiated developer survey, The FreeBSD
Project has adopted a new LLVM-derived [code of
conduct](https://www.freebsd.org/internal/code-of-conduct.html).
The eleventh FreeBSD Core Team was elected by active developers. From a
pool of 23, the 9 successful candidates for core.11 are:
* Sean Chittenden (seanc, incumbent)
* Baptiste Daroussin (bapt)
* Kyle Evans (kevans)
* Mark Johnston (markj)
* Scott Long (scottl)
* Warner Losh (imp, incumbent)
* Ed Maste (emaste)
* George V. Neville-Neil (gnn)
* Hiroki Sato (hrs, incumbent)
A new Core Team secretary, Muhammad Moinur Rahman (bofh), was
unanimously approved by core.11. The outgoing core team met three times
with the new core team to help with the transition. Core.10 wishes
core.11 a successful term.
__________________________________________________________________
FreeBSD Release Engineering Team
Links
FreeBSD 11.4-RELEASE announcement
URL: https://www.freebsd.org/releases/11.4R/announce.html
FreeBSD 11.4-RELEASE schedule
URL: https://www.freebsd.org/releases/11.4R/schedule.html
FreeBSD development snapshots
URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/
Contact: FreeBSD Release Engineering Team <re at FreeBSD.org>
The FreeBSD Release Engineering Team is responsible for setting and
publishing release schedules for official project releases of FreeBSD,
announcing code slushes, and maintaining the respective branches, among
other things.
During the second quarter of 2020, the Release Engineering Team started
work on the 11.4-RELEASE cycle, the fifth release from the stable/11
branch. The release cycle went quite smoothly, with both BETA3 and RC3
removed from the schedule. This allowed the final release to occur one
week earlier than originally scheduled, which was announced June 16.
FreeBSD 11.4-RELEASE is expected to be the final 11.x release.
The FreeBSD Release Engineering Team would like to thank everyone
involved in this cycle for their hard work.
Additionally throughout the quarter, several development snapshots
builds were released for the head, stable/12, and stable/11 branches.
Much of this work was sponsored by Rubicon Communications, LLC
(netgate.com) and the FreeBSD Foundation.
__________________________________________________________________
Cluster Administration Team
Links
Cluster Administration Team members
URL: https://www.freebsd.org/administration.html#t-clusteradm
Contact: Cluster Administration Team <clusteradm at FreeBSD.org>
The FreeBSD Cluster Administration Team consists of the people
responsible for administering the machines that the Project relies on
for its distributed work and communications to be synchronised. In this
quarter, the team has worked on the following:
* Upgrade all x86 ref- and universe-machines
* Setup Amsterdam (PKT) mirror
* Solve hardware issue for bugzilla and svnweb backend
* Setup public beta git server
* Decommission CyberOne Data (CYB) mirror
* Ongoing systems administration work:
+ Accounts management for committers.
+ Backups of critical infrastructure.
+ Keeping up with security updates in 3rd party software.
Work in progress:
* Setup Malaysia (KUL) mirror
* Setup Brazil (BRA) mirror
* Review the service jails and service administrators operation.
* Infrastructure of building aarch64 and powerpc64 packages
+ NVME issues on PowerPC64 Power9 blocking dual socket machine
from being used as pkg builder.
+ Drive upgrade test for pkg builders (SSDs) courtesy of the
FreeBSD Foundation.
+ Boot issues with Aarch64 reference machines.
* New NYI.net sponsored colocation space in Chicago-land area.
* Work with git working group
* Check new hardware requirement from other teams
* Searching for more providers that can fit the requirements for a
generic mirrored layout or a tiny mirror
__________________________________________________________________
Continuous Integration
Links
FreeBSD Jenkins Instance
URL: https://ci.FreeBSD.org
FreeBSD Hardware Testing Lab
URL: https://ci.FreeBSD.org/hwlab
FreeBSD CI artifact archive
URL: https://artifact.ci.FreeBSD.org
FreeBSD CI weekly report
URL: https://hackmd.io/@FreeBSD-CI
FreeBSD Jenkins wiki
URL: https://wiki.freebsd.org/Jenkins
Hosted CI wiki
URL: https://wiki.freebsd.org/HostedCI
3rd Party Software CI
URL: https://wiki.freebsd.org/3rdPartySoftwareCI
Tickets related to freebsd-testing@
URL: https://preview.tinyurl.com/y9maauwg
FreeBSD CI Repository
URL: https://github.com/freebsd/freebsd-ci
Contact: Jenkins Admin <jenkins-admin at FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu at FreeBSD.org>
Contact: freebsd-testing Mailing List
Contact: IRC #freebsd-ci channel on EFNet
The FreeBSD CI team maintains the continuous integration system for the
FreeBSD project. The CI system firstly checks the committed changes can
be successfully built, then performs various tests and analysis over
the newly built results. The artifacts from the build jobs are archived
in the artifact server for further testing and debugging needs. The CI
team members examine the failing builds and unstable tests and work
with the experts in that area to fix the codes or adjust test
infrastructure. The details of these efforts are available in the
weekly CI reports.
During the second quarter of 2020, we continue working with the
contributors and developers in the project for their testing needs and
also keep working with external projects and companies to improve their
support of FreeBSD.
Important changes:
* All -test jobs will run tests under /usr/tests, previously only x86
architectures doing this. See the Continuous Integration on !x86
section in this report for more information.
* Compression algorithm of disk images on the artifact server has
been changed to zstd to speed up compression and decompression.
* The build and test results will be sent to the dev-ci mailing list
soon. Feedback and help with analysis is very appreciated!
New jobs added:
* https://ci.freebsd.org/job/FreeBSD-head-armv7-test/
* https://ci.freebsd.org/job/FreeBSD-head-aarch64-test/
* https://ci.freebsd.org/job/FreeBSD-head-mips64-test/
* https://ci.freebsd.org/job/FreeBSD-head-powerpc64-test/
Work in progress:
* Collecting and sorting CI tasks and ideas here
* Testing and merging pull requests in the the FreeBSD-ci repo
* Setting up a builder dedicated to run jobs using provisioned VMs.
* Setting up the CI stage environment and putting the experimental
jobs on it
* Implementing automatic tests on bare metal hardware
* Adding drm ports building tests against -CURRENT
* Planning to run ztest and network stack tests
* Adding external toolchain related jobs
* Improving the hardware lab to be more mature and adding more
hardware
* Helping more 3rd software get CI on FreeBSD through a hosted CI
solution
* Working with hosted CI providers to have better FreeBSD support
Please see freebsd-testing@ related tickets for more WIP information,
and don't hesistate to join the effort!
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Ports Collection
Links
About FreeBSD Ports
URL: https://www.FreeBSD.org/ports/
Contributing to Ports
URL:
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports-contributing.html
FreeBSD Ports Monitoring
URL: http://portsmon.freebsd.org/index.html
Ports Management Team
URL: https://www.freebsd.org/portmgr/index.html
Contact: René Ladan <portmgr-secretary at FreeBSD.org>
Contact: FreeBSD Ports Management Team <portmgr at FreeBSD.org>
The Ports Management Team is responsible for overseeing the overall
direction of the Ports Tree, building packages, and personnel matters.
Below is what happened in the last quarter.
There are currently 2,373 open ports PRs of which 526 are unassigned,
for a total of 39,628 ports. In the last quarter there were 10,315
commits to HEAD and 476 to the quarterly branch by respectively 178 and
65 committers. Compared to the quarter before, this means a significant
increase in commits and also a slight decrease in open PRs.
During the last quarter, we welcomed Hiroki Tagato (tagattie@). We said
goodbye to seanc@, zleslie@, gnn@ and salvadore at .
A few default versions got bumped:
* Java (new) at 8
* Lazarus to 2.0.8
It is now possible to write pkg scripts in Lua instead of sh.
They have two advantages over their sh versions:
* they run in a Capsicum sandbox
* they respect rootdir, the directory which pkg will use as the
starting point to install all packages under.
Some user-facing packages were also updated:
* pkg to 1.14.6
* Firefox to 78.0.1
* Thunderbird to 68.10.0
* Chromium to 83.0.4103.116
* Ruby to 2.5.8, 2.6.6, and 2.7.1
* Qt5 to 5.14.2
During the last quarter, antoine@ ran 55 exp-runs to test port version
updates, make liblzma use libmd, flavor devel/scons and Lua ports, add
and update library functions in the base system, make malloc.h usable
again, remove as(1) from the base system, and augment sed(1) with -f.
__________________________________________________________________
FreeBSD Office Hours
Links
Office Hours on the FreeBSD Wiki
URL: https://wiki.freebsd.org/OfficeHours
Poll: What time would you prefer Office Hours be at
URL: https://forms.gle/3HjjRx9KMcM3SL4H7
live.FreeBSD.org: Aggregation of Live streams
URL: https://live.freebsd.org/
Contact: Allan Jude <allanjude at freebsd.org>
Starting on the first of April 2020, the FreeBSD project has started
hosting regular video streams to foster greater communication within
the wider FreeBSD community. The first of these sessions took the form
of a public question and answer session, which drew over 60
participants. A second session was held two weeks later at a time more
appropriate for those in Asia, but only drew 20 participants. With the
help of the FreeBSD Foundation, we ran a poll to discover what times
worked best for the greatest number of people.
On May 13th the FreeBSD Foundation hosted a session where the community
could ask questions of or about the foundation. On May 27th many of the
candidates for the new FreeBSD Core Team joined an office hours session
to answer questions from the community. Finally on June 24th another
general question and answer office hours was held.
Each office hours session consists of a video meeting of some FreeBSD
developers or other subject matter experts, live streamed along with an
IRC chat room for viewers to pose questions to the panel. The stream is
recorded and posted to the official FreeBSD youtube channel.
If you would like to host an office hours session, please contact:
* Allan Jude
* Anne Dickison
Sponsor: ScaleEngine Inc. (video streaming)
__________________________________________________________________
Quarterly Status Reports Team
Links
Quarterly status reports
URL: https://www.freebsd.org/news/status/
Git repository
URL: https://github.com/freebsd/freebsd-quarterly
Contact: Quarterly Status Reports <quarterly at FreBSD.org>
Contact: Daniel Ebdrup Jensen <debdrup at FreeBSD.org>
The Quarterly Status Reports Team collects and publish the reports that
you are reading right now.
Many improvements have been done recently and thus we believe it is
useful that the Quarterly Status Reports Team submits a report. Not all
the changes below are specific to the last quarter, but we list them
here anyway since we did not write an entry for earlier reports.
* Reports are now built using Makefiles. Among the many advantages,
this allows us to easily sort reports logically. Indeed, starting
with 2019Q4, all reports are sorted logically, while before they
were sorted alphabetically within each category.
* The conversion from markdown to docbook was performed using a
python script, with some known bugs. Salvadore has rewritten the
script using perl fixing most of the bugs. Some features are
missing and many improvements are possible, but the script is very
unlikely to receive any change since it will become obsolete as
soon as the conversion to Hugo/AsciiDoctor is completed.
* Another perl script to ease the preparation of the mail version of
the reports was written.
* One more perl script has been written to allow the quarterly team
to send quarterly calls automatically using a cron job. We used it
this quarter for the first time.
* As you might have noticed, last quarterly calls have been sent to
freebsd-quarterly-calls@: this is a new mailing list to which you
can subscribe to receive calls for quarterly reports. Please note
this is a moderated list, with very low traffic and a high signal
to noise ratio.
* If you read carefully the last quarterly calls, you should have
noticed that we now ask you to send reports to
quarterly-submissions@ instead of quarterly at . This was done to help
the quarterly team distinguishing internal discussions from
submissions. Please keep in mind however that the quarterly team
prefers receiving pull requests, as they ease the administrative
work.
We would like to thank philip@, from the postmaster team, for having
created the freebsd-quarterly-calls@ mailing list and the
quarterly-submissions@ address for us.
__________________________________________________________________
Projects
Projects that span multiple categories, from the kernel and userspace
to the Ports Collection or external projects.
FreeBSD on Microsoft HyperV and Azure
Links
FreeBSD on MicrosoftAzure wiki
URL: https://wiki.freebsd.org/MicrosoftAzure
FreeBSD on Microsoft HyperV
URL: https://wiki.freebsd.org/HyperV
Contact: FreeBSD Integration Services Team <bsdic at microsoft.com>
Contact: Wei Hu <whu at FreeBSD.org>
Contact: Li-Wen Hsu <lwhsu at FreeBSD.org>
HyperV socket for FreeBSD implemented by Wei was checked into FreeBSD
head branch on May 20th as r361275. It supports guest/host
communications without the need for networking. Some HyperV and Azure
features rely on this to be available in guests.
Details of HyperV Socket is available here.
This project is sponsored by Microsoft.
Li-Wen is working on the FreeBSD release code related to Azure for the
-CURRENT, 12-STABLE and 11-STABLE branches. The work-in-progress is
available here. The 12.1-RELEASE image on Azure Marketplace is
published. The work on the 11.4-RELEASE image on Azure Marketplace is
in progress.
This project is sponsored by The FreeBSD Foundation.
__________________________________________________________________
Git Migration Working Group
Links
Git conversion tooling repo
URL: https://github.com/freebsd/git_conv
FreeBSD-git mailing list
URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git
Beta doc git repo
URL: https://cgit-beta.FreeBSD.org/doc
Beta ports git repo
URL: https://cgit-beta.FreeBSD.org/ports
Beta src git repo
URL: https://cgit-beta.FreeBSD.org/src
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Warner Losh <imp at FreeBSD.org>
Contact: Ulrich Spörlein <uqs at FreeBSD.org>
Work continues on FreeBSD's migration from Subversion to Git. Ulrich
has iterated on updates to svn2git in order to improve the fidelity of
the conversion, particularly in regards to vendor (contrib) code
updates. We believe the conversion is now at an acceptable state, but
may make minor adjustments if additional issues are found. We expect to
push modifications to the converter every two weeks (first and third
Sunday of the month). This means that commit hashes in the beta repo
will remain stable for at least two weeks at a time, to allow others to
test and experiment with the beta repo.
We are now working on updating FreeBSD processes and documentation.
This includes:
* Writing a Git Primer, akin to the existing Subversion primer
* Updates to the Security Team's tools and processes
* Release engineering updates
* Ports and packages process updates
Those with an interest in the migration to Git are encouraged to
subscribe to the FreeBSD-git mailing list and test out the beta src,
ports, and/or doc repositories.
You are also welcome check out the wiki, issues, README and other
documentation at the Git conversion tooling repo.
We expect to be ready for the migration in the next quarter.
Sponsor: The FreeBSD Foundation (in part)
__________________________________________________________________
Lua Usage in FreeBSD
Contact: Ed Maste <emaste at FreeBSD.org>
Contact: Kyle Evans <kevans at FreeBSD.org>
Contact: Ryan Moeller <freqlabs at FreeBSD.org>
Lua is a small, efficient scripting language that FreeBSD imported
before FreeBSD 12.0 for use in the bootloader. Since then, several
projects outside of the bootloader have gained some amount of traction
with Lua usage:
* /usr/libexec/flua is now installed for internal usage
* makesyscalls.sh was rewritten in Lua
* pkg has gained support for lua scripts
* lldb in the base system now supports lua scripting
FreeBSD Lua ("flua") is a version of the lua interpreter that has
several modules built-in for convenient usage within the base system.
flua is installed with a non-standard name and in a location not
included in $PATH so that it is not accidentally found by third-party
software or configure scripts. The FreeBSD project makes no guarantees
about upgrade cadence or module stability. That said, it is available
for use by downstream projects and FreeBSD users aware of those
limitations.
Previous work with flua includes, for example, adding libucl support
and future work includes libifconfig support for scripting usage.
People interested in working with Lua in FreeBSD are welcome to get in
contact to discuss other project ideas. To name a couple of potential
projects, some interesting modules that have not been started but could
prove useful (listed in no particular order):
* libcrypt
* libexpat
* libjail
* libnv
* libxo
__________________________________________________________________
Linux compatibility layer update
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
Earlier Linuxulator work focused on code cleanups and improving
diagnostic tools. Work has now shifted from cleanups to fixing actual
applications. Current status is being tracked at Linux app status Wiki
page. Initial focus is on applications that don't involve X11, mostly
because they tend to be easier to test and debug, and the bug fixes are
not application-specific.
Example problems fixed include buggy madvise(2) handling, which could
break applications linked against jemalloc; uname(2) returning wrong
results for 32 bit apps, which caused problems for Steam; recvmsg(2)
and accept(2) being broken in some circumstances, which was breaking
Redis; and missing support for SO_REUSEPORT, SO_SNDBUFFORCE,
SO_RCVBUFFORCE, and SO_PROTOCOL, which spammed the log files when
running the Python regression test suite. The default soft open files
limit is now automatically adjusted to 1024, as several Linux apps
iterate over all the file descriptors up to that limit instead of using
closefrom(2).
There's ongoing work on cleanups and the debugging framework for Linux
compatibility, such as logging warnings for unrecognized system call
parameters, or adding the compat.linux.debug sysctl to turn the
warnings off.
The Linux Test Project tests that are being run as part of the the
FreeBSD Continuous Integration infrastructure has been upgraded to
20200605 snapshot. This raised the number of test cases from 3670 to
3749, and, predictably, also the number of failures, from 583 to 647.
There's still a lot to do:
* There are pending reviews for patches that add extended attributes
support, and fexecve(2) syscall, and they require wrapping up and
committing
* There are over 500 failing LTP tests. Some of them are false
positives, some are easy to fix bugs, and some require adding new
system calls. Any help is welcome.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
NFS over TLS implementation
Contact: Rick Macklem <rmacklem at freebsd.org>
In an effort to improve NFS security, an internet draft which I expect
will become an RFC soon specifies the use of TLS 1.3 to encrypt all
data traffic on a Sun RPC connection used for NFS.
Although NFS has been able to use sec=krb5p to encrypt data on the
wire, this requires a Kerberos environment and, as such, has not been
widely adopted. It also required that encryption/decryption be done in
software, since only the RPC message NFS arguments are encrypted. Since
Kernel TLS is capable of using hardware assist to improve performance
and does not require Kerberos, NFS over TLS may be more widely adopted,
once implementations are available.
The project to implement this has largely been completed. The code will
slowly be merged into head/current and at least the kernel portion
should be in FreeBSD-13.
To support clients such as laptops, the daemons that perform the TLS
handshake may optionally handle client X.509 certificates from a site
local CA. There are now exports(5) options to require client(s) to
provide a valid X.509 certificate.
The code is now available for testing. See:
https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt Setting up
system(s) for testing is still a little awkward, as explained by the
above rough document.
The main limitation in the current implementation is that it uses
TLS1.2 and not TLS1.3. This should change once the KERN_TLS rx patch
includes TLS1.3 support. Also, testing of pNFS configurations has not
yet been done, but will be tested soon.
Third party testing would be appreciated.
__________________________________________________________________
Kernel
Updates to kernel subsystems/features, driver support, filesystems, and
more.
SoC audio framework and more audio drivers
Links
rk3399_audio
URL: https://github.com/gonzoua/freebsd/tree/rk3399_audio
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
SoC audio framework made a good progress since last report.
Support for AUX devices was added (devices like auxiliary amplifiers
that are not part of main CODEC chip).
To verify the framework design following audio drivers were added:
recording support for RT5640 CODEC(Firefly-RK3399), Allwinner I2S,
Alwinners sun8i and A64 CODECs (Pine A64+), both recording and
playback.
Current work in progress is RK3328 CODEC (Rock64) and ES8316 CODEC
(RockPro64 and Pinebook Pro).
__________________________________________________________________
bhyve - NVMe emulation improvements
Links
bhyve NVMe reviews
URL: https://reviews.freebsd.org/search/query/xvbcF20W__Km/
Contact: Chuck Tuffli <chuck at freebsd.org>
The University of New Hampshire InterOperability Laboratory (a.k.a. UNH
IOL) develops a suite of tests to determine if an NVMe device conforms
to the specification and is interoperable with other NVMe products.
This quarter, I undertook getting bhyve's emulated NVMe device to pass
the mandatory tests. Changes include:
* implement Flush command
* implement Format NVM command
* implement AER support
* implement Namespace Identification Descriptor
* fix Active Namespace list
* fix queue creation and deletion
* validate Deallocate range values
* handle zero length DSM ranges
* fix Get Log Page command
* implement SMART data I/O statistics
* validate the LBA start and count
* add basic Firmware Commit support
* add more compliant Get/Set Features
* add Feature, Interrupt Vector Config
* fix Get Features, Predictable Latency
This was also a good opportunity to restructure parts of the code to
make it more modular and easier to enhance. This includes
* convert logging statements to parameterized macros
* refactor I/O command handling
* add locks around queue accesses
* consolidate CQ update
* base pci_nvme_ioreq size on advertised MDTS
You can help by testing and/or commenting on the code reviews.
__________________________________________________________________
Bluetooth Support
Contact: Marc Veldman <marc at bumblingdork.com>
Bluetooth is a wireless technology for creating personal networks,
connecting peripherals like keyboards and mice but also speakers and
heart rate monitors.
FreeBSD has limited Bluetooth Basic Rate (BR) support and no functional
Bluetooth Low Energy (LE) support.
During this quarter many small improvements have been made to help the
development of Bluetooth LE support. A number of commands have been
added to hccontrol(8), mainly to do LE functions. It is now possible to
scan for LE devices within range using hccontrol. A panic that occurred
when enabling LE support has also been fixed.
Work is still needed to add Attribute Protocol (ATT) and Generic
Attribute Profile (GATT) support.
__________________________________________________________________
DRM Drivers Update
Links
drm-kmod
URL: https://github.com/freebsd/drm-kmod/
Contact: Emmanuel Vadot <manu at FreeBSD.Org>
The drm drivers for FreeBSD 13-CURRENT have been updated to match Linux
5.3 This brings us a little bit closer to the last LTS release of Linux
(5.4).
The current plan is to first update the driver to match 5.4 and then
look at making it work on FreeBSD-12-STABLE to have it ready for the
12.2 release.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
DTS Update
Contact: Emmanuel Vadot <manu at FreeBSD.org>
DTS files (Device Tree Sources) were updated to be on par with Linux
5.7 for HEAD and 5.6 for the 12-STABLE branch.
__________________________________________________________________
ENA FreeBSD Driver Update
Links
ENA README
URL:
https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README
Contact: Michal Krawczyk <mk at semihalf.com>
Contact: Artur Rojek <ar at semihalf.com>
Contact: Marcin Wojtas <mw at semihalf.com>
ENA (Elastic Network Adapter) is the smart NIC available in the
virtualized environment of Amazon Web Services (AWS). The ENA driver
supports multiple transmit and receive queues and can handle up to 100
Gb/s of network traffic, depending on the instance type on which it is
used.
Completed since the last update:
* Fixes for Rx refill to improve stability on low memory conditions
(also released as an errata notice for FreeBSD-12.1)
* Upstream of the v2.2.0 driver, introducing:
+ Add driver support for the upcoming HW features (reporting Tx
drops, disabling meta caching)
+ Add sysctl tuneables for IO queue number
+ Create IO queues with optional size backoff
+ Rework the way of configration of drbr and Rx ring size to be
more robust and stable
+ New HAL version
+ Driver is now marked as epoch ready
+ Default RSS key is created using RNG to improve security
+ Other minor fixes and improvements
* MFC of the ENA v2.2.0 driver to the FreeBSD 11.4
Sponsor: Amazon.com Inc
__________________________________________________________________
Forcible Unmount of UFS/FFS Filesystems on Disk Failure
Links
Phabricator Details
URL: https://reviews.freebsd.org/D24088
Contact: Chuck Silvers <chs at freebsd.org>
Contact: Kirk McKusick <mckusick at mckusick.com>
Commit -r361491 on May 25, 2020 enables a UFS file system to do a
forcible unmount when the underlying media fails or becomes
inaccessible. For example when a USB flash memory card hosting a UFS
file system is unplugged.
The rest of this report describes in more detail how forcible unmounts
are done. Suprisingly, less than 500 lines of file system code were
added or changed.
The strategy for handling disk I/O errors when soft updates are enabled
is to stop writing to the disk of the affected file system but continue
to accept I/O requests and report that all future writes by the file
system to that disk actually succeed. Then initiate an asynchronous
forced unmount of the affected file system.
There are two cases for disk I/O errors:
* ENXIO, which means that this disk is gone and the lower layers of
the storage stack already guarantee that no future I/O to this disk
will succeed.
* EIO (or most other errors), which means that this particular I/O
request has failed but subsequent I/O requests to this disk might
still succeed.
For ENXIO, we can just clear the error and continue, because we know
that the file system cannot affect the on-disk state after we see this
error. For EIO or other errors, we arrange for the geom_vfs layer to
reject all future I/O requests with ENXIO just like is done when the
geom_vfs is orphaned. In both cases, the file system code can just
clear the error and proceed with the forcible unmount.
This new treatment of I/O errors is needed for writes of any buffer
that is involved in a dependency. Most dependencies are described by a
structure attached to the buffer's b_dep field, but some are created
and processed as a result of the completion of the dependencies
attached to the buffer.
Clearing of some dependencies require a read. For example if there is a
dependency that requires an inode to be written, the disk block
containing that inode must be read, the updated inode copied into place
in that buffer, and the buffer then written back to disk.
Often the needed buffer is already in memory and can be used. But if it
needs to be read from the disk, the read will fail, so we fabricate a
buffer full of zeroes and pretend that the read succeeded. This zero'ed
buffer can be updated and "written" back to disk.
The only case where a buffer full of zeros causes the code to do the
wrong thing is when reading an inode buffer containing an inode that
still has an inode dependency in memory that will reinitialize the
effective link count (i_effnlink) based on the actual link count
(i_nlink) that we read. To handle this case we now store the i_nlink
value that we wrote in the inode dependency so that it can be restored
into the zero'ed buffer thus keeping the tracking of the inode link
count consistent.
Because applications depend on knowing when an attempt to write their
data to stable storage has failed, the fsync(2) and msync(2) system
calls need to return errors if data fails to be written to stable
storage. So these operations return ENXIO for every call made on files
in a file system where we have otherwise been ignoring I/O errors.
Sponsor: Netflix
__________________________________________________________________
i.MX 8M support
Links
D25274
URL: https://reviews.freebsd.org/D25274
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
i.MX 8M is the family of application processors from NXP based on Arm®
Cortex®-A53 and Cortex-M4 cores. The initial code drop for the platform
support includes CCM driver and clock implementation, GPC driver, clock
tree for i.MX 8M Quad. Most of the drivers from i.MX 6 can be reused
for i.MX 8M systems with relatively minor modifications. Common changes
include adding clock support and extending list of FDT compat strings.
With the linked patch FreeBSD successfully booted to multiuser with NFS
root on Nitrogen8M SBC.
__________________________________________________________________
Intel wireless and 11ac update
Links
Initial project announcement
URL:
https://lists.freebsd.org/pipermail/freebsd-wireless/2020-April/009055.html
The freebsd-wireless mailing list
URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless
Contact: Bjoern A. Zeeb <bz at FreeBSD.org>
The Intel Wireless cards are one of the most commonly used and ask for
in FreeBSD notebooks.
This project has three main goals:
* newer Intel Wireless device support,
* newer WiFi standards support for Intel Wireless,
* integration of 802.11ac client support and infrastructure in
FreeBSD.
The first one is needed as iwm(4) currently does not support the latest
generations of Intel Wireless cards at all. The second is needed as in
FreeBSD iwm(4) does not even support 802.11n. The third one we want to
catch up and use the improvements the new Wifi standard offers, e.g.,
speed.
One of the decisions made was: rather than improving iwm(4) this work
uses the dual-licensed native Linux driver under BSD license and the
linuxkpi framework to stay as close to upstream as possible as a first
step. This will give us several advantages, such as, the full support
for all cards, quick support for new chipsets, vendor bug fixes, but
also the ability to contribute back.
At this point the lower level hardware attachments and the firmware
loading and initialisation works. I plan to release a patchset for
testing before mid-July, you can see if your currently supported or
unsupported hardware will be detected. This first cut will not support
any wireless operation yet, which will follow later in the year.
If you want to help testing, please watch the freebsd-wireless list.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
amd64 5-Level Paging Structures support
Links
Patch
URL: https://reviews.freebsd.org/D25273
Intel SDM
URL:
https://software.intel.com/content/www/us/en/develop/articles/intel-sdm.html
Intel whitepaper
URL:
https://software.intel.com/content/www/us/en/develop/download/5-level-paging-and-5-level-ept-white-paper.html
Contact: Konstantin Belousov <kib at FreeBSD.org>
Since its introduction, x86 Long Mode (AKA 64bit execution mode, amd64
in FreeBSD terminology) uses 4-level paging structures, which provides
48 bits of virtual address space (LA48). FreeBSD evenly divides the
space between userspace and kernel, giving both 47 virtual address
bits.
In near future Intel CPUs will start providing 5-level paging
structures, i.e. giving 57 bits for virtual addresses (LA57). This
means, with preservation of the existing divide between KVA and UVA, 56
bit for UVA, or 2^9 = 512 times more virtual memory.
The amd64 pmap was modified to support both LA48 and LA57, defaulting
to LA57 if hardware supports it. The tunable is provided to force using
LA48 even if hardware can do LA57.
The most interesting part of the patch is the switch from boot paging
mode to LA57. Loaders, either legacy or UEFI, pass control to the
kernel in Long Mode, which implies that the paging is turned on. This
neccessarily means that it is LA48 mode. SDM states that paging mode
cannot be switched while Long Mode is active, so kernel has to create
new page table structures, turn Long Mode off, then load new %cr3 and
finally re-enable Long Mode.
I decided to only provide the larger virtual address space to usermode
for the initial step, leaving KVA layout intact. The main motivation is
that changing KVA arrangements requires changing the auto-tuning
settings, which deserve separate work. Another argument for it is that
most of the kernel memory is non-swappable, so cannot be over-commited.
We have 2:1 ratio of useful KVA to physical memory (due to direct map),
and until machines get more physical address lines, increasing KVA is
not useful.
After this was decided, creating a 5-level paging structure for kernel
pmap from existing 4-level one is quite straightforward; we need to add
one page for top level, create one PML5 entry to point to existing PML4
page, and create the famous self-referential entry for
vtopte()/vtopde().
Care was taken to provide binary compatible layout of UVA for binaries
that cannot be executed correctly with larger address space. For
instance, programs could have knowledge about used bits in the
addresses and used upper bits for other data, or implemented compressed
pointers. Even if system runs in LA57 mode, specific binary can request
LA48-compatible UVA by procctl(2) or by the flag in the FreeBSD
features ELF note.
Since I do not have access to a machine with LA57, development was done
using QEMU. It would be interesting to try it on the real hardware.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Not-transparent superpages
Links
Patch
URL: https://reviews.freebsd.org/D24652
Contact: Konstantin Belousov <kib at FreeBSD.org>
FreeBSD already provides excellent support for superpages, in a manner
completely transparent to applications. It tries to proactively prevent
fragmentation, reserves contigous runs of the physical pages for linear
allocations in managed objects, and auto-promote runs of small pages
when they form complete superpage.
The shortcomings of this approach directly follows from its strength:
some applications want to get guaranteed superpage mappings, typically
because the underlying physical memory is also offloaded into a
hardware which also has memory mapping unit. For instance, Infiniband
RMDA adapters do memory registration and remapping, which is more
efficient with large pages. In such cases transparent (non-guaranteed)
support cannot be used.
The extension was developed for POSIX shared memory subsystem to allow
the creator request that the shared memory object was backed by
physically contiguous pages, with runs of specified size. The mmap(2)
syscall is aware of such objects, and if the requested mapping is
properly aligned, it will be served by superpages.
The new type of the shared memory objects are backed by modified a
physical pager, which only allocates contigous physical memory. The VM
ensures that mappings of the objects are never split (clipped) on a
non-superpage boundary. The fault handler is specially optimized to be
very fast by quickly installing the superpage PTE, and to avoid
touching all small pages constituing it.
Currently the required pmap support is provided for amd64 with 2M and
1G superpage sizes.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
NXP ARM64 SoC support
Contact: Marcin Wojtas <mw at semihalf.com>
Contact: Artur Rojek <ar at semihalf.com>
Contact: Dawid Gorecki <dgr at semihalf.com>
The Semihalf team initiated working on FreeBSD support for the NXP
LS1046A SoC
LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with
integrated packet processing acceleration and high speed peripherals
including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide
range of networking, storage, security and industrial applications.
Completed since the last update:
* Improve code in a couple of review cycles and merge following new
features to the FreeBSD-HEAD (r361458 - r361464):
+ QorIQ platform clockgen driver
+ LS1046A clockgen driver
+ GPIO support for QorIQ boards
+ QorIQ LS10xx AHCI driver
+ VF610 I2C controller support
+ TCA6416 GPIO expander
+ Epson RX-8803 RTC
Todo:
* Upstreaming of the QorIQ SDHCI driver - it is expected to be
submitted/merged to HEAD in the Q3 of 2020.
Sponsor: Alstom Group
__________________________________________________________________
amd64 pmap Fine-grained pv lists locking
Links
Patch
URL: https://reviews.freebsd.org/D24217
Contact: Konstantin Belousov <kib at FreeBSD.org>
FreeBSD kernel Virtual Memory subsystem handles 'normal' application
memory, i.e. anonymous or file-backed shared and private mappings, with
so called managed pages. Managed page is fully controlled by VM, which
tracks it status. In particular, managed page can be made read-only for
write-back to the file, or unmapped for reuse (paging).
The machine-dependent VM layer, pmap, must support managed pages, for
instance it must provide operations such as pmap_remove_write() to
downgrade all mappings to read-only, or pmap_remove_all() to unmap the
page from all address spaces. To implement this kind of operations,
while not causing the overhead of scanning all page tables, pmap must
track existing mappings of the page. The tracking is done by allocating
a small data structure 'pv entry' per mapping, and linking all pv
entries for the given page into pv list.
Since pv entries come from context of different address spaces, pmap
must provide synchronization to guarantee correctness of the list
structures. Current pmap allocates one mutex per one 2M physical
superpage in NUMA configurations, and MAXCPU == 256 locks hashed by the
page physical address for non-NUMA. The end result is often undeserved
lock aliasing causing pv list locks contention, since all 4k pages in
the 2M superpage share the same lock, and reservations typically cause
adjasted pages to come from the same superpage.
The proposed patch creates a new kernel synchronization primitive
called one byte mutex, which is embedded into the currently unused
padding in machine-dependent portion of the struct vm_page. This way
each page gets dedicated pv list lock without using any more memory. In
the ever-important buildkernel benchmark on non-NUMA config, this
change provides 2x reduction of the system time.
One complication is that old locking distribution scheme made a natural
fit for superpages promotion and demotion, since all embedded small
pages shared the same pv list lock, and the operations basically
fold/unfold corresponding pv entries. Now the promotion and demotion
operations require taking all locks for constituent small pages, which
provides small but measurable impact on them. It is possible to
optimize it further by providing the 'superlock' on the first page from
the superpage run, but the affected operations are relatively rare so
that it is not even obvious that implementing the optimization would
not slow down other pathes.
Another important nuance of the pv entries handling is that sometimes
pv entries allocator must not fail. Typically this is required when
kernel makes a call to pmap_enter() which must establish new mapping,
and for managed page this includes allocating the pv entry if existing
cannot be reused. If allocator cannot get a fresh page from the
vm_page_alloc(9), it opts to destroy some other managed mapping to
either get a reusable pv entry from current pmap, or destroy enough
managed mappings from some other pmap to free whole page.
To do the reclamation, currently all pages from which with pv entries
are allocated, are linked in the global pv chunk list, which is
protected by global (per-NUMA domain) mutex. Any allocation or free of
pv entry has to lock the mutex, which is apparently a contention point
for large machines.
Patch removes the global list of chunks, instead linking all pmaps in
the global list like it is done on i386 (but for different reason). Now
the global lock is only taken for pmap creation and free, which
corresponds to fork/exec and exit of a process, and when pv allocator
starts reclaiming from other pmaps (which is normally does not happen).
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Lockless routing lookups and scalable multipath
Links
Implementation of scalable multipath
URL: https://reviews.freebsd.org/D24141#change-ZOjdMqgDgUr7
Contact: Alexander Chernikov <melifaro at FreeBSD.org>
The primary goal of this work is to bring scalable routing multipath
implementation, enabled by default. Another goal is enabling
high-performance routing lookups.
Multipath will close long-standing feature gap that modern networking
OS must have. Lockless routing lookups will remove lookup bottlenecks,
improve both dataplane and control plane performance for the setups
with large number of routes.
Background
The initial routing kpi was introduced back in 1980. It was a nice
generic approach back then, as no one knew how the protocols would
evolve. It has been enormously successful as it was able to survive for
20+ years.
Unfortunately, this kpi does not try to protect subsystem internals
from the outside users, resulting in tight coupling with other
subsystems. As a result, making changes is hard, leading to compromises
and piling technical debt.
Implementation overview
Most changes are based on introduction of the concept of nexthops.
Nexthops are separate datastructures, containing all necessary
information to perform packet forwarding such as gateway, interface and
mtu. They are shared among the routes, providing more pre-computed
cache-efficient data while requiring less memory. Interested reader can
find more detailed description in D24141. Another overview can be found
in Nexthop object talk describing Linux implementation.
Multipath implementation extends nexthop concept further by introducing
nexthop groups.
Each route has a pointer to either nexthops or a nexthop group,
decoupling lookup algorithm from the routing stack internals. Both
nexthops and nexthop groups are immutable and use epoch(9)-backed
reclamation.
A pre-requisite for lockless routing lookup is the introduction of
modular lookup framework, allowing to attach any longes-prefix-match
algorithm implementation to any IPv4/IPv6 fib.
Currently there are plans to use modified DIR-24-8 algorithm from DPDK
for both IPv4 and IPv6 families as an example of base lockless
implementation.
Status
* Nexthop objects [ DONE ]
* Introduction of nexthop objects [ DONE ]
* Conversion of old KPI users to the new one [ DONE ]
* Conversion of route caching to nexthop caching [ DONE ]
* Conversion of struct rtentry field access to nhop field access [
DONE ]
* Eliminating old lookup KPI and hiding struct rtentry [ DONE ]
Multipath [ IN PROGRESS ]
* Switch control plane consumers to use (rtentry, nhop) pairs instead
of rtentry to allow multipath changes happen transparently [ 90%
DONE ]
* Introduce nexthop group objects
* Add mutipath support for the rib (routing information base)
manipulation functions
* Add flowid generation for outbound traffic to enable load balancing
Modular longest-prefix-match lookup algorithm [ IN PROGRESS ]
* Design control plane framework for attaching algorithms [ 90% DONE
]
* Port IPv6 lockless lookup algorithm [ DONE ]
* Port IPv4 lockless lookup algorithm
__________________________________________________________________
ZSTD Compression in ZFS
Contact: Allan Jude <allanjude at freebsd.org>
Zstandard (ZSTD) is a modern high-performance compression algorithm
designed to provide the compression ratios of gzip while offering much
better performance. ZSTD has been adopted in FreeBSD for a number of
other uses, including compressing kernel crash dumps, as a replacement
for gzip or bzip for compressing log files, and for future versions of
pkg(8).
This effort to complete the integration of ZSTD into ZFS is sponsored
by the FreeBSD Foundation.
Integrating ZSTD into ZFS will further extend the transparent
compression feature of ZFS by offering higher compression ratios
without the performance penalty associated with gzip. ZSTD offers
compression levels ranging from 1 (low compression) to 22 (maximum
compression), plus ZSTD-Fast levels that offer less compression but
even greater speed. This will allow the storage administrator to select
the performance-vs-compression tradeoff that best suits their needs.
Tasks remaining to be completed:
* Extend ZFS to support compression algorithms with large numbers of
levels
* Solve issues around the inheritence of compression settings
* Restore compression level when reading blocks from disk
* Create a future-proofing scheme to handle changing versions of ZSTD
* Extend ZFS replication to handle backwards compatibility with pools
that do not yet support ZSTD
* Resolve issues around backwards compatibility when ZSTD is
configured but not used
Sponsor: The FreeBSD Foundation
__________________________________________________________________
CheriBSD 2020 Q2
Links
CHERI-CPU
URL: http://www.cheri-cpu.org
DARPA FETT Bug Bounty Program
URL: https://fett.darpa.mil
Contact: Alex Richardson <arichardson at FreeBSD.org>
Contact: Andrew Turner <andrew at FreeBSD.org>
Contact: Brooks Davis <brooks at FreeBSD.org>
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
Contact: Jessica Clarke <jrtc27 at FreeBSD.org>
Contact: John Baldwin <jhb at FreeBSD.org>
Contact: Robert Watson <rwatson at FreeBSD.org>
Contact: Ruslan Bukin <br at FreeBSD.org>
CheriBSD extends FreeBSD to implement memory protection and software
compartmentalization features supported by the CHERI instruction set
extensions.
Support for CHERI-RISC-V in CheriBSD has continued to mature this
quarter in tandem with refinements to the CHERI-RISC-V architecture. We
have recently made CheriBSD's "pure capability" (CheriABI) process
environment the default ABI rather than a compatibility layer. It has
grown support for:
* dynamically linked binaries (previously only statically-linked
binaries were supported)
* C++ including exceptions
* sealed return address and function pointer capabilities
("sentries") which provide additional CFI protection
* initial MMU protections for loading and storing tags
At this point, CHERI-RISC-V support in CheriBSD is generally on par
with support for CHERI-MIPS.
Much of this effort has been focused on preparing CheriBSD on
CHERI-RISC-V for inclusion as a demonstrator system in DARPA's Finding
Exploits to Thwart Tampering (FETT Bug Bounty program).
In addition, work has begun this quarter on porting CheriBSD to Arm's
Morello SoC. Morello is a prototype demonstrator board which adds CHERI
extensions to ARMv8-A.
We've recently switched to a dev-branch model where active work takes
place on the dev branch and we periodically merge to master with
synchronization between CheriBSD and dependencies like CHERI-LLVM.
For those interested in what it's like to program for CHERI, we've
recently released a CHERI C/C++ Programming Guide.
__________________________________________________________________
Architectures
Updating platform-specific features and bringing in support for new
hardware platforms.
Continuous Integration on !x86
Contact: Edward Tomasz Napierala <trasz at FreeBSD.org>
For quite a while the FreeBSD CI infrastructure has been running
FreeBSD builds and regression tests, making it easy to spot
regressions. While CI was building images for all architectures, the
regression tests were only run on amd64 and i386, which means they
couldn't detect architecture-specific runtime breakage on non-x86
architectures. This poses a problem not only for FreeBSD itself, but
also for people working on FreeBSD forks for !x86, such as the CHERI
project at University of Cambridge and SRI International.
The goal of this project is to run regression tests on the remaining
architectures supported by FreeBSD: ARM, ARM64, MIPS, POWER, and
RISC-V. The tests are being run using common, mostly
machine-independent scripts. Those required some changes to make it
possible to use QEMU in addition to Bhyve, the hypervisor used for x86
tests. The sysutils/u-boot-qemu-arm and sysutils/u-boot-qemu-arm64
ports were added to the Ports Collection. Finally, each of the
architectures required some tweaks, to account for different
configuration requirements - for example the MIPS kernel doesn't
support VirtIO disks, or even AHCI, whereas on ARM64 the U-Boot gets
confused with more than one VirtIO disk.
On ARM, we're currently at 52 failures and 590 skipped tests, of 5925
tests ran
(https://ci.freebsd.org/job/FreeBSD-head-armv7-test/lastCompletedBuild/testReport/).
On ARM64 it's 19 failures and 160 skipped
(https://ci.freebsd.org/job/FreeBSD-head-aarch64-test/lastCompletedBuild/testReport/).
On MIPS it's 172 failures and 734 skipped
(https://ci.freebsd.org/job/FreeBSD-head-mips64-test/lastCompletedBuild/testReport/).
For POWER, and RISC-V the results are not available yet.
Remaining work:
* Failing regression tests need to be fixed.
* The tests are quite slow on QEMU, for example the ARM64 run takes
about five hours. Running them automatically after each commit
would quickly overload the CI cluster. A solution would be to e.g.
run them daily.
* Some of the jobs still fail to produce results: powerpc64 deadlocks
at the end of regression test suite due to an unkillable process,
riscv64 panics randomly, and on mips64 kyua(1) often crashes on
jemalloc assertion. Those might be fixed by an upcoming QEMU port
update.
Sponsor: DARPA
__________________________________________________________________
FreeBSD/RISC-V Project
Links
Wiki
URL: https://wiki.freebsd.org/riscv
Contact: Mitchell Horne <mhorne at FreeBSD.org>
Contact: freebsd-riscv Mailing List
Contact: IRC #freebsd-riscv channel on freenode
The FreeBSD/RISC-V project is providing support for running FreeBSD on
the RISC-V Instruction Set Architecture. Since RISC-V is still a young
and evolving platform, one of our goals is to have FreeBSD be a
well-supported option for users as RISC-V hardware increases in
availability.
This quarter saw a number of improvements to the boot process.
The physmem interface used by arm and arm64 to enumerate physical
memory resources was moved to machine-independent code and adopted on
RISC-V. As a result, the kernel is now able to detect and exclude
physical memory reserved by devices or firmware. A bug that prevented
the kernel from using physical memory below its load address was fixed.
This typically did not manifest in much waste, as the kernel is loaded
2MB after the start of physical memory by default. In future boot
configurations, the impact would have been much larger.
Our port for OpenSBI was updated to v0.8, bringing several new features
and fixes. In particular, it brought the Hardware State Management
(HSM) extension, which can be used to start and stop CPUs. FreeBSD will
now use this extension whenever it detects that it is available.
There has also been a lot of work done to port FreeBSD's standard
bootloader, loader(8), to RISC-V. This has big advantages in terms of
boot flexibility, and brings us closer to what's needed to produce
official FreeBSD/RISC-V release images. By leveraging UEFI support from
u-boot, loader.efi can be used in a manner similar to arm and arm64.
Next quarter will likely bring u-boot ports for RISC-V targets, pending
the v2020.07 release. Booting the kernel directly via SBI firmware will
continue to be supported.
__________________________________________________________________
Userland Programs
Changes affecting the base system and programs in it.
Import of new implementation of bc and dc
Links
Official repository
URL: https://git.yzena.com/gavin/bc
Repository mirror on GitHub
URL: https://github.com/gavinhoward/bc/
Contact: Stefan Esser <se at FreeBSD.org>
Contact: Gavin D. Howard <yzena.tech at gmail.com>
A new version of bc and dc has been imported into FreeBSD-CURRENT and
enabled by default. An import into 12-STABLE is planned before the end
of July, but it will not be enabled by default (will require
"WITH_GH_BC=yes" to be set in /etc/src.conf).
This version has been developed by Gavin D. Howard with the goal to
provide a highly portable and POSIX compatible implementation. It
offers GNU bc compatibility and should be a drop-in replacement for the
bc in FreeBSD, except for standard-violating behavior of the bc
currently in FreeBSD (e.g., the modulo operator).
Additional features:
* High performance (up to more than a factor of 100 faster than the
current FreeBSD implementation in some tests)
* support of message catalogs with a large number of languages
supported in the current release (contributions of further
translations are welcome).
* Extra built-in functions and operators.
* Extended library of advanced math functions
* Detailed man-page explaining standard conformant and extended
features
* One shared binary for bc and dc (bc is not just a pre-processor
that relies on dc for the actual computations)
The only dc feature not supported by the dc is the execution of
sub-processes, since the author considers it a security hazard for a
calculator to have.
This code is also available as a port and package (math/gh-bc or
gh-bc), e.g. for use with a FreeBSD binary release.
__________________________________________________________________
Binutils Retirement
Links
GPL in Base wiki page
URL: https://wiki.freebsd.org/GPLinBase
Contact: Ed Maste <emaste at FreeBSD.org>
We have been working on migrating to a modern and copyfree or
permissively licensed toolchain for quite some time. In the last
quarter we retired two obsolete GNU bintuils: objdump, and as.
Many uses of objdump can be replaced with readelf, and llvm-objdump is
also available in the base system. Ports that depend on objdump have
been updated to rely on the GNU binutils port or package.
The GNU as utility was used by both the base system and by ports. As
with objdump, ports that require GNU as have generally been updated to
depend on binutils. One file in the base system (skein_block_asm.s)
proved troublesome during earlier attempts to migrate to using Clang's
integrated assembler (IAS). However, after the update to Clang 10 (and
with some trivial modifications to the source) IAS can assemble the
file.
Both GNU as and objdump have been removed from FreeBSD-CURRENT and will
be absent from FreeBSD 13.0.
TODO
The final task in the binutils retirement project is to remove GNU GDB
6.1. It is currently retained for crashinfo(8).
Sponsor: The FreeBSD Foundation
__________________________________________________________________
Run-Time Dynamic Linker improvements
Contact: Konstantin Belousov <kib at FreeBSD.org>
Rtld gets some number of small bug fixes and improvements.
RTLD_DEEPBIND dlopen(3) flag was implemented, despite being a strange
and even unsafe idea, for compatibility with glibc.
Several improvements to the direct execution mode were made. Most
interesting are perhaps the '-v' switch to report some configuration
parameters for rtld, the ability to specify argv0 different from the
executed binary name, and fixes to properly set osrel/ABI for the
directly executed binary.
The link_map structure that is used by tools that need to know the list
of loaded shared objects (like gdb and wine) was made more compatible
with glibc, while keeping existing FreeBSD ABI intact.
In the course of the link_map work, it become apparent that rtld
sometimes needs to report presence of features that cannot be deduced
by just a runtime test for symbol presence or for function behavior.
For that, a scheme of reporting features with uniformingly named
symbols was designed - see the rtld(1) man-page (in CURRENT) for an
explanation.
Position-independent (PIE) binaries on FreeBSD are now marked with the
DF_1_PIE DT_FLAG1 flag. Otherwise, such binaries are just ET_DYN
objects and it is quite hard to distinguish proper dynamically shared
object (DSO) from PIE binary. The problem is that for binaries, the
static linker strips some information which is required for proper
loading as a DSO, and additonally, binaries contains relocations like
copy-relocations that cannot be handled for non-main binaries at all.
With the flag addition, rtld properly detects binaries and refuses to
load them with dlopen() or as DT_NEEDED dependency. ldd(1) also
misdetected PIE vs. DSO, and required a fix to parse dynamic segments
to not try to dlopen() them.
Sponsor: The FreeBSD Foundation
__________________________________________________________________
VHDX support in mkimg(1)
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
VHDX is the successor of Microsoft's VHD virtual drive file format. It
increases maximum capacity of the virtual drive to 64TB and introduces
features to better handle power/system failures.
VHDX is the required format for 2nd generation Hyper-V VMs.
__________________________________________________________________
Ports
Changes affecting the Ports Collection, whether sweeping changes that
touch most of the tree, or individual ports themselves.
Bastille
Links
Bastille GitHub
URL: https://github.com/bastillebsd/bastille
Bastille Templates
URL: https://gitlab.com/bastillebsd-templates
Bastille Website
URL: https://bastillebsd.org
Contact: Christer Edwards <christer.edwards at gmail.com>
Bastille is an open-source system for automating deployment and
management of containerized applications on FreeBSD.
Bastille Templates automate container setup allowing you to easily
reproduce containers as needed.
Bastille is available in ports at sysutils/bastille.
Q2 2020 Status
In Q2 2020 Bastille merged some exciting new features into GitHub.
Changes include:
* experimental support for new Bastillefile template syntax
* added mount and umount sub-commands
* added a default Vagrantfile for simple testing
* experimental support for empty containers
* improvements to VNET DHCP support
* cosmetic bugfixes in error output
* extended config file documentation
* updated bastille help output
* option to (-f) force destroy container
sysutils/bastille was updated to 0.6.20200414 (latest).
New Bastille templates added this quarter include:
* Percona database server
* Asterisk SIP server
* dnsmasq DNS/DHCP server (VNET required)
* nginx pkg server for poudriere
Everything mentioned here was done under COVID-19 quarantine. Special
thanks to everyone that contributed during this time.
__________________________________________________________________
KDE on FreeBSD
Links
KDE FreeBSD
URL: https://freebsd.kde.org/
KDE Community FreeBSD
URL: https://community.kde.org/FreeBSD
Contact: Adriaan de Groot <kde at FreeBSD.org>
The KDE on FreeBSD project packages the software produced by the KDE
Community for FreeBSD. The software includes a full desktop environment
KDE Plasma, IDE KDevelop, a PIM suite Kontact and hundreds of other
applications that can be used on any FreeBSD desktop machine.
This quarter has been an ever-so-peculiar one. While we are used to
working remotely, collaborating over the internet to update the ports
tree, it's qualitatively different when the whole world locks down.
Meanwhile, software continues to be released, so this quarter the kde@
team:
* Restored a patch that closes down a remote TCP held by X11
applications that use the ICE library. Thanks to Colin Percival for
reporting it. It went missing in one of the port updates. To
prevent this in the future, the patch has been upstreamed. PR
229772.
* Chased KDE-adjacent software like CMake, Cutelyst, Latte-dock and
Nheko through new releases. In particular CMake takes a lot of
effort every time because it is a build-time dependency of over
2000 ports.
* graphics/poppler was updated to the latest upstream release. This
is a low-level dependency for many document-viewing applications,
and like CMake requires chasing a lot of other software. Poppler is
one of the components shared between various software stacks (and
"desktop environments") under the desktop@ group, in which kde@
participates.
* KDE Frameworks release like clockwork, reaching KDE Frameworks 5.70
mid-may.
* KDE Applications -- the KDE release service, really, which delivers
libraries, applications, and add-ons -- had one large release, with
20.04.1 landing in the ports tree also mid-may and its monthly
update 20.04.2 in mid-june.
* Some new Wayland support for KDE Plasma -- we have not tested this
on FreeBSD -- has appeared and has been duly packaged.
* A great deal of preparation has gone into Qt 5.15. Many ports have
been pre-emptively patched for this new -- and last -- LTS release
of Qt 5. The update itself has not yet landed, pending a few last
bits of fallout.
The kde@ team would like to thank Antoine for many exp-runs, mikael@
for useful tips, swills@ for patience and kai@ for dealing with
WebEngine.
The next big round of updates for the KDE stack is slated: CMake 3.18,
Qt 5.15 LTS, and KDE Frameworks 5.71.
__________________________________________________________________
Haskell on FreeBSD
Links
Haskell language homepage
URL: http://www.haskell.org/
Ports development repo
URL: https://github.com/freebsd/freebsd-ports-haskell
Contact: Gleb Popov <haskell at FreeBSD.org>
Haskell is a general-purpose strictly-typed pure functional language.
The Haskell on FreeBSD projects strives to provide the up-to-date
Haskell toolchain as well as various application written in this
language.
This quarter brought the long-awaited GHC update, which is now at
version 8.8.3. Along the compiler, the Haskell build system frontend,
cabal-install, was also upgraded to 3.0.2.0. During this update,
numerous Haskell ports were updated too.
All existing ports of Haskell applications were migrated to USES=cabal,
which implements Go-style build proccess - all dependencies are
compiled as part of the build. As a consequence, ports for Haskell
libraries have been deprecated and removed.
Upgrading GHC became a tedious task for a single person, so a new
GitHub repository was created under the FreeBSD organization -
freebsd-ports-haskell. Right now, work is being done on preparing
another GHC upgrade in the ghc-upgrade-810 branch. Any contributions
are welcome.
__________________________________________________________________
rtsx - Porting driver for Realtek SD card reader from OpenBSD
Links
rtsx
URL: https://github.com/hlh-restart/rtsx
PR204521
URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204521
Contact: Henri Hennebert <hlh at restart.be>
The rtsx driver for Realtek SD card reader has been ported from
OpenBSD. Its development snapshot is available via the
sysutils/rtsx-kmod port.
From March to May 2020, the code has been completed with the help of
Gary Jennejohn (gj@) and Jesper Schmitz Mouridsen (jsm@). Some tweaks
have been imported from the Linux counterpart.
The driver has been successfully tested with:
* RTS5209 under head (Lenovo ThinkPad L520)
* RTS5227 under stable/11 and releng/12.1 (HP ProBook 430 g2, Lenovo
ThinkPad T450/T450s)
* RTS5229 under releng/12.1 (Lenovo IdeaPad 120S-14IAP)
* RTS522A under releng/12.1 and head (Intel NUC8i5BE, Lenovo ThinkPad
P50s)
* RTS525A under releng/12.1 (Dell Latitude E5570)
* RTL8411B under stable/12 (Acer Aspire E 15 E5-576-77W6)
The driver should also work for Realtek RTS5249, RTL8402 and RTL8411.
More tests are welcome, especially for the devices not yet tested.
These devices may require more tweaks.
PR204521 contains the bulk of exchanges for completion of the code.
__________________________________________________________________
Valgrind updates
Links
Patch for valgrind
URL: https://reviews.freebsd.org/D25452
Contact: Paul Floyd <paulf at free.fr>
Contact: Kyle Evans <kevans at FreeBSD.org>
A large amount of work has been done to rebase FreeBSD support on top
of Valgrind 3.17.0, and to address numerous test suite failures.
Currently, almost all of the regression tests pass on amd64. This is a
major improvement over the current state of affairs, in which the
Valgrind is quite out of date and is missing important functionality.
Some follow-up work aims to make FreeBSD an officially supported target
platform for Valgrind.
The devel/valgrind-devel port is in the process of being updated to
point at the new work.
__________________________________________________________________
Documentation
Noteworthy changes in the documentation tree, in manpages, or in
external books/documents.
FreeBSD Translations on Weblate
Links
Translate FreeBSD on Weblate wiki
URL: https://wiki.freebsd.org/DocTranslationOnWeblate
FreeBSD Weblate Instance
URL: https://translate-dev.freebsd.org/
Contact: Danilo G. Baio <dbaio at FreeBSD.org>
Contact: Edson Brandi <ebrandi at FreeBSD.org>
This quarter was improved the renderization of RTL (Right-to-left)
languages on the FreeBSD Documentation. We faced this issue after the
first RTL language joined the translations effort (fa_IR).
We are looking forward to receive more languages and translators to the
project as well.
Q2 2020 Status
* 10 languages (No new languages)
* 80 registered users (33 new users since last quarter)
Languages
* Chinese (Simplified) (zh_CN)
* Chinese (Traditional) (zh_TW)
* French (fr_FR)
* German (de_DE)
* Italian (it_IT)
* Norwegian (nb_NO)
* Persian (fa_IR)
* Portuguese (pt_BR)
* Spanish (es_ES)
* Turkish (tr-TR)
We want to thank everyone that contributed, translating or reviewing
documents. And please, help promote this effort on your local user
group, we always need more volunteers.
__________________________________________________________________
Miscellaneous
Objects that defy categorization.
FreshPorts
Links
FreshPorts
URL: http://freshports.org/
FreshPorts blog
URL: http://news.freshports.org/
Contact: Dan Langille <dan at langille.org>
FreshPorts, and its sister site, FreshSource, have reported upon
FreeBSD commits for 20 years. They cover all commits, not just ports.
FreshPorts tracks the commits and extracts data from the port Makefiles
to create a database of information useful to both port developers and
port users.
For example, https://www.freshports.org/security/acme.sh/ shows the
history of this port, back to its creation in May 2017.
git
Work on git started back in September. It was ignored for a while and
started back in mid-June with the creation of new git-specific jails
for commit ingress (commit processing gitdev) and for the website.
Serhii (Sergey) Kozlov created a script to transform GIT commit entries
into XML digestible by FreshPorts. This was a huge step foward for the
effort.
The next step include:
* incorporate a that script the automated processes of FreshPorts
* migrate to new test & stage versions of FreshPorts
* test
* get ready for prod
Help wanted
git is not far away now. I could use helpers to
* review code
* watch the commits on the devgit websites
* catch missing stuff
Thank you
Packages
FreshPorts now displays the packages version available from the repo
sources. This covers all primary tiers (e.g. FreeBSD:12:amd64) and all
secondary tiers (e.g. FreeBSD:13:powerpc64). This helps uses know what
versions they can expect and when then repo was last built.
Dependency lines
Some things are easiest done via copy/paste. If you are working on a
port Makefile and need to add a new dependency, FreshPorts shows the
dependency line for that port. For example:
acme.sh>0:security/acme.sh
Libraries are also covered by this feature.
Python ports were recently adjusted to display
${PYTHON_PKGNAMEPREFIX}virtualenv>0:devel/py-virtualenv
instead of
py37-virtualenv>0:devel/py-virtualenv
You can read more about this change in [issue
#73](https://github.com/FreshPorts/freshports/issues/73).
Watch ports I maintain
The search page has long had the "Ports I Maintain" button (if you are
logged in). This feature recently branched out to a new automated watch
list option: Watch ports I maintain.
This report subscription will notify you of any commits to the ports
you maintain. Your email address on FreshPorts must match the value in
the MAINTAINER field of the port. This is always a daily report.
From time to time, an infrastructure change will occur which touches
your port. This feature ensures you know about that change.
Repology links
Repology links were requested. This allows you to see what versions of
that port are in the repositories of other systems. A link to
repology.org appears on every port page.
Further reading
Based upon things you didn't know FreshPorts can do
There are many things FreshPorts can do, including search Makefile's
and pkg-plist. This is from a recent blog post:
* provides example dependency line. e.g.
p5-XML-RSS>0:textproc/p5-XML-RSS
* list of dependencies for a port
* list of ports depending upon this port
* see default configuration options
* what packages install a given file (e.g. bin/unzip)
* find out what ports a person maintains
* find Makefiles which contain references to bunzip
* search results can be plain-text consisting of a list of foo/bar
ports
* the Maximum Effort checkbox on the search page does nothing
* committers can be notified of sanity test failures after the commit
* find a commit, any commit, based on SVN revision number, e.g. :
https://www.freshports.org/commit.php?revision=352332
Javascript help wanted
We recently upgraded some outdated Javascript modules. This broke our
[JavaScript based graphs](https://www.freshports.org/graphs2.php). We
could use some help on fixing that please. The starting points are
listed on that URL. If you need a working website to play with, please
contact me with a ssh public-key.
Sponsor: hardware provided by iXsystems
__________________________________________________________________
PCI passthrough with bhyve on Intel and for OpenBSD guests
Links
bhyve Intel bug report
URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229852
bhyve OpenBSD bug report
URL: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=245392
PCI passthrough with bhyve (FreeBSD wiki article)
URL: https://wiki.freebsd.org/bhyve/pci_passthru
Contact: Anatoli <me at anatoli.ws>
Contact: Callum <callum at aitchison.org>
Contact: Peter Grehan <grehan at freebsd.org>
bhyve(8) is a hypervisor that supports running a variety of guest
operating systems in virtual machines. bhyve(8) includes support for
PCI devices passthru, a technique to pass host PCI devices to a virtual
machine for its exclusive control and use.
For some years, PCI passthrough (ppt) in bhyve was not working on some
Intel systems and for OpenBSD guests due to two bugs. The first one was
crashing FreeBSD host when bhyve was started with ppt on Intel
processors with two VT-d translation units (IOMMU), included in most
Skylake and newer Intel processors.
The second bug was preventing correct interrupts handling for OpenBSD
guests. As a result, OpenBSD guests running on bhyve were not able to
use any PCI devices passed through to them from the host.
During the last 2 months the second bug was identified and fixed and
they both were backported to 12.1-RELEASE (p7). So now it's possible to
fully take advantage of PCI passthrough (ppt) with bhyve in a
production-ready RELEASE version.
The most typical case for ppt is to pass to the guest network adapters
for its complete control, but you can also pass through USB devices
(including external HDDs). Note though, passthrough of VGA and GPU
devices is not supported yet (for more details see the 3rd link).
A particularly interesting case for ppt is to use OpenBSD guest as a
firewall and a router for a FreeBSD server.
With ppt you can achieve this all inside a single server. You could
pass to the OpenBSD guest a network adapter connected to the internet
and it would take a complete control of it. After filtering the
traffic, it could pass good packets via virtual network interfaces to
other guests or to the host.
Once a network adapter is passed through, a FreeBSD host not only
doesn't see it and hence doesn't handle the network traffic, it doesn't
even have to initialize the adapter (e.g. in case of a WiFi card, it's
the guest that loads the firmware).
In simple terms the host only passes the device interrupts to the guest
as they come from the hardware. Everything related to the device
management happens inside the guest so there's no danger that some
network traffic exploits some issue in the host's network stack and
causes the host to crash or misbehave in other ways.
__________________________________________________________________
SageMath
Links
SageMath site
URL: https://www.sagemath.org/
Contact: Thierry Thomas <thierry at FreeBSD.org>
SageMath is a free open-source mathematics software system licensed
under the GPL. It builds on top of many existing open-source packages:
NumPy, SciPy, matplotlib, Sympy, Maxima, GAP, FLINT, R and many more.
Thanks to SageMath it is possible to access their combined power
through a common, Python-based language or directly via interfaces or
wrappers.
The goal is creating a viable free open source alternative to Magma,
Maple, Mathematica and Matlab.
This is a complex port, with a lot of dependencies, and it has been
broken for some time. Upstream is working on easing its packaging, and
many previously bundled applications can now be replaced by external
packages.
If you are interested, it would be nice to create a team of maintainers
* to maintain some of the dependencies;
* to maintain SageMath itself and prepare the next release (9.2 is
coming!).
__________________________________________________________________
Third-Party Projects
Many projects build upon FreeBSD or incorporate components of FreeBSD
into their project. As these projects may be of interest to the broader
FreeBSD community, we sometimes include brief updates submitted by
these projects in our quarterly report. The FreeBSD project makes no
representation as to the accuracy or veracity of any claims in these
submissions.
chaifi - a tool to simplify joining public WiFi networks
Links
chaifi
URL: https://github.com/gonzoua/chaifi
Contact: Oleksandr Tymoshenko <gonzo at FreeBSD.org>
chaifi is a TUI (text UI) utility aimed at simplifying the process of
joining public WiFi networks in places like coffee shops or libraries.
It replaces the process of scanning and manually editing
wpa_supplicant.conf with an interactive dialog. The utility is in no
way a replacement for full-featured network managers in major desktop
environments. Still, if you're working from a console or using a tiling
window manager, it may save you some seconds (or in worst case minutes)
of your time.
__________________________________________________________________
MixerTUI
Links
mixertui
URL: https://gitlab.com/alfix/mixertui
Contact: Alfonso Sabato Siciliano <alfonso.siciliano at email.com>
MixerTUI is a volume mixer with a Terminal User Interface built on the
FreeBSD sound system. It can show the current Sound Driver
configuration and select an audio device: to get its information, to
change the volume or to set it as default, the last feature allows to
switch easily audio from/to laptop and hdmi, headphones and speakers,
and so on.
MixerTUI can be installed via the audio/mixertui port.
I would like to thank the FreeBSD community for the tips, feedbacks and
patches to improve this project.
__________________________________________________________________
Potluck - Flavour & Image Repository for pot
Links
Potluck Repository & Project
URL: https://potluck.honeyguide.net/
Potluck on github
URL: https://github.com/hny-gd/potluck
pot project
URL: https://pot.pizzamig.dev
Contact: Stephan Lichtenauer <sl at honeyguide.eu>
pot is a jail management tool that also supports orchestration through
nomad.
Potluck aims to be to FreeBSD and pot what Dockerhub is to Linux and
Docker: A repository of pot flavours and complete images for usage with
pot.
This should simplify setting up complex software with many packages and
ports in comparison to manual configuration: Potluck aims to provide a
content library as an additional layer of abstraction, on top of
existing infrastructure like pkg, that pot has to offer.
Pot "flavour" files are provided on
[Github]((https://github.com/hny-gd/potluck)) and fed into a Jenkins
instance. On the Potluck Repository, for each flavour, detailed
descriptions as well as ready-made images to be imported by pot are
provided.
The initial project has been set up, and three simple flavours, along
with a complete Jitsi Meet instance in a jail has been created as a
Proof of Concept that should allow running a fully-fledged video
conference system with just a few easy commands within a few minutes.
As only the initial versions have been set up and implemented so far,
general feedback, tests, as well as additional, useful flavours are
very welcome!
__________________________________________________________________
News Home | Status Home
Site Map | Legal Notices | © 1995-2020 The FreeBSD Project. All rights
reserved.
További információk a(z) BSD levelezőlistáról