Closed
Conversation
This is a rollup of a lot of changes. In summary, lots of bug fixes,
Unicode support and CSV support to match the 2nd Edition of the Awk
Book.
In detail, from AWK's FIXES and FIXES.1e:
From FIXES:
Oct 30, 2023:
multiple fixes and a minor code cleanup.
disabled utf-8 for non-multibyte locales, such as C or POSIX.
fixed a bad char * cast that causes incorrect results on big-endian
systems. also fixed an out-of-bounds read for empty CCL.
fixed a buffer overflow in substr with utf-8 strings.
many thanks to Todd C Miller.
Sep 24, 2023:
fnematch and getrune have been overhauled to solve issues around
unicode FS and RS. also fixed gsub null match issue with unicode.
big thanks to Arnold Robbins.
Sep 12, 2023:
Fixed a length error in u8_byte2char that set RSTART to
incorrect (cannot happen) value for EOL match(str, /$/).
-----------------------------------------------------------------
[This entry is a summary, not a precise list of changes.]
Added --csv option to enable processing of comma-separated
values inputs. When --csv is enabled, fields are separated
by commas, fields may be quoted with " double quotes, fields
may contain embedded newlines.
If no explicit separator argument is provided, split() uses
the setting of --csv to determine how fields are split.
Strings may now contain UTF-8 code points (not necessarily
characters). Functions that operate on characters, like
length, substr, index, match, etc., use UTF-8, so the length
of a string of 3 emojis is 3, not 12 as it would be if bytes
were counted.
Regular expressions are processes as UTF-8.
Unicode literals can be written as \u followed by one
to eight hexadecimal digits. These may appear in strings and
regular expressions.
From FIXES.1e:
Sep 06, 2023:
Fix edge case where FS is changed on commandline. Thanks to
Gordon Shephard and Miguel Pineiro Jr.
Fix regular expression clobbering in the lexer, where lexer does
not make a copy of regexp literals. also makedfa memory leaks have
been plugged. Thanks to Miguel Pineiro Jr.
Dec 15, 2022:
Force hex escapes in strings to be no more than two characters,
as they already are in regular expressions. This brings internal
consistency, as well as consistency with gawk. Thanks to
Arnold Robbins.
Sep 12, 2022:
adjbuf minlen error (cannot be 0) in cat, resulting in NULL pbuf.
discovered by todd miller. also use-after-free issue with
tempfree in cat, thanks to Miguel Pineiro Jr and valgrind.
Aug 30, 2022:
Various leaks and use-after-free issues plugged/fixed.
Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
May 23, 2022:
Memory leak when assigning a string to some of the built-in
variables. allocated string erroneously marked DONTFREE.
Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
Mar 14, 2022:
Historic bug: command-line "name=value" assignment had been
truncating its entry in ARGV. (circa 1989) Thanks to
Miguel Pineiro Jr. <mpj@pineiro.cc>.
Mar 3, 2022:
Fixed file management memory leak that appears to have been
there since the files array was first initialized with stdin,
stdout, and stderr (circa 1992). Thanks to Miguel Pineiro Jr.
<mpj@pineiro.cc>.
December 8, 2021:
The error handling in closefile and closeall was mangled. Long
standing warnings had been made fatal and some fatal errors went
undetected. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
Nov 03, 2021:
getline accesses uninitialized data after getrec()
returns 0 on EOF and leaves the contents of buf unchanged.
Thanks to Volodymyr Gubarkov, and Todd C Miller.
Oct 12, 2021:
The fix for freebsd#83 changed the code to insert 2 chars, but the
call to adjbuf just above it only allows for 1 char. This can
cause a heap buffer overflow.
Sponsored by: Netflix
Measure the total deferred time (from the time we decide to defer until we try again) for busdma_load requests. On systems that don't ever defer, there is no performnce change. Add new sysctl hw.busdma.zoneX.total_deferred_time to report this (in microseconds). Normally, deferrals don't happen in modern hardware... Except there's a lot of buggy hardware that can't cope with memory > 4GB or that can't cross a 4GB boundary (or even more restrictive values), necessitating bouncing. This will measure the effect on the I/Os of this deferral. Sponsored by: Netflix Reviewed by: gallatin, mav Differential Revision: https://reviews.freebsd.org/D42550
…ge versions This is consistent with version numbers used in releng/13.2. PR: 275051 Reviewed by: bapt MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D42562
When KMSAN is configured, the instrumentation inserts calls to __msan_get_context_state() into all function prologues. The implementation dereferences curthread and thus assumes that x18 points to the PCPU area. This applies in particular to init_secondary(), which currently is responsible for initializing x18 for APs. Move initialization into locore to avoid this problem. No functional change intended. Reviewed by: kib, andrew MFC after: 2 weeks Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D42533
Release notes at https://www.sqlite.org/releaselog/3_44_0.html. Obtained from: https://www.sqlite.org/2023/sqlite-autoconf-3440000.tar.gz MFC after: 2 weeks Merge commit '3a88fe13354325b9f56f07927e654de4f5398707'
The tty_rubchar() code handling backspaces for UTF-8 characters didn't properly check whether the beginning of the current line was reached. This resulted in a kernel panic in ttyinq_unputchar() when prodded with certain malformed UTF-8 sequences. Fixes: PR 275009 Reviewed by: christos MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D42564
All of these used the 'immediately at beginning' variation of the BSD-2-Clause license. This wasn't intentional, just what I copied from from a random file in the tree back in 2005. It was not an intentional decision. The different arch bus.h files are a mix of BSD-2-Clause and BSD-4-Clause that have various copyright holders (Charles M. Hannum, Christopher G. Demetriou, The NetBSD Foundation and KATO Takenori), and some of the content of these files were likely copied from there. However, apart from the uncopyrightable interface lines, there are very few comments. It's unclear if these comments are 'original material' here to copyright, but to the extent that there is, license it under the standard BSD-2-Clause copyright that's the norm for the project today. In any event, the standard BSD-2-Clause is also closer to those originals. In addition, FreeBSD uses different type definitions than the original NetBSD code in part. The comments that were copied have been copied a lot, but appear in NetBSD's bus.h files in NetBSD 1.3. While I'm here, assign the copyright, to the extent any exists from me, to the FreeBSD Foundation. I just cut and pasted these into _bus.h from the different machine files and those files have a rich history of modification from the original imports from NetBSD over more than 25 years so it's tricky to say who, exactly, wrote each bit. Given the size of the files, this seems like the best compromise. Also add an acknowledgement to the NetBSD 1.3 bus.h files and their authors (there were no additional FreeBSD authors listed in the various sys/*/include/bus.h files). Finally, use the SPDX identifier instead of multiple copies of the text. Differential Revision: https://reviews.freebsd.org/D42532 Sponsored by: Netflix
Add the forgotten field to these implementations as well. Pointy hat to: me (make universe vs make tinderbox) Fixes: 20f8814 Sponsored by: Netflix
The get operations change the data pointed to by the structure, but do not update the contents of the struct. Mark the struct mac arguments of mac_[gs]etsockopt_*label() and mac_check_structmac_consistent() const to prevent this from changing in the future. Reviewed by: markj MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D14488
The option was added to parallel the CLANG_IS_CC which was removed in commit 20a66ab. Reviewed by: imp, dim, emaste Differential Revision: https://reviews.freebsd.org/D42575
Release notes at
https://www.nlnetlabs.nl/news/2023/Aug/30/unbound-1.19.0-released/
MFC after: 2 weeks
Merge commit '16fd0b24910488e59ca1941387b9ac7fb646a837' into unbound
This avoids possible NULL dereference when checking mnt_vfc names. Reviewed by: jah, rmacklem, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D42554
…_FILE_RANGE() This is required e.g. for nullfs to ensure liveness of the lower mount points. Reviewed by: jah, rmacklem, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D42554
MFC after: 1 week
Ensure that ldconfig-managed elf and elf32 hints always include the standard library paths that are known independently to rtld. PR: 275031 Reviewed by: kib MFC after: 2 weeks Sponsored by: Saltant Solutions LLC Differential Revision: https://reviews.freebsd.org/D42557
Upstream tarball included config.h~. This was previously manually removed from vendor where in fact it should not have been as it would have caused a merge conflict. Reported by: "Herbert J. Skuhra" <herbert@gojira.at> MFC: 2 weeks
This was missed earlier today. Reported by: "Herbert J. Skuhra" <herbert@gojira.at> Fixes: 16fd0b2 MFC: 2 weeks
Fix a typo in a debug print that prevents compilation. Sponsored by: Netflix
Call sigexit rather than exit1 so that a core is generated. If running the SIGABRT handler is desired, this would need to use kern_psignal() instead. In that case a userspace wrapper in libc would be needed to force an exit if the handler doesn't exit. Given that abort2(2)'s intended use case is when userland is in a sufficiently bad state such that it can't safely call syslog(3) before abort(3), a userspace abort2(3) wrapper in libc might be dubious. Reviewed by: Olivier Certner <olce.freebsd@certner.fr>, emaste Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D42163
Use compare_numarray() method for binary search. This fixes table lookups for keys greater than UINT16_MAX. Obtained from: Yandex LLC MFC after: 1 week Sponsored by: Yandex LLC
Merge in the November 2nd, 2023 version of one true awk. This brings in Unicode support, CSV support and a number of bug fixes. Sponsored by: Netflix Reviewed by: delphij Differential Revision: https://reviews.freebsd.org/D42447
When the macro PACKAGE_BUILDING is set, then consider we are building package for pkgbase, this has already been used in Makefile.inc1 and reuse the PACKAGE_BUILDING macros already used for that purpose in the ports tree In the future this should be tied to REPRODUCIBLE_BUILD MFC After: 3 days Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D42569
In 188fe88 OSVERSION has been set to OSRELDATE which is the RELDATE of the building OS while we wanted to use SRCRELDATE which is the RELDATE of the target system
There is no reason in truncating 32 bits cookie value to 16 bits. Reviewed by: glebius MFC after: 2 weeks
Dummynet re-injects an mbuf with MTAG_IPFW_RULE added, and the same mtag is used by divert(4) as parameters for packet diversion. If according to pf rule set a packet should go through dummynet first and through ipdivert after then mentioned mtag must be removed after dummynet not to make ipdivert think that this is its input parameters. At the very beginning ipfw consumes this mtag what means the same behavior with tag clearing after dummynet. And after fabf705 pf passes parameters to ipdivert using its personal MTAG_PF_DIVERT mtag. PR: 274850
ihoro
pushed a commit
that referenced
this pull request
Nov 15, 2024
Avoid calling _callout_stop_safe with a non-sleepable lock held when
detaching by initializing callout_init_rw() with CALLOUT_SHAREDLOCK.
It avoids the following WITNESS warning when stopping the service:
# service ipfilter stop
calling _callout_stop_safe with the following non-sleepable locks held:
shared rw ipf filter load/unload mutex (ipf filter load/unload mutex) r = 0 (0xffff0000417c7530) locked @ /usr/src/sys/netpfil/ipfilter/netinet/fil.c:7926
stack backtrace:
#0 0xffff00000052d394 at witness_debugger+0x60
#1 0xffff00000052e620 at witness_warn+0x404
#2 0xffff0000004d4ffc at _callout_stop_safe+0x8c
#3 0xffff0000f7236674 at ipfdetach+0x3c
#4 0xffff0000f723fa4c at ipf_ipf_ioctl+0x788
#5 0xffff0000f72367e0 at ipfioctl+0x144
#6 0xffff00000034abd8 at devfs_ioctl+0x100
#7 0xffff0000005c66a0 at vn_ioctl+0xbc
#8 0xffff00000034b2cc at devfs_ioctl_f+0x24
#9 0xffff0000005331ec at kern_ioctl+0x2e0
#10 0xffff000000532eb4 at sys_ioctl+0x140
#11 0xffff000000880480 at do_el0_sync+0x604
#12 0xffff0000008579ac at handle_el0_sync+0x4c
PR: 282478
Suggested by: markj
Reviewed by: cy
Approved by: emaste (mentor)
MFC after: 1 week
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.